#323 David Ha: Why Model Merging Could Be the Next AI Breakthrough

This episode is sponsored by tastytrade. Trade stocks, options, futures, and crypto in one platform with low commissions and zero commission on stocks and crypto. Built for traders who think in probabilities, tastytrade offers advanced analytics, risk tools, and an AI-powered Search feature. Learn more at https://tastytrade.com/ Artificial intelligence is reaching a turning point. Instead of building bigger and bigger models, what if the real breakthrough comes from letting AI evolve? In this episode of Eye on AI, David Ha, Co-Founder and CEO of Sakana AI, explains why evolutionary strategies and collective intelligence could reshape the future of machine learning. We explore model merging, multi-agent systems, Monte Carlo tree search, and the AI Scientist framework designed to generate and evaluate new research ideas. The conversation dives into open-ended discovery, quality and diversity in AI systems, world models, and whether artificial intelligence can push beyond the boundaries of human knowledge. If you're interested in AGI, evolutionary AI, frontier models, AI research automation, or how AI could start discovering science on its own, this episode offers a clear look at where the field may be heading next. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI(00:00) AI Should Evolve, Not Just Scale (03:54) David's Journey From Finance to Evolutionary AI (10:18) Why Gradient Descent Gets Stuck (18:12) Model Merging and Collective Intelligence (28:18) Combining Closed Frontier Models (32:56) Inside the AI Scientist Experiment (38:11) Parent Selection, Diversity and Innovation (49:25) Can AI Discover Truly New Knowledge? (53:05) Why Continual Learning Matter

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Craig S. Smith?

  Let's say you have ten agents and you give them the same problem and then you come up with solutions. What are you combining at that point and is it being done in pairs? I mean, does each offspring have two parents or can you combine more than one?

• 0:16Speaker 01

  Our first paper on model merging is, does require access to the weights of the model. It's an impractical assumption because in reality, most of the frontier models are closed. And even if we did have access to the waste, they probably do not share architectures.

• 0:31Craig S. Smith?

  This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures and crypto, all in one platform with low commissions, including zero commissions on stocks and crypto,

• 1:15Craig S. Smith?

  so you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One Click Trading and Smart Order Tracking. And if you're still learning, Tasty Trade offers dozens of free educational courses

• 1:59Craig S. Smith?

  plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker dealer and member of FINRA, NFA, and SIPC. I'm interested in evolution, neuroevolution, which in your co-author on a book that just came

• 2:39Craig S. Smith?

  out, I've talked to Risto and Sebastian, but I'm interested, I don't want to talk about the book so much, I want to talk about neuroevolution or evolutionary strategies and what you're doing at Sakana or Sakana in those realms. And it seems like you've got a few layers, I mean, you had the model merging, which we can talk about, and then, you know, evolutionary AI

• 3:17Craig S. Smith?

  generally, but then AI, a scientist, takes those principles and applies it to research. So can we start by talking about evolutionary AI or evolutionary strategies and your work in artificial life, which is part of all of that, and how you came to that area, particularly since you came from finance, which doesn't sound related.

• 3:54Speaker 01

  Yeah, oh, it is quite related, I think. You know, I feel like having lived, went through the financial crisis in 2008, it really made me learn about extreme events in the world. I think anything can happen. Like, I think people are usually underestimating these sort of tail effects, and I think my experience at that time kind of shaped my views about how our life, you know, how our civilization.

• 4:32Speaker 01

  Has came to be, um, you know, how our intelligence has came to be. And that in turn has shaped my views on how artificial intelligence could be developed. Hmm. So if if we take a step back, um, when I first, uh, no, I I studied neural networks when I was an undergraduate in the University of Toronto. Uh, some of the uh earliest, I did my thesis back then on using neural nets to do computer vision. Uh, analyzing uh whether there's uh these steganography marks in images uh back then. Uh, but um, you know, after entering the finance world, uh, I I did come back to the AI world when I first stumbled onto a few papers.

• 5:16Speaker 01

  Uh, and articles about neuro evolution. Uh, this was in particular, the the works and the presentations by Kenneth Stanley and Risto. Uh, who is my co author of the book. Uh, so back then, you know, around more than ten years ago, people were quite excited about um image recognition. You have uh these GPUs, uh, now they can recognize images and winning Alex nets and so on. Yeah. Uh, but so that was, that was, you know, fine and dandy, you know, people can recognize cat from a dog. But for me, I I was more interested in in more general AI. And then I was, I was like browsing and learning about more about what's going on in AI, because it seems to be coming back.

• 6:01Speaker 01

  And you had these, uh, I I encountered a slide deck, actually, published by Risto. Uh, it was for a workshop, a tutorial in, I I think it was one of the earlier evolution conferences, like a Gecko conference. Uh, where they were, they were using neural networks uh for, you know, playing games, uh, having characters move around. Evolving the brains of these artificial creatures that can work together. So on one hand, you know, you have a bunch of neural nets that that can recognize computer digits or images. And on the other hand, you have this area where you're, you're evolving the brains of these uh artificial creatures that can play in games. And I thought that was really cool, right?

• 6:45Speaker 01

  So I said, how how come the the rest of the world are focused on training these neural network models for? For kind of these narrow classification tasks. And you have this other field that no one talks about that are using, uh, that are evolving these neural networks to, to create arts, to actually have multiplayer games and to kind of, uh, uh, evolve the morphology of neural networks as well. So that really got me in. And the more I studied about neural evolution, the more I understand that's, you know, like the, it's a broader view of how, uh, of, of AI.

• 7:24Speaker 01

  You know, like we had machine learning back then where we're, we're essentially training predefined neural networks to do a particular task. Uh, and I think, uh, in the field of, uh, neural evolution or evolutionary computation, there are many ideas that are broader which appeal to me. Uh, there's the idea of, you know, not, not just sticking to systems where you can calculate the gradients, uh, to do something. Cause I think in, in our world, like many things, uh, many problems are such that there's actually no particular gradient signal in. Yeah. And the idea that if you just chase the gradient, sometimes you can get stuck. Uh, sometimes you kind of have to wander around the solutions to find the even better solution.

• 8:07Speaker 01

  Uh, I like the idea in evolution of this, uh, open-ended discovery or open-ended search. Uh, so rather than having one particular objective in mind, your objective is to find new objectives, find new novelty. So I think this idea is not just, it came out of the neural evolution community, but it's not like, uh, implicitly an evolutionary thing. It's basically defining goals that's haven't been done before. And the, this idea has actually moved beyond the evolution community into broader AI. Uh, and I also like the idea of, uh, Of using evolutionary computation to evolve new neural network architectures and morphologies.

• 8:52Speaker 01

  So this was actually an idea that was formed in the evolutionary area. And I think when I joined Google in two thousand and sixteen, some of these ideas has been incorporated into deep learning. So now, uh, you know, when evolution, actually, the the core concepts has been embraced within like a Google Deep Mind and Google Brain when I joined in two thousand and sixteen. So people then took these ideas to develop them further. Now we ended up into things like neural architecture search. Using evolution to optimize how these connections of the transformers were connected.

• 9:32Speaker 01

  So I think right now, looking back, many of the ideas in evolutionary competition in the last decade has been embraced by the broader AI community. And we're seeing some good progress because of this as well. Now, now, you know, there's many groups working on open-ended discovery rather than just a fixed, fixed optimization. Many groups are using evolution to evolve architectures of neural networks, evolving in the L L M world, evolving the the prompts or the relationship between agents. And, you know, getting back to your earlier question, I I I became interested in AI because of evolution.

• 10:17Speaker 01

  But I'm not stuck just in evolution and I joined Google, you know, in their Google Brain research team. And I was also really excited about the deep learning revolution as well. So all of this fantastic progress about large neural networks that can generate images, generate videos and so on. So when I joined Google in two thousand and sixteen, I wasn't just working on evolution. I I worked on some of the earlier works in generative AI. Uh, I with my collaborator, Jürgen Schmitheuber. Uh, we actually extended some of his ideas in nineteen from the early nineties. Uh, and we brought the world models concept back to the modern era.

• 10:59Speaker 01

  So, uh, I conceived of some of the earlier video generation models. Uh, so the at the time, it was the the goal was not just for creating a video. That that's more like, you know, uh, uh, uh, specific application. But is to build world models for general uh agents uh that can learn uh inside of their imagination. So that they can be more sample efficient, they can uh halluc- they can kind of like a have a have a model of how to plan actions. Without actually trying them in the actual world and to to actually have a framework for building knowledge. So I was actually quite excited about, you know, deep learning in general, uh, in in particular world models.

• 11:44Speaker 01

  Uh, all this generative AI, now we call them generative AI, but before it was more like uh gen- generating. The data distribution of what you learn and how to combine uh generative AI or deep learning. Uh, with concepts in artificial life and evolution. So I think, you know, as I became uh uh more of a professional researcher, uh, even though I started. Uh, you know, getting interested in artificial life and evolution. Uh, I I worked on the core deep learning for several years. And many of my my works, uh, I would say it's a bridge between uh neuro evolution and deep learning. Uh, for example, when I was working on world models, uh, I would use evolution uh components.

• 12:29Speaker 01

  To help find adversarial weaknesses of the world models. Now, how to actually uh evolve different strategies to to trick the world model. Uh, I would, I was using uh evolution to evolve the morphologies of these uh virtual robots. Which will then use reinforcement learning to train the robots. So kind of I view evolution as an outer loop uh of of how things progress and deep learning as the as the inner loop. And, you know, now, uh, fast forward many years when I started the Sakana AI company. Uh. We continue to combine uh the frontiers of generative AI and deep learning with evolution.

• 13:14Speaker 01

  Uh. You know, one of one of the things that we worked on, as you mentioned earlier, is developing the technology to combine frontier models. Uh. Whether they're open source or closed source, because I think uh the, I think more more fundamentally or philosophically. Our intelligence as a species is not like a giant big brain. We we are a collective intelligence of like, you know. Um. Eight billion people walking around with thirty watts in in this thing here, and we talk to each other and exchange ideas. So. This is how our human uh artificial or whether it's an artificial or biological intelligence is formed.

• 13:54Speaker 01

  And I think this is also a good model of how uh artificial intelligence could be developed. So in in the AI space, we have many, you know, our observation was we have like a a large ecosystem of open source models. Developed enthusiastically by many people working on AI uh models. And we have closed models developed by all these great companies like Google, OpenAI, uh Cohere, Anthropic, and in China, DeepSeek and Alibaba and so on. So each of these models have their own unique uh strengths and weaknesses. And it kind of makes sense for me uh if we wanted to build a model with the very, very best performance on a particular thing.

• 14:40Speaker 01

  Is to conceive of a scheme to combine uh different models and the different strengths of all these models. So one of the first experiments we did is we use an evolutionary algorithm. Uh. And we use evolution strategies to merge different open models so that we can create custom capabilities. Like if we wanted to have an open Japanese language model that can do math a bit better or coding a bit better, we couldn't do that. Uh. And recently at uh at NERIPs this year, we also presented another paper uh called A B M C T S where we. Use a Monte Carlo tree search, uh, which is kind of a form of um like a um a tree based approach

• 15:22Speaker 01

  of combining closed models, where we can combine OpenAI's models, Google's models, DeepSeek models to get the very, very best performance on like AGI to some of these hard challenges. So one of our theme is to try to build systems to bring the best out of our foundation models. And I think evolution is a fairly strong candidate in that domain. Uh, the the other parts that uh our company works on, I would say like the this collective intelligence part is more like our our developing a foundation layer, developing the very, very best foundation layer technology to get high performance on a particular task.

• 16:04Speaker 01

  Uh, the other observation that we've been seeing is with the advent of these frontier models. Uh, I think large language models can, you know, they they hallucinates a lot. Uh, but at the same time, just like humans hallucinate a lot, uh, some of these hallucinations lead to great ideas. Uh, sometimes we just hallucinate and, you know, uh, some of the best inventions come out of serendipity. Uh, you know, we did, we didn't really plan for this stuff to happen. Uh, and we, our company explored this, our lab explored this last year. Uh, by using evolution, uh, combined with L L M's for the purpose of idea generation.

• 16:45Speaker 01

  So last year, uh, we presented a paper at NURIPS in two thousand twenty four called Disco Pop. And in that paper, I believe we're we're the first or one of the first, uh, published, uh, papers to show that L L M's can actually conceive of new L L M training algorithms. So the the key idea behind Disco Pop is you take a frontier model back then. It could be like ChatGPT or it could be like Gemini, uh, or it could be like Lama. And you would use that model to produce thousands of ideas on uh how to produce a better algorithm. To train the L L M, uh, so that we can train L L M's more efficiently.

• 17:30Speaker 01

  So we actually called this idea internally called L L M squared. It was a bit ambitious and we would use evolution uh to find some of the best solutions that were hallucinated. Uh, and this solution has to be tested on actual code. They actually have to run as a Python code uh on PyTorch. And evolution would combine these ideas for the for the next iteration and next generation. And then we ended up with a a pretty good state of the art algorithm for training and fine tuning L L M's. So that that was uh one of the uh uh one of one of the test cases of using evolutionary computation.

• 18:09Speaker 01

  To select and breed the best algorithmic ideas generated using L L M's to train another L L M. And I think that was a really neat because ultimately this technology is consuming so much resources in the world. That's it makes sense to use A I itself to make A I more efficient. Uh, so after that work, uh, we we were, we went a bit further. Uh, we thought that it was quite promising that L L M's can conceive of L L M training algorithms. Uh, then we proceeded to uh propose the A I scientists, which is a general framework for a system of agents. To conceive of new scientific ideas. Uh, they have to, you know, for us, we we limit ourselves to computational sciences.

• 18:54Speaker 01

  Uh, because we we may not have the resource to run a wet lab uh in our company. And we would have this system of agents like run experiments on our humble cluster. They have to actually collect and refine the experiments. And perhaps even publish a, you know, write, write up an academic paper using LaTeX. Uh, and we would also train an a another L L M agent for the purpose of reviewing these papers. So it's kind of, you kind of have an adversarial thing going on. You have your scientists and then you have reviewers that were calibrated on scientific review. Uh, and then you can have an evolutionary system in the loop to kind of select the best ideas and iterate on. So I I feel like, um.

• 19:36Speaker 01

  Using evolution combined with the search process of L L M's makes a lot of sense. Because you know, rather than having just one L L M telling us what to do. It's much more reliable to have L L M's tell us a thousand different things. And have an evolutionary algorithm or a search algorithm go through each of them to let us know what is the best thing. And then iterate the ideas from there. So evolution for us has been a really good tool. Of you know, in summary, to combine the frontier models to get the very best performance. It's also been a remarkable tool for for you using this technology.

• 20:19Speaker 01

  And repurposing them for the purpose of of A I driven discovery. So we we've been, you know, I think our lab is known for these two themes. And we've been able to, you know, grow as a company to build on these two R and D developments. And you know, in the last six months or eight months, using these tools, we've been actually able to to grow grow a healthy and sustainable enterprise business in Japan. Japan of all places to deploy systems. So yeah, things are going well. So we're looking forward to to next year and beyond as we grow this thing.

• 20:59Craig S. Smith?

  Yeah, I just want to ask your your work on world models. I just had Fei Fei Li on talking about marble and and her world labs. And in the past, I've spoken a lot to Yan Le Kuhn about Japah and his, you know, his world model work. And there's a fundamental difference between what Fei Fei is doing and what Yan is doing. Fei Fei is creating an internal representation that that then outputs an explicit model of the world of whatever world you want to output.

• 21:43Craig S. Smith?

  Yan is working on building an internal representation of the world as it exists.

• 21:50Craig S. Smith?

  In order to reason on that model. And what, whi- which side is Sakana on? And how do you build your world models?

• 22:02Speaker 01

  Yeah, I mean, this is, I, I was actually quite proud of like the, the concept of world models being an actual thing. In, in two thousand twenty five, there's, there's many companies working on this and many startups. And I think like both Fei-Fei Li and Yan Le Cun are great researchers. Uh, on their own right, they're probably exploring things that are worth exploring. Uh, my impression, you know, as you mentioned, I think Fei-Fei Li's startup, World Labs, focuses on more like a developing a representation for realistic three D environments. Uh, you can tell by many of the demos that, that they look like, you know, three D games. Uh, this is similar to Google's genie efforts as well. Uh, you, you look at many of the generations and they, they look kind of a three D generated.

• 22:46Speaker 01

  So I think the emphasis is to create a representation of a realistic three D world. Uh, where, you know, the, uh, the, the principle behind that is if you're able to, to harness the compute and scale, uh, you're able to create and higher and higher for the fidelity models. Where you can then, you know, do many things inside of the world models, even training entire agents inside the world models to learn new skills. Uh, so this is not just Fei-Fei Li's work, but recently, CIMA, which, which is released by Google, uh, also exported this concept of a high fidelity world models that is hyper realistic.

• 23:31Speaker 01

  And I think Yan, Yan Le Cun's view is more like, uh, uh, developing more of the abstract representations uh, for world models, uh, where these abstractions are required to, to learn, uh, reasoning concepts. Uh, you know, you, you want to have these like a thought vectors inside the world models. And this is how you, you would do things. So, so both, both are, you know, very, you know, actually required directions for this technology to be done. One is scaling and one is. More like investigating the representation. Now, for for me, uh, when I started the world models project with Schmidt Huber, um, uh, around, you know, two thousand fifteen, sixteen. Uh, I think the Schmidt Huber's earlier concepts is more, actually more.

• 24:16Speaker 01

  I would say it preceded Yanekun's ideas. Like, I mean, I mean, usually that's the case. Uh, but a lot, lot of the motivation behind the world models work that I worked on is on, is on developing representation. Like, I, I didn't really care whether the world models would output a really realistic. Right. Uh, like a ren- rendering of the real world. And in fact, we showed that's the more realistic it is. Uh, the hu- the higher fidelity it is, the, the easier for evolution or the agent to exploit some really weird bug in your simulation.

• 25:00Speaker 01

  So it, it's kind of a, I, I think this is, this is more like a paradox. The, the larger your model is and the, the more detailed it is, it's actually easy to actually find some particular small thing for your agents. If your agent is to, to beat that game that's simulated in your role model, it'll, it'll figure out how to move in such a way to get unlimited scores right away. Uh, but what we found is if you make your model noisier, where there's actually a, a bigger gap between your, your simulated reality and the re- the real game or the real environment, your agent actually is forced to learn real, like a more important skills

• 25:43Speaker 01

  or it, it, it has to actually fend for itself uh, in, in this uh, really nasty game. So you, you imagine your, your representation is, is not so perfect. Uh, but it's actually more tricky. Uh, you know, it, it's more tricky to navigate in the dream than the real world. It actually uh, presents some, some of a difficulty for your agents uh, to, to develop the skill. So I think it's an open question. I think it's a balance. Of course, you cannot have a. You cannot go, you know, have a pure latent model with no grounding in reality. Uh, then that would be foolish and it would not learn anything useful. But I'm really interested in studying this balance.

• 26:26Speaker 01

  Uh, I think as humans, uh, uh, I I'm sure that's we don't have the same type of capability as Fei-Fei Lee's world models or or Gini model. Like, we cannot, uh, you know, explicitly, like, at least me, maybe not everyone on Earth, cannot like draw out like photorealistic version of a video of what I'm imagining in my head. But somehow we can reason in this, uh, in this latent representation. And I I'm actually quite sympathetic to Yan Le Cun's direction. Uh, maybe I, uh, his future company would focus on more of this, uh, latent, uh, higher order representation of world models.

• 27:08Speaker 01

  And and to be fair, this is also things that Jürgen Schmidhuber has been conceiving since the early nineties.

• 27:15Craig S. Smith?

  Yeah.

• 27:15Speaker 01

  Uh, you know, he he has been working on, like, actually even using, getting agents to dissect directly into the hidden states of the world model. So you you don't actually use the world model's hidden, uh, latent representation directly. The agents would basically examine directly into all the hidden connections of your world model to be able to figure out intuitively what it should do, even skipping the planet, uh, planning part. Uh, so there there's a lot of work to do and, you know, for for me as a as a startup guy, like, I I really am excited about the world model's direction. Uh, but we have to really focus on our, you know, what we're building as a startup.

• 27:59Speaker 01

  So we decided to focus mainly on on collective intelligence, combining existing models, and using these models for scientific discovery. Uh, you know, because, uh, although, you know, deep inside me, we I I I'm quite passionate about this world model's direction, and I think, uh, we we do want to get back into it at some point.

• 28:19Craig S. Smith?

  Yeah, on the the paper that was printed presented at NERIPS, um, you know, I understand, uh, model merging where you have, um,

• 28:30Craig S. Smith?

  You know, essentially, two code bases and and the evolutionary system decides which, you know, how to combine the two code bases, what to keep, what to throw out, how to fit it all together. But I I didn't understand how you do that with proprietary models where you don't have access to the weights. And it's you're talking about kind of a meta layer that that. Well, explain to me what what the paper that was presented that that. Easier than me.

• 29:10Speaker 01

  It is getting really interesting because our first paper on model merging is it does require access to the weights of the model.

• 29:17Craig S. Smith?

  Yeah.

• 29:18Speaker 01

  I think like a it's it. It'll be in it's an impractical assumption impractical because in reality, most of the frontier models are closed. And even if we did have access to the waste, they probably do not share architectures with each other. So. What we decided to do, and it's also, you know, as these models become bigger, the context length also becomes longer as well. So there is a duality between the prompts and being access accessing the weights of the model. The more context you can put in, it's in a way influencing what the model is doing.

• 30:03Speaker 01

  So in this ABMCTS work that we presented on Europe this year as a spotlight, we we use this scheme to, you know, prompt or search through the reasoning techniques of multiple models. You would actually use the the sort of the bandit approach to give the variations of the same questions for multiple different models, get the responses and decide based on the responses how to actually ask the next iteration of responses in a tree-like fashion.

• 30:39Speaker 01

  Rather than using evolution to combine the weights, we're using like an evolution type scheme. In this case, Monte Carlo tree search uh to actually conceive of different uh conversations with the model. So how to drive the conversation uh with these batch of frontier models to get really great answers for a particular task.

• 31:03Craig S. Smith?

  And and so you're you're prompting a a range of models in in in you're you're taking the the outputs and uh then creating a new set of prompts from those outputs and then re prompting that same uh series of models.

• 31:23Speaker 01

  Yes, that's correct. Uh and it's done away in a in a tree search method. So you you would actually go down the rabbit hole of the most interesting uh you know, dialogues. So it's like if you have a party with one hundred people, uh you would find the five most interesting conversations. And then you were like continue to only talk to those five people and then you keep on going and going. Uh so that that's one work we're doing. Uh in our company, we're actually exploring multiple different ways of combining frontier models as well. So we we will have some exciting work coming up in the next few months. Uh where we we we actually have better results uh for for this direction.

• 32:05Speaker 01

  So I I think i- it's quite exciting for for for combining different models.

• 32:10Craig S. Smith?

  When you say uh in the coming, you're talking about working um on the context and not not you're not talking about using open weight models and combining uh code bases and creating new merged models. Is that right? You're talking about this other strategy.

• 32:33Speaker 01

  Yes.

• 32:33Craig S. Smith?

  You're saying that, yeah, yeah.

• 32:35Speaker 01

  Yeah.

• 32:36Craig S. Smith?

  And the the can you, you know, with AI scientists, I wrote a little article in that when you're uh that paper was written by the model and.

• 32:48Craig S. Smith?

  System and submitted, I think it was the ICML, was it right? I clear, I can't remember.

• 32:56Speaker 01

  Submitted, I believe we submitted, it's quite early work. I mean, the paper is not like particularly good or anything, but we.

• 33:04Craig S. Smith?

  No, I know. Yeah.

• 33:06Speaker 01

  We had to do a few things because right now of the using generative AI to produce papers at an academic conference, it is like a, it's an issue that we have to guide very carefully. So when we ran this experiment, we obtained permission from the program chairs of ICLR. I clear.

• 33:27Craig S. Smith?

  I clear.

• 33:28Speaker 01

  Experiment. We also obtained explicit permission from the workshop organizers for this experiment. And we, we had like the ethics committee at the University of British Columbia approved this experiment. So it's like, because we're running a human experiments. And what we did was we submitted three papers into our workshop at ICLR earlier this year. And I think one of them obtained a score that would have passed the workshop.

• 33:58Craig S. Smith?

  Right.

• 33:59Speaker 01

  But what we agreed with the organizers is by default, even if they were accepted, we would, we would actually reject it, withdraw them because it's part of the experiment. So that was kind of the blind test that we ran.

• 34:13Craig S. Smith?

  That's right. But what was happening in the background there is that, are you, maybe, so to back up a little bit, what's interesting about evolutionary strategies is that unlike gradient descent where you're following one path and trying to find

• 34:35Craig S. Smith?

  a local or universal optimum or optimal, right? You're, you're, you're searching, you have many models. And so you're looking across a much larger space.

• 34:53Craig S. Smith?

  And so is is how it just describe how AI scientists worked in that experience experiment and how it's evolving.

• 35:05Speaker 01

  Yeah, question. So there is a progression in the way we're building the AI scientists program as well. The very first version AI scientist that we released last year. It was more of a AI co-scientist where the user would would actually presents the AI scientist with an initial idea and an initial code base for the experiment. And the AI scientist is more like a grad students where it would take that code base and it would find novel ideas to extend it, run experiments and to get more interesting results. For for the I create workshop experiments, it was a test for our AI scientist version two.

• 35:46Speaker 01

  So we actually worked on this variation where we there's less. There's less of a requirement to provide it with a so-called templates of the particular idea that it wants to work on. It could actually work on its own idea. We just have to guide it by an initial prompts or a general theme. And and it would in the AI scientist V two come compared to V one. There's a a tree type progression of evaluating these ideas. So in inside the AI scientist, it would actually branch out to different variations of ideas and and decide what is the most interesting things you should pursue.

• 36:28Speaker 01

  So we we ran this AI scientist a few times and use the reviewer to select the the three most highly rated papers according to to the the automated reviewer. And we submitted those three to to the workshop. I I think there's still a lot more room to be done going forward. Right now, the what the limitation is, the AI scientist is still one system.

• 36:56Speaker 01

  I think science as we know it is not done by like one person, but like we're building on the shoulders of giants. And we keep on exchanging ideas and building on top of each other's ideas. So one, one thing that we see in the future is not necessarily just one single AI scientist, but more like an AI community of multiple different AI scientists working together and incrementally building on top of each other's ideas and branching out to discover new ideas. So hopefully in, in the future, we're able to make more progress on that fronts.

• 37:33Speaker 01

  Uh, cause I think right now, one of the biggest limitation is, is you know, you have this kind of a single agent's multi-shot idea generation, uh, which is great for, for kind of this, I would say graduate student descent. You, you have this grad students that would improve an idea, but uh, uh, our goal is to actually conceive of novel transformative ideas, uh, which I think is a big limitation in the currents AI scientists in general for, for all labs and all companies. And this is one of the, the grand challenges that we want to work on.

• 38:11Craig S. Smith?

  Yeah. And, and how does that work? So you have a population of models and you give it, give them all the same initial problem, right? And then they, they will end up with a solution or an output. And then you, you score those outputs and you take the, the best ones and combine them in pairs. Uh, and then, uh, ask those

• 38:46Craig S. Smith?

  chi- child models, I mean, the, the offspring, uh, the same question, and then they go through another generation. Can you just describe how that works?

• 38:56Speaker 01

  Yeah, thi- this is a very exciting, open question. That in- inside our lab, we're working on several variations of this. But actually, we're not the only ones. Uh, I think recently at Stanford, they also uh are launching this so-called a AI conference. Or like where people would be submitting papers generated using AI agents. And then you have a conference-type system that would vet these papers to see whether they're good or not. Uh, so I think there, there are many ways that one can go about. Like, you know, one, the, the simplest way, as you alluded to, is, okay, you have a thousand agents.

• 39:36Speaker 01

  And all of them have to just figure out a new uh incremental improvement to this idea. Uh, and you, you would use a evolutionary algorithm to combine the, uh, what are the most promising ideas? And we would call these parents, uh, and you would pre- conceive of child ideas. And this child ideas, you would then branch out to another, the same one thousand agents, and you keep on doing it, uh, and go so forth. And in fact, this paradigm has already been done this year, uh, in, in code space. And it was initially released by Google called Alpha Evolve. Uh, I'm sure you heard of the Alpha Evolve. Of course, yeah.

• 40:17Speaker 01

  Our company extended it into something called Shinkai Evolve, which be- we believe to be a more efficient, a sample-efficient version of Alpha Evolve. Where we showed that the, uh, scaling up the agents for, you know, hundreds of different agents. We can come up with different ideas and you use parent-child sampling. And it would create better code ideas, create be- better algorithms, uh, to solve a particular problem. And this is great. It, it, we can now have like state-of-the-art task on like circle packing, optimization algorithms. These, these sort of difference, different code algorithms. It's really useful when you have a particular task and you want your agents to design the best program for that.

• 40:59Speaker 01

  Uh, I think, uh, uh, if we want to be a bit more ambitious, uh, so this is great for. For like uh software engineering uh or at least designing algorithms for a different task. If we want to have like the grand challenge of coming up with new ideas, it's not good enough. I think for for the the new ideas, we have to insert the concept of interestingness and open-endedness into the search concept. Where uh these agents are not only optimizing for how well they're doing in a particular task. They're also looking at scoring themselves about whether they believe their solution is not novel or interesting.

• 41:38Speaker 01

  Or it adds something to the table. So you know, what what what is, you know, uh what we're trying is also to have these artificial communities built on top of each other's ideas. Uh and you know, many of these these concepts are not novel. Uh in the evolutionary computation community, there's a concept called a quality diversity. So we're, you know, we're not just optimizing for the quality or the score of a task, but we're optimizing for the diversity of the task as well. So if everyone, if if your approach is too similar to the other approach, you have to get the very top score of this batch of approaches. If your approach is completely weird and you know, not really explored, then you get a bit of a leeway.

• 42:21Speaker 01

  Like we don't expect that much of a of a good quantitative score from you, that the bar is better uh just because you're doing something new. So uh many experiments uh before deep learning has shown that this sort of approach leads to uh leads to agents with better generalization, like the robot that can continue to walk even if you damage your legs and so on. It was published by uh by uh Jeff Kuhn and his uh group uh Nature uh many years ago. And I think the same applies to to the AI scientist type work as well. So you you will want to see an agent produce interestingness.

• 42:57Craig S. Smith?

  Yeah, I I I just have a a naive question, so forgive me, but I'm just a journalist. When you, if you, let's say you have ten agents and and you give them the same problem and and then you you come up with solutions and you want to combine.

• 43:21Craig S. Smith?

  Or come, or the combine those, what are you combining at that point? And is it being done in pairs? I mean, does each offspring have two parents? Or can you combine more than one?

• 43:36Speaker 01

  Yeah, this is a great question. And I, I think, you know, you could do it in, in several different ways. Uh, the approach that works quite well is if you take two parents, two solutions. Uh, and then you would feed those two solutions just as a context. Like you can concatenate them and say, hey, L L M, these are two previous solutions. That seems to work. Uh, you know, combine these two solutions and, you know, take the best and produce one more solution for me. And, and the L L M's would be able to do that. It will, it will, it would actually give you the solution. So this is one of the magical things about the current foundation models.

• 44:18Speaker 01

  Because you can pull that random if, if you're writing English. Like before, before deep learning, we have to conceive of a, a particular way of combining things. It waits space, things have to fit together. But now if you're generating ASCII code, you can concatenate them into a longer context prompt. And just tell it to combine these solutions. So I thi- it's quite, you know, we're, we're in this sort of a, a interesting stage in our technology. When we can treat some of these L L M models as an employee. Or we can tell them in plain text what we want to do.

• 44:55Craig S. Smith?

  Yeah, and you're not simply combining, uh, discrete. I mean, you, you, you have 10 models and you have 10 solutions. There are multiple pairs within those 10 solutions. So you're going to have more than 10 offspring, right? Or more than five offspring, I guess. Uh, because you can, you know, A with B or A with C or A with D, B with A, B with C, B with D, right?

• 45:27Speaker 01

  Yeah.

• 45:27Craig S. Smith?

  Uh, and is all of that done, uh, in one, uh?

• 45:35Craig S. Smith?

  Epoch or one go?

• 45:38Speaker 01

  Yeah, so this uh, the, the open question, and this is something that our company explores quite deeply, is in the parent selection process.

• 45:49Craig S. Smith?

  Yeah.

• 45:49Speaker 01

  So, you know, if you have a system of agents that comes up with like a thousand possible different solutions, and each solution, uh, you know, uh, in the quality diversity framework, they can have a performance and they can have like, how, how weird are these solutions?

• 46:06Craig S. Smith?

  Yeah.

• 46:07Speaker 01

  And so you can have a grid of these scores. And the, the, the open question is how to select the next uh, ten solutions to breed? And I think, you know, um, studies before have shown that if you don't, you, you could do the, uh, in the literature, they call it the elitist combination. You would simply just take the ten best performing ones and throw away everyone else and just combine the, the elitist solutions. But you end up with getting into some local minima quite earlier, uh, if you just do that. So what the quality diversity literature suggests, if you, you would combine, you know, some of the, sometimes you would combine

• 46:51Speaker 01

  the low performing but highly weird and innovative solutions with the top performing ones, uh, to actually get new innovations. And I think these are the ideas that we're exploring as a lab, uh, how to best combine them. And I think we've, we've showcased, uh, some of these, uh, our, our rhythmic innovations in choosing in our, uh, in our open source project called Shinkai Evolve, where we actually have explicitly, uh, these sort of a heuristics of how to combine parent selection to generate offspring. And we experimentally showed that leads to a dramatic improvement in sample efficiency, getting, getting the best solutions much more earlier. But I, I think there's still a lot of work to be done in this idea.

• 47:36Speaker 01

  Maybe in the future, it could be an L L M agent's that's tasked with choosing which, which ones you should use. So it's quite interesting.

• 47:45Craig S. Smith?

  Yeah, and I know you don't have a lot of time, so let me shift to sort of a more philosophical uh, question. If this, if you get this working well, uh,

• 47:58Craig S. Smith?

  The the the scope of human knowledge

• 48:05Craig S. Smith?

  is finite, obviously. But the the universe of possible knowledge is potentially infinite, right? And and using gradient methods or or just human creativity, it's very difficult, if not impossible, to think outside the scope of of what's known. And, you know, periodically, you have someone who's extremely creative that that comes up with a new idea. Is the ambition or the belief that

• 48:48Craig S. Smith?

  an AI system could push beyond the the boundaries of existing

• 48:55Craig S. Smith?

  knowledge? And and really be creative or in that these models are ultimately all trained on existing knowledge? Is it? Is it, you know, yes, there's there's some new ideas in mixing existing knowledge. But getting out of the the bubble of human knowledge

• 49:22Craig S. Smith?

  may may not be possible.

• 49:24Speaker 01

  This is a great question. I I I think, you know, with with you know, of course, we don't know for sure, but I'm quite confident that's eventually either us or other labs would be able to create systems of agents that would be able to extend the boundaries of human knowledge and creativity. You know, I I don't think that the current systems are that capable. I think humans are still great at creativity, especially when there are eight billion of us. And of course, some of us are very creative. But that's why I believe in this collective intelligence as we scale up the number of agents, like giving them their own, you know, you know, room to explore. We are good.

• 50:08Speaker 01

  Due to law of large numbers, there will be uh many, you know, tail possibilities of new things being discovered. I think most of the technology would not discover new things. But if if as you scale up the number of agents, uh the number of artificial scientists to a larger number. Of course, some of them would discover new ideas and new possibility. And I think uh to your points about uh training within the bubble of human knowledge, and that's a valid one. Because you're constantly outputting things in the data distribution that you're trained on.

• 50:39Craig S. Smith?

  Yeah.

• 50:39Speaker 01

  But how I think we're gonna get around that is these AI scientists don't operate in a vacuum. I think there are many AI scientist project happening right now, you know, our company, we're focused on like, you know, the algorithmic generation. Other companies are focused on like material synthesis, uh AI for real scientists or drug discovery. And ultimately, these agents have to interact in with the real world. So even in our case, the agents actually have to, you know, write code and run the code on on Python and run it on a cluster. So you are interacting in the real world and collecting feedback, new feedback from the real world, which may not be there in the first place.

• 51:18Speaker 01

  So I I do think that uh these agents uh will collect new data as the experiments within real world. And that's gonna lead to new innovation and new discovery. You know, what whether uh real world interaction is a requirement or not to discovering idea. That isn't, that is an open question. That one, like, you know, there are proponents to say that maybe that maybe you don't actually don't need uh embodiments. But I think it certainly helps because you're getting feedback from from the real.

• 51:52Craig S. Smith?

  Yeah, and embodiment or at least grounding in, you know, in a world model, right?

• 51:58Speaker 01

  Exactly.

• 51:58Craig S. Smith?

  But the um and and then one more question, I forgot to plug in my.

• 52:07Craig S. Smith?

  Power. Um, one other question is, don't you need continual learning in order to get to to true creativity? Uh, and and there are blockers to that, or I I know your

• 52:29Craig S. Smith?

  your your systems can adapt and decide to forget some things, to learn new things. And so there it's not a static

• 52:41Craig S. Smith?

  knowledge base. It's court sort of evolving as the model explores. But ideally, you wouldn't be forgetting you would be

• 52:50Craig S. Smith?

  adding knowledge and then the synthesis of of old knowledge and new knowledge may create new ideas. It's are you guys, uh, do you think that's right? And and how important is continuous learning?

• 53:05Speaker 01

  I I think that's a fair point. And, you know, at NewRIP this year, continual learning is, you know, mentioned many times at several keynotes as the next big AI challenge, even Richard Sutton's keynotes.

• 53:18Craig S. Smith?

  Yeah.

• 53:18Speaker 01

  As you mentioned, if if the agent just doesn't remember what it was just has experienced, how could it come up with new knowledge? I think the way we're doing it right now, you know, I think it definitely helps to have a breakthrough in continual learning. But absence of that, uh, we can still continue to chug along as if we're synthesizing a continual learning agents. You know, whereas like one, one of the good things about these agents is they they can use tools. They can actually have access to, um, you know, a set of knowledge in the past. So as we're building these agents with scientific discovery, you could actually log everything in a text file

• 53:58Speaker 01

  or in in a markdown documents or in a PDF paper where it would actually have to know, um, uh, you know, retrieve all these ideas. And this is a problem, not just for agents, it's for real scientists as well. Like, how many ideas gets rediscovered by young scientists? Like, you know, you cannot expect even human. Human scientists to to know every single publication out there. So there is a need for the agents capability to to to do research to search archive of existing ideas. Whether those ideas are known ideas or conceived by itself or earlier versions of itself. Uh, you would still need to do that regardless of whether you have a continual learning breakthrough or not.

• 54:40Craig S. Smith?

  Yeah, that's interesting. Um, okay, we're up to uh the hour. I want to ask one last question. I asked Faye Faye Lee. What's your guilty pleasure? I mean, you work very hard on research. Is there uh, do you play Jim Rummy in the evenings or do you uh?

• 55:01Speaker 01

  I I like to watch uh retro anime.

• 55:05Craig S. Smith?

  Oh, is that right?

• 55:06Speaker 01

  Yeah.

• 55:06Craig S. Smith?

  In Japan, oh, that's very.

• 55:08Speaker 01

  Like the Dragon Ball and and so on.

• 55:10Craig S. Smith?

  Or Astro Boy or yeah.

• 55:12Speaker 01

  Bring back the anime at the time when, you know, it was purely hand drawn without any computer tools.

• 55:18Craig S. Smith?

  Yeah, yeah, that's fascinating. Yeah. This episode is brought to you by Tasty Trade. On Eye on AI, we talk a lot about how artificial intelligence is changing how people analyze information, spot patterns, and make more informed decisions. Markets are no different. The edge increasingly comes from having the right tools, the right data, and the ability to understand risk clearly. That's one of the reasons I like what Tasty Trade is building. With Tasty Trade, you can trade stocks, options, futures, and crypto all in one platform with low commissions,

• 56:01Craig S. Smith?

  including zero commissions on stocks and crypto, so you keep more of what you earn. The platform is packed with advanced charting tools, backtesting, strategy selection, and risk analysis tools that help you think in probabilities rather than guesses. They've also introduced an AI-powered search feature that can help you. Discover symbols aligned with your interests, which is a smart way to explore markets more intentionally. For active traders, there are tools like Active Trader Mode, One Click Trading,

• 56:40Craig S. Smith?

  and Smart Order Tracking. And if you're still learning, Tasty Trade offers dozens of free educational courses, plus live support from their trade desk reps during trading hours. If you're serious about trading in a world increasingly shaped by technology, check out Tasty Trade. Visit tastytrade.com to start your trading journey today. I'm going to myself. Tasty Trade Inc. is a registered broker dealer and member of FINRA, NFA, and SIPC.