He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI]

The Transformer architecture (which powers ChatGPT and nearly all modern AI) might be trapping the industry in a localized rut, preventing us from finding true intelligent reasoning, according to the person who co-invented it. Llion Jones and Luke Darlow, key figures at the research lab Sakana AI, join the show to make this provocative argument, and also introduce new research which might lead the way forwards. **SPONSOR MESSAGES START** — Build your ideas with AI Studio from Google - http://ai.studio/build — Tufa AI Labs is hiring ML Research Engineers https://tufalabs.ai/ — cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst Submit investment deck: https://cyber.fund/contact?utm_source=mlst — **END** The "Spiral" Problem – Llion uses a striking visual analogy to explain what current AI is missing. If you ask a standard neural network to understand a spiral shape, it solves it by drawing tiny straight lines that just happen to look like a spiral. It "fakes" the shape without understanding the concept of spiraling. Introducing the Continuous Thought Machine (CTM) Luke Darlow deep dives into their solution: a biology-inspired model that fundamentally changes how AI processes information. The Maze Analogy: Luke explains that standard AI tries to solve a maze by staring at the whole image and guessing the entire path instantly. Their new machine "walks" through the maze step-by-step. Thinking Time: This allows the AI to "ponder." If a problem is hard, the model can naturally spend more time thinking about it before answering, effectively allowing it to correct its own mistakes and backtrack—something current Language Models struggle to do genuinely. https://sakana.ai/ https://x.com/YesThisIsLion https://x.com/LearningLukeD TRANSCRIPT: https://app.rescript.info/public/share/crjzQ-Jo2FQsJc97xsBdfzfOIeMONpg0TFBuCgV2Fu8 TOC: 00:00:00 - Stepping Back from Transformers 00:00:43 - Introduction to Continuous Thought Machines (CTM) 00:01:09 - The Changing Atmosphere of AI Research 00:04:13 - Sakana’s Philosophy: Research Freedom 00:07:45 - The Local Minimum of Large Language Models 00:18:30 - Representation Problems: The Spiral Example 00:29:12 - Technical Deep Dive: CTM Architecture 00:36:00 - Adaptive Computation & Maze Solving 00:47:15 - Model Calibration & Uncertainty 01:00:43 - Sudoku Bench: Measuring True Reasoning REFS: Why Greatness Cannot be planned [Kenneth Stanley] https://www.amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237 https://www.youtube.com/watch?v=lhYGXYeMq_E The Hardware Lottery [Sara Hooker] https://arxiv.org/abs/2009.06489 https://www.youtube.com/watch?v=sQFxbQ7ade0 Continuous Thought Machines [Luke Darlow et al / Sakana] https://arxiv.org/abs/2505.05522 https://sakana.ai/ctm/ LSTM: The Comeback Story? [Prof. Sepp Hochreiter] https://www.youtube.com/watch?v=8u2pW2zZLCs Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley] https://arxiv.org/pdf/2505.11581 A Spline Theory of Deep Networks [Randall Balestriero] https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf https://www.youtube.com/watch?v=86ib0sfdFtw https://www.youtube.com/watch?v=l3O2J3LMxqI On the Biology of a Large Language Model [Anthropic, Jack Lindsey et al] https://transformer-circuits.pub/2025/attribution-graphs/biology.html The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff] “The ARChitects” https://www.youtube.com/watch?v=mTX_sAq--zY Neural Turing Machine [Graves] https://arxiv.org/pdf/1410.5401 Adaptive Computation Time for Recurrent Neural Networks [Graves] https://arxiv.org/abs/1603.08983 Sudoko Bench [Sakana] https://pub.sakana.ai/sudoku/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Hey Spotify listeners! When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi. Sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 0:27Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies. 18+.

• 0:52Speaker 02

  Despite the fact that I was involved in inventing the transformer, luckily, no one's been working on them as long as I have, right? With maybe the exception of the other seven authors. So I actually made the decision earlier this year that I'm going to drastically reduce the amounts of research that I'm doing specifically on the transformer because of the feeling that I have that it's an oversaturated space, right? It's not that there's no more interesting things to be done with them, and I'm going to make use of the opportunity to do something different, right? To actually turn up the amount of exploration that I'm doing in my research.

• 1:35Speaker 03

  We just released the continuous thought machine. It's a spotlight at Europe's 2025 this year. You should care about it because it has native adaptive compute. It's a new way of building a recurrent model that uses higher-level concepts for neurons and a synchronization as a representation that lets us solve problems in ways that seem more human by being biologically and nature-inspired.

• 2:02Speaker 02

  The atmosphere in AI research was actually quite different back during the Transformer years, because it doesn't feel like something similar could actually happen right now because of the reduced amount of freedom that we have, right? The Transformers was very, very bottom up, right? It's not that somebody had this grand plan that came down from one high that this is what we should be working on. It was a bunch of people talking over lunch, thinking about what the current problems are and how to solve them, and having the freedom to have, you know, literally months

• 2:41Speaker 02

  to dedicate to just trying this idea and having this new architecture for us. We've spent hundreds of millions of dollars. The biggest sort of evolution-based search is probably in the tens of thousands. We have all this compute. What happens? What happens if you scale up these search algorithms? And I'm sure you will find something interesting, you know, when someone eventually does buy that bullet and really scale up these evolutionary sort of A-life experiments. Because I pitched it in an environment where people were just going all in on this one technology, I got zero interest.

• 3:22Speaker 02

  So now I have my own company and I can pursue those directions.

• 3:28Tim Scarfe?

  This podcast is supported by Cyberfund.

• 3:31Speaker 05

  Hey folks, I'm Amar, product and design lead at Google DeepMind. We just launched a revamped vibe coding experience in AI Studio that lets you mix and match AI capabilities to turn your ideas into reality faster than ever. Just describe your app and Gemini will automatically wire up the right models and APIs for you. And if you need a spark, hit I'm feeling lucky and we'll help you get started. Head to ai.studio/build to create your first app.

• 3:58Tim Scarfe?

  Two for AI Labs is a research lab based in Zurich. They've got a team of amazing ML engineers and research scientists. They're doing some really cool stuff. If you look at their website, for example, you can see what their approach was for winning the ARC AGI3 pub competition, which closed out a few months ago. And they are hiring amazing ML engineers and research scientists. They also care deeply about AI safety. So if any of that is a fit for you, please go to Two for AI Labs and give it a go. The audience will know I'm a huge fan of Kenneth Stanley's ideas. So his book, Why Greatness Cannot Be Planned, changed my life. It was absolutely insane. And what he was speaking to is that we need to allow people to follow their own gradient

• 4:41Tim Scarfe?

  of interest, unfettered by objectives and committees and so on. Because that is how we do epistemic foraging, that when you have too many agendas involved in the mix, you kind of end up with a grey goo and you don't discover, you know, interesting novelty and diversity. And I suppose that's basically the thesis of your company, Sakana, is to lean into those ideas.

• 5:05Speaker 02

  Yes, exactly. At the company, we're a massive fan of that book. We're hoping to have him come and talk at our company next week, actually. And it's a philosophy that we do talk about internally, right? We have copies of the books, including the recent Japanese translation. As, you know, one of the co-founders, one of my main jobs, one of the main things that I have to keep doing for this company is making sure that we protect the freedom that the researchers currently have, right? Because it's a privilege, really, that we have the resources to be able to do that. And inevitably, as I've seen happen, as the company grows, more and more pressure comes in and it narrows the freedom.

• 5:50Speaker 02

  But I think because, you know, we believe in this philosophy so strongly, I'm hoping that we can give people all the research freedom that we do now for as long as possible.

• 6:04Tim Scarfe?

  And what are those processes that curtail freedom as a company matures? I mean, how would you describe that?

• 6:10Speaker 02

  It's great that there's never been so much interest and people and talent and resources and money in the industry. But unfortunately, that just increases the amount of pressure people have in order to compete with all the other people working on it and trying to get the value out of this technology and making money. And I think that's what just happens. Like, just a startup, you have a feeling of, you know, excitement and trying something new. And right at the beginning, you have a bit of a runway. So you have the freedom to try different things. But inevitably, people are starting to ask for returns on their investments, or they're expecting you to churn out some product.

• 6:55Speaker 02

  And this just unfortunately reduces the creativity that researchers have because, you know, the pressures to publish or the pressure to create technology that's actually useful for the products that we have goes up. And so the feeling of autonomy, I think, starts to go down. But, you know, I literally tell people when they start working for the company, I want you to work on what you think is interesting and important. And I mean it.

• 7:29Tim Scarfe?

  There is, I mean, in YouTube, there's a phenomenon called audience capture.

• 7:33Speaker 02

  Right.

• 7:34Tim Scarfe?

  And I think there might be a phenomenon called technology capture, which is that in the early days of Google, it was quite open-ended. And, I mean, transformers is now the ubiquitous backbone of all AI technology. And it's a huge achievement that you're involved in that. But, I mean, there's a similar story with OpenAI. They're now starting to see all of these commercialization opportunities. They can, I mean, they're going to become LinkedIn. They're going to become an application platform. They're going to become a search platform. They're going to become a social network. And I guess this could happen to you guys, that there's a very strong chance, especially with your new paper that we're going to talk about today, this Continuous Thought Machines. It could be a revolutionary technology, but then it will become obvious how it could be

• 8:17Tim Scarfe?

  commercialized. And that's how those pressures come in.

• 8:20Speaker 02

  I like the, I like the audience capture analogy. I think there's definitely been some kind of capture by large language models, right? They work so well that everyone wanted to work on them. And I'm really worried that we're kind of stuck in this local minimum now, right? And we sort of need to try to. Try to escape it. So we spoke about the transformers, but there's a, there's a time just before the transformers that I'd like to talk about because I think it's quite illustrative. So of course, the, the main technology before transformers was recurrent neural networks.

• 9:01Speaker 02

  Right. And there was a similar feeling, right? When recurrent neural networks came in and we, we, you know, we discovered this new sort of sequence to sequence learning. That was also a massive breakthrough, right? The, the, the, the, the translation quality went up massively. Right. Um. Voice recognition, uh, quality went up massively. And there was this, a similar sort of feeling then of like, okay, yes, we've, you know, we've found the technology and we just need to sort of perfect this technology. And back then even, my, my favorite task was, uh, character level language modeling. Right. So every time a new R N N based character level language modeling paper came out.

• 9:47Speaker 02

  I got quite excited, right? Um, I, I, you know, I'd wanna like quickly read the paper like, okay, how did they, you know, how did they get the improvements? But the papers were always these, just these slight modifications on the same architecture, right? It was L S T Ms and G R Us and maybe, um, initializing it with the identity matrix to, so that you could use the value function. Or like maybe if you put the gate in a different place, or if you, if you layered them in a slightly different way, or if you had gating going upwards as well as, as sideways. Um, and I remember one of my favorites was this, uh, like hierarchical L S T M. Where it would actually decide to compute or not compute at the different layers.

• 10:30Speaker 02

  And if you trained on Wikipedia and you looked at the structure of when it was deciding to compute or not compute. It kind of looked like the structure of, of the, the sentences were actually being picked up by the model. And I still love that sort of stuff, right? Um, but the, the, the, the improvements were always like 1.26 bits per character, 1.25 bits per character, 1.24. That was the result. That was publishable, right? That was exciting. But then after Transformer, the team that I went on to afterwards, right, we applied for the first time, very deep Transformer models, decoder only Transformer models to

• 11:12Speaker 02

  language modeling, and we immediately got something like 1.1.

• 11:18Speaker 02

  But so something that was so good that people actually come to our desk and politely tell us like, I think you you made it error, like a calculation, do you think it's nat, not bits per character? And we're like, no, no, we, you know, it really is the the the correct, the correct number. What struck me later is that all of a sudden, all of that research, and to be clear, very good research, was suddenly made completely redundant.

• 11:47Tim Scarfe?

  Yes.

• 11:47Speaker 02

  Right, all of those endless permutations to to R N Ns was suddenly seemingly a waste of time. We're kind of in the situation right now where a lot of the papers are just taking the same architecture and making these endless amount of different tweaks of like, you know, where to put the normalization layer and slightly different ways of training them. And we might be wasting the time in exactly the same way, right? Like, I personally don't think we're done, right? I don't think that this is the final architecture and we just need to keep scaling up. There's some breakthrough that will occur at some point, and then it will once again become obvious that we're kind of wasting a lot of time right now.

• 12:32Tim Scarfe?

  Yeah, so we are a victim of our own success. And this basin of attraction, there are so many basins of attraction. Sarah Hooker spoke about the hardware lottery, and this is a kind of architecture lottery. And it it it actually made me think of the agricultural revolution, which is that this kind of phase change happened, and all of the folks that had these skills that were so necessary, these diverse skills for living and surviving, they died out. And that's actually quite paradoxical, because we need those skills to take the next step. And so we we're now in this regime, we've got the term foundation model. And the implication is that you can do anything with a foundation model.

• 13:15Tim Scarfe?

  In the corporate world, we used to have data scientists, you know, there were M L engineers doing these architectural tweaks, even in, you know, midsize enterprise. And now we just have A I engineers who are just doing prompt engineering and so on. So you're saying that the fundamental skills that we need to be diverse, to think of new solutions and new architectures, they're dying out.

• 13:38Speaker 02

  I think I'm gonna disagree with that. I think the problem is we have plenty of very talented, uh, very creative researchers out there, but they're not using their talents, right? For example, you know, if you're in academia, there's pressure to publish, right? And if there's pressure to publish, you think to yourself, okay, well, I have this really cool idea, but it might not work, it might be too weird, right? It might be difficult to get it accepted because I have to sort of like sell the idea more, or I can just try this new position embedding, right?

• 14:22Speaker 02

  The problem is that the current environment, both in academia and in companies, are not actually giving people the freedom that they need to do the research that they probably want to do.

• 14:36Tim Scarfe?

  I mean, there's also this interesting thing that even in spite of great new research, I mean, I was speaking to Sep Hotwider, and he's got all of these new architectural ideas, and OpenAI aren't implementing them. I mean, Google are doing this diffusion language model, which is quite cool. And I'd like to know your opinion on why that is. So there's a few philosophies floating around, like this concept of a universal representation, that there are universal patterns and the transformer representations resemble those in the brain. And it's rather led to this idea of, well, we don't need to use different architectures because if we just have more scale and more compute, then all roads lead to Rome. So why would we bother doing it any differently?

• 15:17Speaker 02

  There's actually better, right? There is actually already architectures that have been shown in the research to work better than transformers, okay?

• 15:28Speaker 02

  But not better enough. In order to move the entire industry away from such an established architecture, where you're familiar with it, you know how to train it, you know how it works, you know how the internals work. Right? You know how to fine tune them. You have all this software is already set up for training transformers, fine tuning transformers inference. So if you want to move the industry away from that, being better is not good enough. It has to be obviously crushingly better. Transformers were that much better over R N Ns. Okay, transformers where you just applied it to a new problem.

• 16:09Speaker 02

  And it just was so much faster to train and you just got such higher accuracy. That you just had to move. And I think the deep, the deep learning revolution is also another example of that. Right? Where you had plenty of skeptics. And people were pushing neural networks even back then, and people are going, no, we think symbolic stuff will work better. But then they demonstrated it as being so much better that you couldn't ignore it. This fact makes finding the next thing even harder. Right? That's the gravitational pull of always pulling you back to. Okay, but a transformer is good enough. And yeah, you made a cool little architecture over here that yeah, it looks like it's it's.

• 16:53Speaker 02

  Got better accuracy, but OpenAI over here just made it 10 times bigger and it beats that. So let's just keep going.

• 16:58Tim Scarfe?

  May I also submit that there could be an additional reason? Which is, you know, I love that fractured and tangled representations paper. And there's, there's this shortcut learning problem. And I think that there's a little bit of a mirage going on here. And there, there might be problems with these language models that we don't, you know, that we're not fully aware of. And there's also this thing that we're seeing that we are starting to bastardize the architecture. So we know we need to have adaptive computation for reasoning. We know we want things like uncertainty quantification. And what we're doing is, is we're bolting these things on top. Rather than having an architecture which intrinsically does all of these things that we know we need.

• 17:38Speaker 02

  Yeah, and I, and I think the, the, our continuous thought machine is, is an attempt at addressing those more directly, right, which, which Luke will be able to tell you more about later. There's something still not quite right with the current technology, right? I, I think the, the phrase that's becoming popular is jagged intelligence, right? That the fact that you can ask an L L M something and it can solve literally like a PhD level problem. And then, you know, in the next sentence, it can say something just so clearly, obviously wrong that it's, it's, it's jarring. Right? So and I think this is actually a reflection of something

• 18:19Speaker 02

  probably quite fundamentally wrong with the current architecture, as amazing as they are. The current technology is actually too good, okay?

• 18:32Speaker 02

  Another reason why it's, it's difficult to move away from them, right? So that they're too good in, in, in the following sense. And you, you spoke about the fact that we have these foundation models that's okay, so that we have the foundation, we can do anything with them. Yes, I think current neural networks are so powerful that if you have enough patience and enough compute and enough data, you can make them do anything. But I don't necessarily think that they want to, right? We're sort of forcing them by their universal approximators. But I think there are probably a space of, you know, function approximators

• 19:14Speaker 02

  that will more want to represent things in the way that a human represents them. So there's actually quite an obscure paper that is my poster child for this. It's called Intelligence Matrix Exponentiation. And I think it was actually rejected. So, you know, you can probably project the image of a figure one, but there's an image of it solving, you know, the, the classical spiral data set of needing to separate the two classes in the spiral.

• 19:45Speaker 02

  And it has the decision boundary for both a classic RNN, multi-layer perceptron, and a tanh multi-layer perceptron. And you can see, they both solve it, right? Technically, they both solve the problem because they classify all the points correctly and get a very good test score on this very simple dataset. And then they show you the decision boundary for the M-layer that they built in this paper. And it's a spiral. The layer represented the spiral as a spiral. Shouldn't we? You know, if the data is a spiral, shouldn't we represent it as a spiral?

• 20:27Speaker 02

  And then if you look back at the decision boundaries for the spiral in the classic ReLU multi-layer perceptron, it's clear that you just have these tiny little piece-wise linear separations. And that's what I mean. Yes. If, you know, if you train these things enough and you push these little piece-wise linear boundaries around enough, it can fit the spiral and get a high accuracy. But there's no feeling, when I look at that image, that the ReLU version actually understands that it is a spiral, right?

• 21:12Speaker 02

  And when you represent it as a spiral, it actually extrapolates correctly because the spiral just keeps going out.

• 21:18Tim Scarfe?

  You're touching on something fascinating there because, you know, we were talking about the need for adaptivity and adaptive computation. I'm really inspired by Randall Bellistria's spline theory of neural networks, and we've had them on many times. And you can look on the TensorFlow playground, you can look what happens when you have a ReLU network on this spiral manifold. And, you know, you'd be forgiven for thinking that these things are basically a locality-sensitive hashing table, right? Because they do, they partition the space and they can predict the spiral manifold, right? But we want to do something a little bit more different than that. And it also comes into this imposters thing because just tracing the spiral manifold...

• 22:04Tim Scarfe?

  But not continuing the pattern, there's a big difference between that. So from an imposter perspective, just just tracing the pattern is not learning it abstractly or constructively, right? If we learned it constructively, so you know, you speak about this in your paper, this complexification, the abstract building blocks, and you can do adaptive computation, you understand the spiral. That means that with adaptive computation, you can continue the spiral, and then you can update the model's weight so it has adaptivity, because that's so important for intelligence. So we know that we need models that can do these things. But for some reason, they're so sycophantic, they're almost better than an adaptive intelligent

• 22:47Tim Scarfe?

  system, because they tell us exactly what we want to hear. They seem so intelligent, but we know that they're missing these fundamental properties.

• 22:55Speaker 02

  I'm still fairly skeptical when I see video generation models. You know, we went through a phase where you could detect them because of the number of fingers on somebody's hand, right? And yes, with more data, with more compute, with better training tricks, okay, they submit, and now they usually do have five fingers. But did we fix the problem? Or did we just use more brute force to just, you know, force the neural network to know it's five fingers? Where something that actually had a much better kind of representation space, it's

• 23:39Speaker 02

  almost mad that it's controversial to say that we should represent a spiral like a spiral. But, you know, something that could do that generally, that if it represented a human hand the way that, you know, maybe I represent a human hand, then maybe it would be much easier to count how many fingers are on a hand. It's unfortunate that they work so well. It's unfortunate the scaling works so well, because it's too easy for people to just... Sweep these problems under the carpets.

• 24:06Tim Scarfe?

  You guys have possibly created what I think might be the best paper of of the year. This could actually be the innovation which takes us to the next step. And you did. Did you get the spotlight at NURIPS as well?

• 24:18Speaker 02

  Yeah.

• 24:18Tim Scarfe?

  This year and congratulations on that. So I think that's testament to how amazing this paper is.

• 24:23Speaker 02

  The C T M, the Continuous Thought Machine, it's actually not that far outside of the local minimum that we're stuck in. Right. It's not as if we went and found this completely new technology. Right. We took quite a um a simple biologically inspired idea. Right. So these of the fact that neurons synchronize. And not even necessarily in a biologically plausible way, right? Brains don't literally have all their neurons wired together in a way that they work out their synchronization.

• 24:58Speaker 02

  But it's. It's the sort of research that I want to encourage people to do. And the way to sell it is quite easy, I think. At no point did we have to worry about being scooped. Right. That stress was taken away from us completely. So and there was so there was no pressure to sort of rush hour with this with this idea. Because like, well, there's probably somebody else working on exactly this. And I think the reason that we, you know, we were able to get a spotlight is because we were able to create such a polished paper. We took the time to do the science properly, to get the the base, the baselines that we wanted. And do all the the the tasks that we wanted to try.

• 25:43Speaker 02

  Encouraging researchers to take a little bit more of a of a risk. Right. To try these slightly more speculative, long term ideas. I think is is the sad thing is I don't think it's necessarily a very difficult thing to sell. And I want to have the C T M as like a poster child of. It works, right? It was a bit of a risk. We didn't know if we were going to find something interesting. But, you know, it was our first shot and we did find something interesting and it became a successful paper.

• 26:13Tim Scarfe?

  If we do find a system which can acquire knowledge, design new architectures, do the the open-ended type of science that you're speaking to, can you see a future where at some point the locus of progress will be mostly driven by the models themselves?

• 26:30Speaker 02

  I think so. Whether or not that's going to replace us completely, I go back and forth on. Powerful algorithms are finding, uh, helping us do research, right? And I think it might just end up being a more powerful version of that. I know the the A I scientists that we we released, we showed that you could actually go end to end, right? Go from seeding the system with an idea for a research paper and then just take your hands off and just let it go. Think about the idea, write the code, run the code, collect the results and write the paper. Uh, to the point that we were actually able to get it to um

• 27:12Speaker 02

  a one hundred percent A I generated paper accepted to a workshop recently, right? But I think we did that to show that you could do it, right? As a sort of demonstration. In a real system, I think I would want it to be much more interactive. Right? I would want to be able to see it with an idea and then have it come back with more ideas. Have a discussion with me, then go away to write the code. I want to look at the code and check it and then discuss the results as they're coming out. So that's the sort of near term future that I that I would envision of how I would like to do. Research with an AI.

• 27:55Tim Scarfe?

  And could you introspect on that? Is it because you feel we need supervision because the models don't yet understand? You know, there's this path dependence idea. So we need to do supervision because we have the path dependence so we can guide the generation of the language models. Maybe in the future, the language models will just understand better themselves. But there's also the output dimension, which is that we want to produce artifacts that extend the phylogeny of human interest. We want it to be human relevant.

• 28:22Speaker 02

  Yeah, I I think it's more that, you know, in that initial seed idea, it's probably impossible to actually describe exactly what you want. It's exactly the same with, you know, when I have an intern. I can't just have an intern come into the company and I go, I have this mad idea and then just explain it to them and then just leave them alone for four months. There's a back and forth because I have a particular idea that I want to explore and I need to keep steering them in the direction that's I I, you know, that I had in my my mind originally. So I think it's more like that, basically.

• 28:57Tim Scarfe?

  You have such a deep understanding. So you have this rich provenance and history and path dependence. And that means you can take creative steps, intuitive steps for you, respect the phylogeny. They respect all of this deep abstract understanding that you have and interns don't yet have that.

• 29:13Speaker 02

  Right.

• 29:13Tim Scarfe?

  But maybe AI models in the future will have that.

• 29:16Speaker 02

  Yeah, sure. If they if they get to the point where my inputs becomes detrimental, then, yeah, that'll be a thing. It's it's kind of like chess, right? There was a point at which chess engine and human fusion actually beats chess engines. That's not, that's not true anymore, right? Adding a human into the mix actually makes the the bots worse.

• 29:41Tim Scarfe?

  Oh, interesting. I wasn't aware of that.

• 29:42Speaker 02

  Yeah. So so what to do when that day comes for AI scientists is a is a is a broader discussion, I think.

• 29:50Tim Scarfe?

  I think now is a good segue to talk about this paper in a little bit more detail. So this continuous thought machines, you were just pointing to it before. Luke, first of all, first of all, mate, introduce yourself and set this thing up for us.

• 30:04Speaker 03

  My name's Luke. I am a research scientist at Sakana AI. And my primary sector of research is this continuous thought machines. It took us somewhere in the region of about eight months, working on this project with the whole team. I did a lot of the work, but we also had a lot of people in different areas and doing different parts of it. That I think an eight month lifecycle for a paper seems a bit long for AI research at the moment. But yes, to the to the actual technical points of the paper. So we call it continuous thought machines. It originally had a different name. We called it asynchronous thought machines before. But every single time people asked us, what's the asynchronous part, it became a bit confusing.

• 30:48Speaker 03

  So continuous thought machines basically depends on three novelties. The first one is having what we call a internal thoughts dimension. And this is not necessarily something new. It's related conceptually to the ideas of latent reasoning. And it's essentially applying compute in a sequential dimension. And when you start thinking about ideas and problems in this domain and in this framework, you start understanding that many problems that just look like intel or solutions to problems that look intelligent are often solutions that have a sequential nature. So for instance, one of the primary tasks that we tested in the continuous thought machines was this maze solving task.

• 31:30Speaker 03

  And solving mazes for deep learning is quite trivial. It's really easy to do if you make the task easy for machines. And one of the ways to do this is you give an image of a maze to a neural network, like a convolutional neural network, and it outputs a image, same size of the maze. And it's zeros where there isn't a path and there's ones where there is a path. There's some really brilliant work showing how you can train these in a careful way and scale them up essentially indefinitely. And this is fascinating and a really interesting idea of how to solve this. However. When you take that approach out of the picture and you ask, what is a more human way to solve this problem?

• 32:15Speaker 03

  It becomes a sequential problem. You have to say, well, go up, go right, go up, go left, whatever the case may be, to trace a route from start to finish. And when you constrain that simple problem space and you you ask a machine learning system to solve it like that, turns out you actually get much, much more challenging. So this became our hello world problem for the C T L. And applying an internal sequential thought dimension to this is how we went about solving this. Two other novelties that we can touch on and talk about, we we sort of rethought the idea of what neurons should be. There is a lot of excellent research in this world in cognitive neuroscience, particularly

• 32:59Speaker 03

  exploring how neurons work in biological systems. And then we get on the other side of the scale, how deep learning neurons work, which the quintessential example is a ReLU. It's off or on, in a sense. And this very, very high level abstraction of neurons in the brains feels a little bit myopic. So we approached this problem and said, well, let's let's on a neuron by neuron basis, let this neuron be a little model itself. And this ended up doing a lot of interesting work on how to build dynamics in the system. The third novelty here is, as I said before, we have this internal dimension over which thinking happens. We asked the question, well, what is the representation?

• 33:41Speaker 03

  What is the representation for a biological system when it's thinking? Is it just the state of the neurons at any given time? Does that capture a thought? If you wish, if I can be controversial and use the term thinking and thought. And my philosophy with this is no, it doesn't. That the concept of a thought is something that exists over time. So how do we capture that in engineering speak? We, instead of measuring the states of the model that is recurrent, we measure how it synchronizes, how neurons synchronize in pairs along with other neurons. And this opens up the door to a huge array of things that we can do with this type of representation.

• 34:22Tim Scarfe?

  You were talking about this sort of sequential nature of reasoning. And Devil's Advocate, I mean, there was that anthropic biology paper. And they were talking about planning and thinking. And they were saying that this thing is planning ahead. Because I think your system, actually, we can say it does planning. It's actually different computation. Can you explain that?

• 34:46Speaker 03

  Yes, I think the boundary in terms of computation from a Turing machine perspective, if you wish, is really interesting because the notion of being able to write your tape, read from that tape and then write again to be in a Turing compute system, Turing compute system is obviously an incredible idea that has completely changed the world. And I think the primary difference with, let's talk about transformers versus what we're trying to do with the C T M, is that the process that the C T M thinks in, we can apply that process, that internal process to breaking down a problem. So the problem itself can be a single, there is a single solution to this problem. And you could do that in one shot.

• 35:31Speaker 03

  You could, as I explained with the maze, you could just process that in one shot. But there are certain phrasings of problems that are real problems that doing so becomes exponentially more challenging. So in the maze task, a really good example is that if you try to predict a hundred, two hundred steps down the path in one shot, no models that we could train, not even our model could do that. And we needed to actually build an auto curriculum system where the model first predicted the first step. And then when it could predict the first step, then we started training it on the second and third and fourth step. And the sort of resultant behavior of this is where it gets interesting. One of the One of the ways that I like to do research and that I encourage people who work with me to do

• 36:13Speaker 03

  research is understand, if you wish, the behavior of a model. We are getting to a point now where the models that we build are demonstrably intelligent in ways that keep surprising us. And breaking that down into a single set of metrics, or even a finite, single metric about performance, seems maybe not to be the right way to do it for me. And understanding the behavior and the actions that those models take, when you put them in a system and train them in a certain way, seems to reveal more about what's actually going on under the hood.

• 36:47Tim Scarfe?

  Very cool. And I think I didn't pick up on this. So you're doing a fixed number of steps. So you have like a context window. And did you say that you've set that around 100 steps?

• 36:59Speaker 03

  So for the maze task, the model always observes the full image at every step. The C T M will observe the full image. For argument's sake, those images could be tokens from a language, the output of a language model. Those inputs could be numbers that the model has to sort, whatever the case may be. It should be agnostic to data. That's how we've tried to build it. But in the maze task, the model can continuously just observe the data. No matter where, it can look at the whole image simultaneously, but it uses attention to retrieve information from the data. And it has, let's call it 100 steps, that it can think through. And what we do is we pick up, at some point, the model solves three steps through the maze. So it says, I'm going to go up, up and right.

• 37:44Speaker 03

  And then it's correct, but then it makes the wrong turn. And at that point, we stop supervision. We only train it to solve the fourth step. So one more than what it could. In practice, we do it five, but the principle holds. And when you do that, it's a self bootstrapping mechanism. And I think the intuitive listener will understand how that extends to other domains, other sequential domains, for instance, like language prediction, many tokens ahead, that sort of thing.

• 38:11Tim Scarfe?

  So I'm really interested in this idea of adaptive computation. So I guess the first question is, how sensitive was the performance to the number of steps? And then the next question would be, could you have an arbitrary number of steps? Which means that, you know, perhaps based on uncertainty or, you know, some kind of criterion, you could do fewer steps. And then the final question is, could you have potentially like an arbitrary or unbounded number of steps?

• 38:38Speaker 03

  Yeah, a really super question. I think that I think that I'll answer the uncertainty question first about the sensitivity to steps. So a very good example of this is we just trained the model on ImageNet classification. And our loss function is quite simple. What we do is we run it for, for example, fifty steps. And we pick up two points, two distinct points. The first one is where is it performing the best, i.e. where is the loss the lowest? And the second one is where is it most sure, where is it most certain? And those give us two indices between zero and forty-nine inclusive. And we apply cross entropy at both of those points. We just make the loss the average of the cross entropy at those points.

• 39:19Speaker 03

  So what this does is it induces a behavior where easy examples are solved almost immediately in one or two steps. Whereas more challenging examples will naturally take more thinking. And it enables the model to use the full breadth of time that it has available to it, just in a natural fashion, without having to force it to happen.

• 39:37Tim Scarfe?

  So you've decided to model every neuron as an MLP, which is really fascinating, talk about that. But also, there's this notion of synchronization. And I think you use the inner product to determine the extent to which the parameters are synchronized. And this kind of unfurls over time as the driving force. Can you explain that in a bit more detail?

• 39:56Speaker 03

  Absolutely. I think it's a it's a good point to explain the neuron level models, as we call them in the paper, or NLMs first, because it ties into this. So you can imagine a recurrent system is a state vector, a state vector that is being updated from step to step. We track that state vector and that state vector unfolds. And for each individual neuron, each i neuron in the system, we have a unfolding time series. It's a continuous time series. Well, it's discrete, but it's a continuous value. And those time series define what we call the activations over time. And synchronization is quite simply just measuring the dot product between two of these time series.

• 40:40Speaker 03

  So you have a system of D neurons and essentially you have D over two squared, different synchronization pairs. So neuron one can be related to neuron two by how they synchronize. And neuron one can also be related to neuron three, et cetera, et cetera. The neuron level models, they function by taking in a finite history, like a FIFO queue of neuron of activations coming in. And instead of being just a ReLU activation, they use that history as information to process a single activation out. And that is what moves from what we call pre activations to post activations. And the principle here is that this might seem rather arbitrary. And does it help for performance?

• 41:23Speaker 03

  Turns out it does, but that's not really the catchall solution here. That's not what we're after. What we're after here is trying to do something biologically plausible.

• 41:33Speaker 03

  Find the line somewhere between biology, which is how the brain implements things in the biological substrate that we have versus deep learning, which is highly parallelizable, super fast to learn, backprop amenable, all of the nice properties of that that have got us this far. And find a line somewhere where we can take some sprinkling of biological inspiration, but still train it with deep learning. And it turns out that neuron level models is a nice interim that we can do this with. The concept of synchronization is applied on top of the outputs of those neuron level models.

• 42:03Tim Scarfe?

  So on this, on the scaling, I think the time complexity is quadratic in respect of the dimension of the synchronization matrix. Right. And in your paper, you were talking about sub sampling to improve the performance. But how did that affect the stability? And the, you know, like, were there any things that cost you doing that?

• 42:23Speaker 03

  Yeah, it's a neat question. I think in terms of stability, what's, what we found was kind of fun. And this was a sentiment that we had throughout the. Experiments that we ran with this paper was it tended, no matter what we tried it on, it it just kind of worked with all spreads of hyperparameters. And the problems that you have with backprop through time, typically with recurrence models like R N Ns and L S T Ms, it's a challenge. And you run for many internal ticks with the R N Ns or the L S T Ms and the learning seems to break down. But the fact that we use synchronization, in some sense, touches all of the neurons through all of the time. So it really helps with gradient propagation. A nice, interesting point that's maybe a bit oblique to what you asked about synchronization

• 43:04Speaker 03

  is we have a system of D neurons and like I said earlier, there are D over two squared possible combinations. This essentially means that our underlying state or underlying representation to the system is quite a lot larger than what you would get with just taking those D neurons. And as to what that means in terms of downstream computation and performance and the things that we can do with this is what we're actively exploring right now.

• 43:28Tim Scarfe?

  You guys use an exponential decay rate?

• 43:31Speaker 03

  You have the system that unfolds over time. It would be maybe a little bit too constrained if the synchronization between any two neurons depended on the same time scale. So for instance, there are neurons in your brain that are firing over very long time scales and very short time scales. The way that they fire together impacts other neurons and causes those neurons to fire. But everything in biological brains happens at diverse time scales. It's why we have different brain ways for different thinking states, for instance. But beside that point, what we do with the exponential decay in the continuous thought machines is it allows us for a very sharp decay to say that these two neurons that

• 44:12Speaker 03

  are pairing together, what only really matters is how they fire together right now. Right? But if we had a very long and slow decay, essentially that's capturing a global sense of how those neurons are firing over an extremely long period of time. So this was essentially a way of us capturing this idea of how different neurons could maybe fire together very quickly and other neurons can fire together very slowly or not at all. And this lets that representation space that I spoke about, that D over two squared representation space, lets it again become more rich and we can enrich that space with more subtle tweaks to how we compute those representations.

• 44:50Tim Scarfe?

  So we were speaking about this yesterday, Luke, that when folks apply transformers to things like the ARC challenge or things that need reasoning, we need to do lots of domain-specific hacks. So the architects who were the winners of last year's challenge, they did depth-first search sampling, and some folks have been experimenting with using language representations or, you know, using DSLs. And some part of this is to do with the reachability of language, right? And language is quite dense, which means you can kind of monotonically increase. But if I understand correctly, your system might have some interesting properties for reasoning and for discrete and sparse domains, and also for sample efficiency. Because we want to build

• 45:32Tim Scarfe?

  a system that can actually do well on things like the ARC challenge. But can you kind of explain in simple terms why you think this architecture could be significantly better than transformers for doing those things?

• 45:44Speaker 03

  I think a lot of the really fascinating work in the last few years that I found fascinating in the literature of language models has been related to what one can actually call a new scaling dimension. I, in some sense, see a chain of thought reasoning as a way of adding more compute to a system. That's obviously just one small part of what that really is and what that really means. But I think it's quite a profound breakthrough in some sense. Now, what we're trying to do is have that reasoning component be entirely internal, yet still running in some sort of sequential manner. And I think that that's rather important. And you spoke earlier about Geminized Diffusion language modeling. And I think that there are

• 46:29Speaker 03

  a lot of different directions that are exploring this right now. I do think that the continuous thought machine with the ideas of synchronization and multi-hierarchical temporal representations gives a certain flexibility on that space that other people are not yet exploring, and that richness of that space, being able to project the next step to solve the ARC challenge, and the next 100, the next 200 steps, to be able to break that down into a process that a model can then very quickly search that process in its high-dimensional latent space, becomes something that feels like a good approach to take.

• 47:05Tim Scarfe?

  Do you see any relationship between this architecture and, you know, Alex Graves' neural Turing machine?

• 47:11Speaker 03

  Yes, that's really interesting. I do, I think that the, one of the, the most challenging parts about working with a neural, neural Turing machine is the concept of writing to memory and reading to memory because it is a discrete action. And that's, that has its own challenges associated with it. And, yes, I wouldn't go so far as to say that the continuous thought machine is a definitive in your Turing complete, but the notion of, the notion of doing reasoning in a space that is latent and letting that space unfold in a way that is rich towards a different set of tasks.

• 47:53Speaker 03

  And this, this actually brings me to a, a point that I find quite interesting, that I'd like to share with you. Consider again the ImageNet task or any sort of classification task. It's a, it's a nice test bed. There are many images that are really easy and there are many images that are really difficult. When we train, for instance, a VIT or a CNN to do this task, it has to nest all of that reasoning in the same space. It has to put all of its decision-making process for a very simple, obvious cat versus some complex, weird, underrepresented class in that system, in that data set. And it has to nest it all in parallel in a way that is, we get to the last layer and then we classify.

• 48:38Speaker 03

  I, I think breaking that down, where you have different points in time, where you can say, now I'm done, I can stop, versus now I'm done, I can stop, lets you take a data set or take a task and actually naturally segment it into its easy to difficult components. And I think we know that curriculum learning and learning in this continuous sense, again, seems to be a good idea. It's how humans learn. And if we can get at that architecturally and just have that fallout in a model, again, this seems like a, something worth exploring. I'm not sure if you know much about model calibration and how neural networks tend to be poorly calibrated.

• 49:16Tim Scarfe?

  Well, go for it, Tommy.

• 49:18Speaker 03

  It's, it's a bit of an old finding, but if you train a neural network for long enough and it fits really, really well, and you've regularized it, regularized it really, really well, you'll find that the model is uncalibrated, which essentially means that it is very certain about. Some components where some classes where it's wrong and uncertain for some classes where it's correct. Essentially, what you want for a perfectly calibrated model is if it predicts a probability that this is in class, the correct class with fifty percent, fifty percent of the time, you want it to be correct about that class and so on and so forth. So well calibrated model, if it's predicting a probability of zero point nine that it is a cat, then ninety percent of the time it should be correct.

• 50:01Speaker 03

  And it's actually turns out that most models that you train for long enough get poorly calibrated. And there are loads of post hoc tricks to fixing this. We measured the calibration of the CTM after training and it was nearly perfectly calibrated, which is again, a little bit of a smoking gun that this actually seems to be probably a better way to do things.

• 50:21Speaker 00

  Hey Spotify listeners. When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 50:47Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a fifty-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses setup required. Compatibility and availability varies. Eighteen plus.

• 51:13Speaker 02

  The flavor of this kind of research is such that we didn't actually go out and actually try to create a very well calibrated model, right? And we didn't even try to create a model that was necessarily going to be able to do some kind of adaptive computation time, right? I was a very big fan of the, the paper, yeah, adaptive computation time was Alex Graves, was it? But that paper, it had a massive amount of hyperparameter sweeps in it because in that paper, he needed to have a loss on the amount of computation that was being done. Yeah.

• 51:55Speaker 02

  Anytime you try to do some sort of adaptive computation time research, what you're fighting is the fact that neural networks are greedy. Right? Because obviously, the way to get the lowest loss is to use all the computation that you have access to. So unless you had like an extra loss that had a penalty that said, okay, actually, you're not allowed to use all the computation. That's and and very, very carefully balanced loss. That's when you actually got the interesting dynamic computation time behavior falling out of the the model in that paper. What was really gratifying to see with the the continuous thought machine

• 52:37Speaker 02

  is that because of the way that we set up the loss that Luke described earlier, adaptive computation times seem to just fall out naturally. So that's more the way that I think research should go. Okay, because we don't actually have like a specific goal.

• 52:59Speaker 02

  Or like a specific problem we're trying to fix like that, or something we're trying to invent. It's more that we have this interesting architecture and that we're just following the gradients of interestingness.

• 53:11Tim Scarfe?

  Yes, and on on that point, I think maybe the most exciting thing about your paper is, you know, we were talking about path dependence and having this understanding, which is built step by step, this process of complexification. And I mean, maybe this is this is apropos in in the theme of world models in in general. And also active inference, and I say active inference in big quotes because it's not Carl Friston's active, you know, maybe adaptive inference or something like that. But we want to build agents that can continue to learn, that can update their parameters. And most importantly, can construct path dependent understanding. And because that's completely different to just understanding what the thing is. It's how you got there is very important.

• 53:56Tim Scarfe?

  And this architecture potentially allows these agents using this algorithm to explore trajectories in spaces, find the best trajectories and actually construct an understanding which carves the world up by the joints.

• 54:10Speaker 03

  Yeah, that's a that's a really neat perspective. I haven't actually thought about it like that. But yes, I think that particular stance becomes really interesting when you.

• 54:20Speaker 03

  Think about ambiguous problems. Because carving the world up in one way is as performant as carving it up in another way.

• 54:29Tim Scarfe?

  Yeah.

• 54:29Speaker 03

  Uh, you know, perhaps the hallucination in language models is carving the world up in some fine way, but it's just not performance in our measure of this is hallucination. And actually, that's not true. Uh, but in some other trace down the path of wanting to carve the world up through a auto regressive generation of tokens, you end up in a different carve up of that world. And being able to train a model that can be implicitly aware of the fact that it is actually carving up the world in a different way and can explore those manners, those uh descents down the carve up is something that we're after. And I think it's quite an exciting approach to be trying to take a stance of let's, let's

• 55:13Speaker 03

  break up this problem into small solvable parts and learn to do it like that. And how can we do this in a natural way without too many hacks?

• 55:22Tim Scarfe?

  Yeah, it's something I've been thinking about because um Chollet, as much as I love his measure of intelligence um ideas, is for him, adapting to novelty is getting the right answer. And the reason why you gave that answer is very, very important. And in machine learning, we have this problem that we, we come up with this kind of cost function that rather leads to this shortcut problem. But, you know, we could just build a symbolic system where we could be Go5-pilled and, and we could say, okay, we need to do this um principled kind of construction of knowledge, maintaining semantics. Well, we're not doing that. We're doing a hybrid system. But there must be some natural way of doing reasoning where in spite of the end objective

• 56:06Tim Scarfe?

  being this cost function, that because of the way that we traversed these open-ended spaces, that we can actually have more confidence mechanistically that we're doing reasoning which is aligned to the world.

• 56:19Speaker 03

  I think that's a great way of seeing this particular. Uh avenue of research. And I think that obviously we're not the only people thinking like this. And we're not the only ones trying to do this. What we have is an architecture that's amenable to it. And surprisingly so, it wasn't, again, wasn't the goal. It's not the goal to to do this type of research. It's not the goal to be able to break the world down into these small chunks that we can actually reason over in in a way that seems natural. Instead, what we did was pay respect to the brain, pay respect to nature and say, well, if we build these inspired things, what what actually happens?

• 57:00Speaker 03

  What what different ways of approaching a problem emerge? And then when those different ways of approaching a problem emerge, what big philosophical and intelligence based questions can we then start to ask? And that's where we're at right now. So it might feel at times, especially for me, uh too many questions and too few hands to answer those questions. But I think the fun and exciting thing and the encouraging thing that I I can, you know, try to encourage other younger researchers out there is that,

• 57:32Speaker 03

  you know, do what you're passionate passionate about and figure out how to build the things that you care about. And then see what that does, see what doors that opens up and see how to explore deeper into those domains.

• 57:45Tim Scarfe?

  We were talking about this yesterday, weren't we? That you can think of language as being a kind of maze.

• 57:51Speaker 03

  Yes.

• 57:51Tim Scarfe?

  Like what is to stop us from taking this architecture and building the next generation language model with it?

• 57:58Speaker 03

  I mean, that that's honestly, as you know, something that I am actively trying to explore right now. And yeah, I think the maze, the maze task gets really interesting when you add ambiguity to it. When there are many ways to solve the maze. Honestly, this isn't something I've tried yet, and maybe it's something I should try next week. But it's essentially, you can imagine a an agent or the C T M in this case, observing the maze and taking a trajectory and surprisingly, we saw this. We have a section in our recently updated paper on archive, the final camera ready version of this paper, where we added an extra supplementary section that is not in the main technical report. And that supplementary section is basically, hey, we saw this cool stuff happen.

• 58:40Speaker 03

  And we list, I think, fourteen different interesting things that happened while we were doing the research that obviously didn't make it into the paper, but we wanted people to know about these strange things that happened. And this is one of the strange things, where we watched during training what was happening. And at some time during training, maybe halfway through the training run, we could see what the model would do is it would start going one path in the maze. And then suddenly it would realize, oh, no, damn, I'm wrong, and would backtrack and then take another path. But eventually it gets really good, and it does some sort of distribute, say, learning in this because it's got a attention mechanism with multiple heads, so it can actually just figure out

• 59:20Speaker 03

  how to do this pretty well and refine its solution. But sometime early on in the learning, it descends multiple paths and comes back and backtracks. We have a really fascinating set of experiments that also showed, and this, this, we actually have some supplementary material online showing this where, and I don't really know what this says, it's kind of a deep philosophical thing, but if you're trying to solve a maze, but you don't have enough time, turns out that there's a, there's a faster algorithm to do it. And this was, this blew my mind when I saw it. So if we constrain the amount of thinking time that the model has, but still get it to try solve a long maze, instead of tracing out that maze, what it does is it quickly jumps ahead to approximately where it needs to be, and

• 1:00:03Speaker 03

  it traces backwards, and it fills in that path backwards. And then it jumps forward again, leapfrogs over the top and traces that section backwards, then leapfrogs. And it does this fascinating leapfrogging behavior that is based on the constraint of the system. And again, you know, this is just an observation we made and what that means in a deep sense, and how it's related to giving a model time to think versus not. And is it enough time to think? What happens? What different algorithms does the model learn when you constrain it in this way? I find that quite fascinating and an interesting thing to explore. Does it tell us something about how humans think? Does it tell us something about how how we think under constrained settings versus open ended settings? So it's a number of cool questions you can ask on this front.

• 1:00:48Tim Scarfe?

  You you guys are both huge fans of, you know, population methods and collective intelligence. And because we can, we can scale this thing up and we can scale it out. And what would it mean to scale this thing out? Not only just in a kind of, what do they call it? Trivial paralization, but in terms of having some kind of weight sharing between parallel models and so on. What what what would what would that give you potentially?

• 1:01:12Speaker 03

  This is, this is a fun area of research. So one of the active things that we're trying to explore in our team is concepts of memory, long term memory, and what what does this mean for a system like this? So an experiment that one can construct, for instance, is to put some agents in a maze and let them try solve this maze, not not how we did it in the paper, but in a very constrained setting where a agent can only see maybe a five by five region around it. And we give that agents some mechanism for saving and retrieving memories. And the task, if you wish, is to solve that maze, find your way to the end. And the model needs to learn how to construct memories such that it can

• 1:01:54Speaker 03

  get back to a point where it's seen before and know I did the wrong thing last time and go a different route. And you can then see this with parallel agents in the same maze with a shared memory structure and see what actually happens when you can all access that memory structure and have a shared global, like almost like a cultural memory that we can access and solve this global task by having many agents trying to use this memory system. And I do think that memory is going to be a very key element to what we need to do in the future for AI in general.

• 1:02:27Speaker 02

  So the subject of reasoning came up just a second ago. And I think there's a perception that recently we made a lot of progress in reasoning, right? Because it's actually one of the main things that I think people are are working on. We released a data set recently called Sudoku Bench. And I was actually quite happy to see it come up organically on your podcast a few weeks ago.

• 1:02:55Tim Scarfe?

  Chris Moore.

• 1:02:56Speaker 02

  Right.

• 1:02:56Tim Scarfe?

  Yes.

• 1:02:57Speaker 02

  So.

• 1:02:59Speaker 02

  I wanted to tell you a little bit about this benchmark because

• 1:03:04Speaker 02

  I think I've been having a little bit of issue promoting it because it doesn't on the surface sound particularly interesting. Because Sudoku has a sort of a feeling that it's already been solved, right? So how interesting can a collection of Sudoku's be for reasoning exactly? We're not talking about normal Sudoku's. We're talking about variant Sudoku's. And what variants Sudoku's are, are usually normal Sudoku's, right? So put the numbers one to nine in the row of the column in the box.

• 1:03:42Speaker 02

  But then literally any additional rules on top of that. And they're all handcrafted. They all have extremely different constraints.

• 1:03:57Speaker 02

  Constraints that actually require very strong natural language understanding. So for example, there's one puzzle in the data set where it tells you the constraints of the puzzle in natural language. And then says, oh, by the way, one of the numbers in that description is wrong. Right? So you have to be able to met reason about the rules themselves even before you start solving the puzzle. There are other puzzles where you have a maze overlaid on the Sudoku and the rat has to work out a way through the maze by following

• 1:04:38Speaker 02

  a path to the cheese, but then there are constraints on the path that it takes of like what numbers and what they can be add up to.

• 1:04:47Speaker 02

  It's difficult to really describe how varied these, these, these variants Sudoku's are. And I think they're so varied that if anyone was actually be able to beat our benchmark, they would necessarily have to have created an extremely powerful reasoning system.

• 1:05:12Speaker 02

  Right now, the best models

• 1:05:17Speaker 02

  get around fifteen percent, but they're only the very, very simplest and the very smallest Sudoku puzzles in the set. We're going to be putting out a blog post about GPT-5's performance, and it is a jump. But it's still completely unable to solve puzzles which are, you know, humans can solve. And what I really like about this dataset and actually was the catalyst for me creating it in the first place. It was that there was a there was a quote by Andre Capathie saying, okay, so we have all this data, it's from the internet.

• 1:06:03Speaker 02

  But what you really want, right, if you wanted AGI, you wouldn't want all of the text that humans have ever created. You would actually want the thought traces in their head as they were creating the text. Right? If you could actually learn from that, then you would get something really powerful. And I thought to myself, well, that data must exist somewhere.

• 1:06:30Speaker 02

  My first thought was maybe

• 1:06:34Speaker 02

  philosophy, like, you know, there's a type of philosophy where you just write down your thoughts without thinking, like just dream of consciousness. I thought, hmm, maybe that could work. But then, when I wasn't thinking about it, and I was, you know, in my leisure time, I was watching a YouTube channel called Cracking the Cryptic. Yes. Where these two British gentlemen will solve these extremely difficult Sudoku puzzles for you. Like, sometimes their videos are four hours long and they're professionals, like this is their job. And what was perfect, I realized, is they tell you in agonizing detail exactly what reasoning they're used.

• 1:07:20Speaker 02

  To solve those particular puzzles. Right, so we, with their permission, took all of their videos, which represents thousands of hours of very high quality human reasoning, like thought traces.

• 1:07:36Speaker 02

  And scraped them and made that available for imitation learning. Right.

• 1:07:45Speaker 02

  We did try to do this internally. Turns out that I did a little bit too much of a good job of really creating a very difficult benchmark. Right, so we're still trying to get that stuff working. We published it that if we have some success.

• 1:08:01Speaker 02

  Yeah, I wanna, I wanna really sell the fact that this, this reasoning benchmark really is different, right? Not only do you get something that's super grounded, like you know exactly if it's right or wrong, so you can do RL to your heart's consent.

• 1:08:16Speaker 02

  But you can't generalize very easily. Each puzzle is deliberately designed by hand to have a new and unique twist on the rules called a break-in that you have to understand. And right now, despite all the progress we've made.

• 1:08:40Speaker 02

  The current AI models can't take that leap. They can't find these break-ins. Right, they'll fall back to, okay, I'll try, I'll try five, I'll try six, I'll try seven. Right, the reasoning becomes really boring and nothing like what you see in the transcripts that we've, we've open sourced from this, from this YouTube channel. So I just wanna put the challenge out there, right? That this, this is a really difficult benchmark and I think progress on this benchmark will really mean progress in AI generally.

• 1:09:13Speaker 00

  Hey Spotify listeners, when you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admiral's Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 1:09:40Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration blog, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 1:10:05Tim Scarfe?

  Could you reflect a bit? So after watching this cracking the cryptic YouTube channel, how diverse were the patterns? Because Chris was saying to me, oh, you know, these guys, they go on Discord servers. They get these creative, crazy ideas. And I'm obsessed. Maybe I'm just being idealistic, but I love this idea of there being a deductive closure of knowledge. Right? That there's this big tree of reasoning, and we're all in possession of different parts of the tree to different depths. So the smarter and the more knowledgeable you are, the deeper down the tree you go. But in this idealized form, there is one tree, and all knowledge kind of originates or emanates from these abstract principles.

• 1:10:50Tim Scarfe?

  And we could, in principle, build reasoning engines that could just reason from first principles. And it might be computationally irreducible. So you have to perform all of the steps. And it feels like because we're not in possession of the full tree, what we need to do is kind of fish around. We fish around to find Lego blocks. So that's a good Lego block. I can apply that to this problem. And maybe that's just what we need to do in AI for the time being, is we need to just acquire as much of the tree as possible. But could we just do it all the way down?

• 1:11:22Speaker 02

  Yeah, fascinating question.

• 1:11:26Speaker 02

  That tree is probably massive, right? And as a human is solving these puzzles, they're definitely learning in real time and discovering new parts of this tree. And it's sort of a meta task, right? Because it's not just reasoning. Your reasoning about the reasoning. And I don't think we can. We have that in AI right now. Because if you watch the videos. They'll say something like, okay, this looks like a parity. Task or this is a set theoretic problem, or. You know, maybe I should get my path tool out and trace this this around. And of course, the professionals.

• 1:12:12Speaker 02

  They do have this this already massive collection of reasoning. Lego blocks, as you say, in their head, so they'll recognize, okay. That type of rule usually needs this kind of Lego block. It's actually fascinating to watch how good they are at just intuitively knowing. Where, you know, someone like me haven't solved as many. Needs to spend a lot of time looking around like, okay, maybe I should try this one, maybe I try this one. But even they're not perfect, so you can watch them. Take a certain kind of reasoning. And start building up, okay, maybe we should solve it like this. And then go. And no, that doesn't disambiguate it enough and then backtrack. And then go down another path.

• 1:12:57Speaker 02

  Again, something that we do not see current AIs doing. When they're trying to solve this this benchmark.

• 1:13:05Tim Scarfe?

  The the tree is very big, and I guess the phylogenetic distance. Between many of these motifs in in the tree is just so large, so it's so difficult to jump between. And and I and I think that's why as a collective intelligence. We work so well together because we actually find ways to jump to different parts of the tree.

• 1:13:25Speaker 02

  Right, and I and I think that's probably why. The RL, the the current state of the RL algorithms that we're trying to apply to this. Just isn't working. Because. In order to learn how to get these breakthroughs. To to understand what the sort of nuanced reasoning is to get these puzzles. You have to sample them and that it's such a rare space. You know, it's such a specific kind of reasoning that's required to get to the the specific breakthrough. That this kind of technique doesn't work, right? And there's definitely a feeling in the community like, okay, this is how you just solve things now. Like we have RL, yes, we can get these language models to do what we want.

• 1:14:10Speaker 02

  It doesn't work for this for this data sets.

• 1:14:13Tim Scarfe?

  Guys, it's been an absolute honor having you on the show. Just before we go, are you hiring? Because we've got a we've got a great audience of ML engineers and scientists. And I think working for Sakano would be the dream job.

• 1:14:28Speaker 02

  That's very kind of you. Yes, we are definitely hiring. And as I said earlier in this interview, I honestly want to give people as much research freedom as possible. I'm willing to make that bet, right? I think things that are very interesting will come out of this. And I think we've already seen plenty of interesting things coming out with this. So if you want to work on what you think is interesting and important, come to Japan.

• 1:14:57Tim Scarfe?

  And Japan just happens to be the most civilized culture in the world.

• 1:15:00Speaker 02

  Right.

• 1:15:01Tim Scarfe?

  It might be the opportunity of a lifetime, folks. So yeah, get in touch. Guys, seriously, thank you so much. It's been an honor having you both on the show.

• 1:15:09Speaker 02

  Thank you very much.

• 1:15:10Speaker 03

  Thank you so much. It's been great.

• 1:15:16Speaker 06

  Stonefire naan is made with real buttermilk and ghee. It's the perfect flatbread for any occasion.

• 1:15:21Speaker 07

  I put out Stonefire naan with dips at every dinner party.

• 1:15:24Speaker 06

  I make naan grilled cheese when my daughter has friends over.

• 1:15:27Speaker 07

  I use Stonefire naan to reward the humans who do my bidding. Right, it's perfect for camping trips. Dance for it, feeble mortal. Sorry, is he talking to me or? Dance.

• 1:15:41Speaker 06

  From our tradition to yours, Stonefire.

• 1:15:47Speaker 07

  When you shop Kroger Delivery, you can expect the savings you love and fresh groceries delivered right to your door. From delicious produce to everyday essentials, enjoy the convenience of delivery and the trusted selection you've come to know from Kroger. Because bringing you fresh, quality groceries is what we do best. Right now, enjoy free delivery on orders of $50 or more with your digital coupon. Restrictions apply. See site for details. Kroger. Fresh for everyone.