Latent Space: The AI Engineer Podcast
Original podcast
Watch the full show on YouTube: https://www.youtube.com/watch?v=Xkwok_XXQgw
3,221 words · 3 speakers
Yeah, okay. Today, I'd like to talk to you all a little bit about the frontier directions that we're all thinking about in terms of open source agentic R L. And what, basically, sketching what the roadmap might be for it's training something that looks like an O3 model. What does it take to get something that is like a very good general purpose agent for doing long horizon tasks? And we'd want a thing that can go off and browse the web and write code and look at images and do all this stuff in service of solving increasingly important problems. And so like, we know it's doable because OpenAI has done it, other people have done
things like this, but there really isn't currently an open source model that behaves in the way that these models do. We have things like R1, which are great at kind of the single turn math and code reasoning problems, but they are not in the general purpose agents world yet. And so I think there's a lot of things we need to solve along the way, but I think these problems are things that are solvable and that we also kind of like have a sketch of how to do it. And so I'm gonna, in this talk, be mapping out like what are these questions, what are the roadblocks, and what are like the plausible things we have to do that will actually unlock this? And so what we want is models that can often do stuff. And for L L M's doing stuff, it seems like it's best done with some kind of tool.
So this is one of the reasons that like everyone's crazy about M C P's, but it's like a standardized way to like give L L M's tools. But that's really what it is. You want models that are really good at doing multi-turn tool use. That's kind of the O3 magic is lots and lots and lots of multi-turn tool use with multimodal input in very general settings beyond just chat. And whether or not you want to call this like N minute A G I, I kind of like the phrase 10 minute A G I for just like how to think about O3. Is it anything that you can do as a human in 10 minutes? O3 is usually going to be the new reason we want. And that, if you can crack that scaling direction of like pushing the boundary of how long these models can go out and do these things for
That is a plausible path towards things to become more robust.
And so, O3, you can like give it hard problems, it like does a pretty good job. After 15 minutes of thinking, like sketching out a pretty good version of this solution. And so, the table stakes version of this, uh, which is kind of the area that I've been doing some good source work in, is multi-turn RL. So like, rather than just like, you train the model to like guess the right answer, you train the model with tools in the loop to be able to use these tools in service of solving some problem, and then you give it rewards based on the final output of this model after the sequence of tool calls.
And so, we've had a move beyond the single-turn RL, like JET world of like, JET TBTs or things, as well as the single-turn RLVR world, the models like R1 and R1. And we want to like have these things become more agentic, and it seems like the path is to incorporate re-invert forced learning into this process. Some of the challenges are like, the versions of these models that are quite good can do like many, many tool calls. Like, deep research can do like 100 tool calls. You probably want some kind of intermediate verification where you're not just waiting for the final answer at the end, but you want something like turn level reward potentially, where you want to be able to ensure that the model is getting credit for the moves it makes along the way,
because it's no longer just like one thing it's doing, it's doing 100 things, and the credit assignment problem becomes very tricky as you scale the, the length of these tasks. You might want to do this thing, these for things outside of math and code, where verifying whether an answer is right or wrong is not necessarily straightforward. You want to avoid like blowing up your context line, like if you have a model like reading websites, and if you put all the website in context, and the model is like doing this in RL, this like blows up your context limit really quickly if you have reading lots and lots of websites. You want infrastructure for managing the resources of these rollouts, uh, in terms of computer environments. If you're having models that are like doing data science, you want them to be able to like do data science quickly as part of this training process.
And you also ideally want to, whether you're If you're doing this centralized or decentralized, moving the direction of like everything being async and overlapped. Because that is like, otherwise you have these efficiency bubbles that like pop up all over your compute structure, your frame structure. And you ideally want to be able to like hide all of that by having everything going with some lag in parallel. And so that's kind of what the path forward seems to look like. And so this is just a little sketch of some work recently about kind of like working towards this where you kind of can do multi-turn RL by incorporating tool calls into the loop.
There's been a big sequence of papers of people doing like many applications of like coding agents and search agents. And people are kind of in the public sphere documenting a lot of their learnings of this process. And I think there's a lot of value that is kind of floating around in scattered papers where we just need to like understand the lessons learned, build from them and aggregate this into kind of training recipes that can become more scalable by crowdsourcing the answers to some of these questions. Some of the popular agents are using the number of tool calls scales quite dramatically. The search surface, like I said, can go up to 100. And we also do see that like for really hard problems, doing more tool calls, searching the web more gives you a nice scaling curve where you get better answers by putting in more effort, by like
spending more time browsing the internet essentially. And so this is the thing that we want to have a process for scaling. We want technology, we want infrastructure, we want recipes and experiments to push along these trajectories.
Multimodal is also one of the big unlocks in O3. Where I think for a while some people were like speculating like, oh, what if you have the model like generate images in its chain of thought reasoning where everything is like end-to-end multimodal input and output. And it seems like you don't really need to do that because for a lot of the like multimodal things that are useful for reasoning, it's actually just more efficient and easier to have models use like programmatic tools to manipulate images. So like O3 can like crop an image and zoom in and rotate. And I think there's a lot of mileage in using this sort of thing where the ability to work with these images allows you to understand them in different ways by shifting the way that the model has them in its time window to emphasize certain things more.
And so I think just like leaning into thinking of modes of intelligence as tool calls is a very powerful like mental model and practical trick for having models do the things that you want them to do. And once you have these kind of things that exist in the tool call format, which I think there are several other examples of, then you start being able to have a way to train the model to do the thing. And so like geoguessor, for example, or like the model is like zooming in to like look at the license plate. There's all sorts of these things that humans do kind of instinctually, for example, by like focusing our attention. So what does that look like for an L L M?
How do you like refocus your attention? The simple way is just crop the image and zoom it. And so that is one piece of the puzzle we probably do want if we're working towards scaling up these kinds of systems and we want to replicate those sorts of models. Additionally, I think people were like excited about multi-agent systems for a while, like the crew AI, sort of thing of like, oh, I'm gonna put my coder agent, my finance agent in a group chat. And like a lot of these are just kind of silly. They don't actually work very well because these models are not trained to do that. They're not ever trained in a way that actually reinforces those patterns. Multi-agent RL is hard. I did five years of it in grad school. It's like not easy.
And to, the algorithms just don't really even exist for the things you would really want to do. But there are some kind of shortcuts where you do get very focused versions of multi-agent learning that have good synergies with the scaling paradigm and that we're trying to do. So some of these are just like allowing other models to be tool calls. So let's say you're Raspberry Internet and you want a model that can like go read a hundred websites. You probably don't want all hundred full websites in your context window of the main model. But what you might want to do is have sub-agents whose only job is to answer a question about a website. And once you have this, now you can do things much more efficiently. You can parallelize aspect of the recent chains and especially for things that can be done with like off-the-shelf small
You don't need to train these end to end because they're already decent at being an L M. And there's a lot of passive reasoning that can be offloaded to just like any good enough L M. At which point you now can just train the main model to learn how to prompt these models about to use them. This is also something that we might want to do with things like video summarization. You probably don't want that all in context for most cases. Navigating the web, it might be the case of like having really large images every second in your context window like blows things up really quickly. But if you can offload this with like kind of more condensed interfaces to using V L M's as tool calls. Then you can have much more logic exist in your reasoning chain within your context window.
And you can then scale to much longer chains of call calls without needing as many tokens it takes to have an image of the website here in your context window at every single second. And so this is going to be a problem, especially for doing this in an open source centralized way. You like scaling to like million token context is like really, really hard. There, I don't think there are real like open source replications of many token context scaling beyond like tiny like academic model sizes. And if you want to do that type of reasoning, you want to kind of use every trick you can. And I think offloading in regards to this to sub models is like a very effective one that you can specialize with R L.
Because once the subversion is solved, you kind of freeze that version of the Kelpner model as a tool call and it broadens it more.
Another thing on context management is like these models benefit from thinking a lot. But the thinking takes all the tokens. And so if you're doing multi-step learning, you have all these thinking chains that blow your code up. And some of these models like for inference, they're not actually keeping those thinking chains around. They're like, since you're doing the thinking for a step, throwing away the thinking tokens and then continuing on. And we kind of need new algorithms for doing this. Like the existing paper people are writing about multi-turn R L are not actually incorporating this. And it kind of like breaks all the math where you need to like the recent paper blog post like two days ago from cognition for their like kernel bench A I does this in kind of like a simple form where
each step is essentially like a final answer, but they allow it to like do many final entries in a row. And they have like their one verification system. To do this efficiently, you kind of do need to do like true turn-level training, where you have this big forking process, where you need to be able to assign rewards to each one. But sometimes if these are like actually intermediate reasoning steps, you don't have a good verifier off the shelf, which is going to need both like new algorithm research, as well as the ability to do verification for things like, was this a good Google query? Was this a good crop of the image? Was this thing helpful in service of the final goal?
So that's a question that we need to be able to answer efficiently repeatedly in this kind of training. And the most, one of the most promising ways, I think, towards doing this is having the reward models also be able to answer harder questions by themselves being reasoning models. So there's a recent DeepSeat paper on this sort of thing, where you train models to be good like general reward models by doing inference scaling, where they break down tasks into sub-criteria rubrics, and then they can evaluate these numerically based on following the rubric in reference to the thing that they are storing. And this, among other tricks, there's lots of like clever reverse engineering things you can do to kind of have other measures of similarity or quality with structure
of the ground truth answer. I won't go into all the details there, but the core idea there is if you think of reward models as reasoners, who then you can also implement for time scaling to do harder and harder tricks, harder and harder problem-solving, step verification, you have a way of giving these rewards throughout the process, which you kind of need to do if you want to scale along connotes. And, but that also introduces the bubble of like, okay, now reward modeling might take a minute for validating one turn, which really means you want to push really heavily in the direction of everything being async and like happening later, which might mean your model's policy is being trained on turns that happened 20 minutes ago, which is kind of like, in old school RL days, people did this all the time with like replay buffers
and whatnot. You really want both your tool calls and your inference and your reward modeling and your training to all be happening in parallel with whatever. As much of a lag as you can tolerate, as well as your communication. And so that is, I think, one of the big reasons why, like, pushing on async RL is really huge, not just for instance, like, training, but for things like scaling over modeling, for things like scaling call calls, scaling resources. And I think Prime and Alex has done some very cool work showing that this is like a direction you can push. You can go off policy. You can allow your models to be trained on like slightly stale data and it still works because these things are pretty robust. And if you set everything up right, you tune your parameters and you ensure that the
training process is like stable otherwise, then you have a paradigm which allows you to do all of these things. You can scale onto a boat. You can scale on over modeling. You can scale on generality of tasks. And everything still fits into this nice async pipeline where everything is flowing from one to the next thing without like blowing up your train bubbles. You can also decentralize your skill acquisition. You can train on many things in parallel and it seems like model merging like works weirdly well. That was one thing that I kind of updated on, which is like you can have a model trained on code and a model trained on math, a model trained on Spanish, and you can literally average the weights and it works, which on one hand sounds crazy, but on the other hand,
it's like this is kind of what is being done by the local self-retraining at smaller scales anyways. And it seems like the updates made to model weights are orthogonal enough for specialized tasks that this is actually like totally compatible. Things are nice and linear in most cases. Things are nice and orthogonal. And you can get away with a lot of async updates to models that are then merged even without full communication. And so co-figured this very extensively in their latest flagship model, Command A. And it seems like we have a lot of ways where you can split up different pieces of the process of learning to do X, Y, or Z, or games, or coding agents, or math. And these can all go off and happen and then come together and merge.
And if you can do this, then you can train decentralized models with lots and lots of capabilities because the capabilities themselves can be learned in decentralized methods. So what does this mean? Like, it's gonna be hard. It's not gonna be easy to like put all of these things together and make it work well and really train these kinds of models superficially. But it seems like it's doable. We kind of know the sorts of problems that we have to solve to do it. We kind of know what things are going to be really annoying and how we can kind of shortcut some of them by like using kind of fallback cheap tricks, because these cheap tricks are like unreasonably effective. And this does kind of give us a roadmap and a recipe of what questions we need to answer,
what things we have to build,
and what we have to accomplish in order to train decentralized end-minute AGI.
Okay, we have a special guest that we're pretty excited about. So, here we go.
All right, actually, we're going to pause for one second and just do another transition, so.
This is Moji!
Page updated October 25, 2026 · transcript and summary are machine-generated
