[AIEWF Preview] Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

In an otherwise heavy week packed with Microsoft Build, Google I/O, and OpenAI io, the worst kept secret in biglab land was the launch of Claude 4, particularly the triumphant return of Opus, which many had been clamoring for. We will leave the specific Claude 4 recap to AINews, however we think that both Gemini’s progress on Deep Think this week and Claude 4 represent the next frontier of progress on inference time compute/reasoning (at last until GPT5 ships this summer). Will Brown’s talk at AIE NYC and open source work on verifiers have made him one of the most prominent voices able to publicly discuss (aka without the vaguepoasting LoRA they put on you when you join a biglab) the current state of the art in reasoning models and where current SOTA research directions lead. We discussed his latest paper on Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment and he has previewed his AIEWF talk on Agentic RL for those with the temerity to power thru bad meetup audio. Full Video Episode Timestamps 00:00 https://www.youtube.com/watch?v=furFAnmbam0 Introduction to the Podcast and Guests01:00 https://www.youtube.com/watch?v=furFAnmbam0&t=60s Discussion on Claude 4 and AI Models03:07 https://www.youtube.com/watch?v=furFAnmbam0&t=187s Extended Thinking and Tool Use in AI06:47 https://www.youtube.com/watch?v=furFAnmbam0&t=407s Technical Highlights and Model Trustworthiness10:31 https://www.youtube.com/watch?v=furFAnmbam0&t=631s Thinking Budgets and Their Implications13:38 https://www.youtube.com/watch?v=furFAnmbam0&t=818s Controversy Surrounding Opus and AI Ethics18:49 https://www.youtube.com/watch?v=furFAnmbam0&t=1129s Reflections on AI Tools and Their Limitations21:58 https://www.youtube.com/watch?v=furFAnmbam0&t=1318s The Chaos of Predictive Systems22:56 https://www.youtube.com/watch?v=furFAnmbam0&t=1376s Marketing and Safety in AI Models24:30 https://www.youtube.com/watch?v=furFAnmbam0&t=1470s Evaluating AI Companies and Their Strategies25:53 https://www.youtube.com/watch?v=furFAnmbam0&t=1553s The Role of Academia in AI Evaluations27:43 https://www.youtube.com/watch?v=furFAnmbam0&t=1663s Teaching Taste in Research28:41 https://www.youtube.com/watch?v=furFAnmbam0&t=1721s Making Educated Bets in AI Research30:12 https://www.youtube.com/watch?v=furFAnmbam0&t=1812s Recent Developments in Multi-Turn Tool Use32:50 https://www.youtube.com/watch?v=furFAnmbam0&t=1970s Incentivizing Tool Use in AI Models34:45 https://www.youtube.com/watch?v=furFAnmbam0&t=2085s The Future of Reward Models in AI39:10 Exploring Flexible Reward Systems This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Hello AI engineers. We're back with a quick reaction pod for Claude 4 with the new reasoning research lead for Prime Intellect, Will Brown. Will Brown's talk at AIE NYC and open source work on verifiers have made him one of the most prominent voices able to publicly discuss the current state of the art in reasoning models and where current SOTA research directions lead. We discussed his latest paper on reinforcing multi-turn reasoning in LLM agents via turn level credit assignment, and he has previewed his upcoming AI Engineer World's Fair talk on Agentic RL linked in the show notes. We're excited to share that Will will be back at the upcoming AI Engineer World's Fair in San Francisco, which now has expo tickets on sale.

• 0:46Speaker 00

  He will be headlining the new RL plus reasoning track with Mishal Laskin, Nathan Lambert, Christian Segarty, Greg Camrat, Kyle Corbett, and more. Join us at ai-engineer. Watch out and take care.

• 1:02Alessio Fanelli?

  Hey, everyone. Welcome to a Lightning plus Emergency News, a Late in Space Podcast episode. I'm Alessio, Partner and CTO at Decibel, and joined by my co-host, Wix, founder of Small AI.

• 1:13Speaker 02

  Hey, hey. And yeah, honestly, we knew that Claude 4 was coming, and we just didn't, we're just too busy to like have a dedicated episode. So like, this is our makeup dedicated episode with a special guest, Will Brown from Now I Can Say It, Prime Intellect.

• 1:29Speaker 03

  Hey, how's it going? Great to be on. And so excited to know each other for a little bit. And this is my first time on the podcast, I believe. Great to chat with you guys. Big news day, I guess. So lots of stuff out in the world.

• 1:45Speaker 03

  There's always a news day.

• 1:47Speaker 02

  I think this week is particularly heavy for some weird reason, like Monday was Microsoft build, Tuesday, Wednesday, Google, and today is Claude. I wonder what tomorrow will bring.

• 1:57Speaker 03

  We had I/O, and then we had I/O, and then.

• 2:00Speaker 02

  Yeah, yeah. Different IOs, exactly. Yeah, so like, we actually were supposed to record this morning, and we all wanted to watch the Claude keynote. So we went and watched the Claude keynote. Obviously, a good model, you know, good model, big model, they're really emphasizing coding. They didn't really talk much about reasoning, to be super honest. They were just like, it runs for longer now. What are you guys' takes?

• 2:23Speaker 03

  Yeah, so I mean, like, one thing I've kind of been. Seeing coming for a little bit that I think people are kind of also all aware of now is that like the thing that's going to make the next wave of stuff be powerful is just like everyone wants better agents, everyone wants models that can like go off and do stuff.

• 2:41Speaker 03

  And like reasoning was kind of like a precursor to that a little bit like, I mean, I always think of like OpenAI is like five levels framework for like chatbots was like the RLHF era.

• 2:53Speaker 03

  And then reasoners was like the one and R1. But like really what people were thinking of was reasoners are a step on the path towards agents. And so I can kind of see why Claude or Anthropic is not like, oh, we have the best reasoner. They're really like showing off their sweet agent and like tool, tool use and like function calling benchmarks, multi-turn stuff. Because I think that's really like what people care about more for actual applications as opposed to like did really good on this math competition. Like the math competition was like, that stuff was all like a signal that was supposed to think we were getting somewhere. But the thing we were getting towards for a lot of people at least is practical agents.

• 3:30Alessio Fanelli?

  Yeah, the, I, I think the extended thinking mode, I think they removed the uppercase. I think in the Claude 3 release, it was like extended thinking, kind of like a capitalized and now just like extended thinking with tool use. So I think they're also, yeah, done playing whether or not it's reasoning or not. I think they're trying to merge everything together. And it's, I, I mean, I didn't realize that, but extended thinking could not use tools before the way they worded it. And now they can. And, and opus four. So that's great. But yeah, they, they haven't put it as far center as last time.

• 4:01Speaker 02

  Do we have any, this is like already veering off from Claude directly into speculation, but do we have any idea if there are any material differences between how Claude extended thinking works versus like the old series models? Do we know?

• 4:17Speaker 03

  The biggest difference seems to be at least, and this is kind of a thing that's been, I mean, I don't, this is all speculation, of course, but from the start, Anthropic had always kind of had this like little thinking thing where you could.

• 4:30Speaker 03

  Sometimes even like Claude 3.5 would do like a tiny bit of thinking and it was really just like deciding which tool to use for the most part. Like if it was doing

• 4:41Speaker 03

  an artifact in the Claude UI, it would have this little thing where it would think for like two sentences about which tool to use. And it seemed like Anthropic's kind of attitude has been that extended thinking is an instance of tool use and that it's the kind of thing you want to equip the model with the ability to do. But it's not like, oh, it's a thinking model. It's just a sync for the model to like brain vomit because that brain vomiting will help it like find a nice thing to do next. In the same way that doing search or doing code execution are like ways to kind of get more information on the path towards like finishing a problem.

• 5:20Speaker 02

  Yeah, inference time compute, as they say. I did meet somebody who claimed to have coined to found the Scratchpad paper. And this was obviously before the Jason Wei Chain of Thought paper, but it's all the same sort of method, general family of techniques. I think the question for me is also like, is there some model routing going on? Like, are they different models, the thinking, non-thinking? Or are they the same models with like, just like you turn off the end of turn token generation?

• 5:47Speaker 03

  I mean, I think these models should be the same model and Anthropic knows what they're doing well. Like, it's not that hard to like, Quinn did it in a very kind of like simple way and they kind of talked about how they did it a little bit. But it's not like too difficult to like have whether or not I'm all things like be the sort of thing. I mean, like, obviously, all this stuff is like hard at like serious scale, but like conceptually at least, it's not like a big problem to solve about how would you ever do it? It's like, no, we have reinforcement learning. We can kind of like.

• 6:23Speaker 03

  Or we just SFT on like different things. We can kind of teach models skills like that pretty.

• 6:29Speaker 02

  Yeah. Um, you have some work that you've published recently on like G R P O and relationship for and you're doing a lot of work on multi-turn RL. I think I think I wanted to just kind of round out any other Claude highlights. You know, maybe I saw.

• 6:43Speaker 03

  Yeah.

• 6:44Speaker 02

  There is a, there is controversy that I'm leaving towards the end, but like any other technical highlights that you guys want to focus on?

• 6:51Speaker 03

  I mean, I think it seems like a really cool model, but I think like a Calama is like tweeted this earlier today. It seems like it's linear progress, which is like great, but it doesn't feel like there's not anything that I've seen from it. It feels like a paradigm shift in terms of like the sorts of stuff Daria talks about, which like, I think maybe we're still on the path to get there. Um, and it feels like this is just like gone up in terms of complexity of agents. I think the one thing that to me was really nice to see, I haven't like done too much testing myself yet, but in their reported benchmarks, the reward hacking issues to like Sonnet three, seven

• 7:32Speaker 03

  loves to like do stuff that to me feels reward in the sense of like it'll try to, you ask it a coding question and it would like do your question and then seven other things also presumably because there was some RL environment where there wasn't really a penalty for doing that or there was an enough penalty and covering its bases like was more likely to pass test cases on some coding thing. Like you could imagine like a speed bench kind of thing where there's a minimal diff is really what you want, but there's like, you could do a ton of other stuff and put all these other things in place. It as long as you don't, as long as it's not enough that you trip over your feet, it's just like extra stuff that's there if it helps pass the test cases.

• 8:12Speaker 03

  And what I really think you want to do with these models is like kind of min max, like you want the models to like do, do the thing and no more. And they had some internal benchmark for this that went from like 45% down to 15% for both for Sonnet and for OPUS as opposed to three, seven. And so I'm hopeful that these models are much more like friendly to code with and maybe more like trustworthy. And that's the thing that I kind of have buckets for models of like, how much can I trust them in a code base? Um, especially something beyond like a single file like. Old Gemini to me was very trustworthy. GPT 4.1 is very trustworthy. New Gemini is not. Three Sonata is not. O3 is not. Haven't decided which bucket New, New Sonata, New Opus are gonna fall into.

• 8:57Speaker 02

  Trustworthy in terms of reward hacking.

• 9:00Speaker 03

  Just like not gonna make them like the, they're gonna do the right thing in the codebase and like worst case they'll do it like dumb. But they're not gonna like go break a bunch of stuff. They're not gonna leave a bunch of like extraneous comments and helper functions all over the place that aren't really needed. Or like make seven new files just to have them there. Like this is the sort of thing that 37 does a lot.

• 9:24Speaker 02

  Yeah, it was like I had already have the function in my codebase. I'll just make a new one just because it felt like it. Yeah. Uh, one thing I always often wonder about those things is like, it's just for RL environments in general, like, why is it? Token costs more of a thing in the penalties, you know, like that's the one rule above all. Like you can, you can actually skip a lot of reward hacking by just, hey, the, the more tokens you use, the worse it is.

• 9:49Speaker 03

  I mean, that's not what the model, they're selling you tokens. They want you to.

• 9:54Speaker 02

  Okay.

• 9:56Speaker 03

  But like, so like, there's that element of it, but I think also it's that there was this initial kind of reaction of everybody of like, more tokens is better. If you look at the line, it goes up as you spend more tokens, your accuracy goes up. And so I think the pressure to like really tamp down on token usage was not that serious for a lot of people. Especially because the companies are like, you to sell you more tokens. But it is the sort of thing that you can have some more controls over. So like, Quinn did this in kind of like a very kind of abrupt way. Where they can like, you can, in the UI, you can like set a token budget and just like truncate the thought. So it seems like artificially truncating the thought is actually like fine.

• 10:36Speaker 03

  Like the model can, even if like it got cut off mid-sentence with a injected like think token. These are smart enough models that they can kind of. Finish with the best that they got from that point. And so that's like one way to do it. The other and like that's becoming a kind of standard API feature now is like your think budget. Claude has that. Yeah, we did a little bit of experimentation with that in our last in like to run it. I'm an elect, which it was before I joined, but thinking budgets are the kinds of things that you can insert into a reinforcement objective. And you can see the model like get better at targeting the right amount of thinking based on like, let's say something goes in your system prompt. And you can have the prompt, just say use X amount of tokens. It doesn't need to be like, but if you've kind of trained the model to like, respect this,

• 11:21Speaker 03

  you would hope that if you like execute this correctly, the model learns to like roughly think the right amount.

• 11:27Speaker 02

  Okay, this actually changed my opinion of thinking budgets. Because previously I was thinking that reasoning effort was better than thinking budgets. Thinking budgets is kind of like a max cutoff.

• 11:36Speaker 03

  The same thing.

• 11:37Speaker 02

  It's a target, right? It's not, it's not a.

• 11:39Speaker 03

  It's okay, the effort is a target, probably.

• 11:41Speaker 02

  Yeah. Right, right, right. Yeah, because I actually want to set effort. I don't super care about cutoff apart from the cost and like giving me, you know, 64 bits of cutoff or whatever, it doesn't matter.

• 11:53Speaker 03

  I'm not sure that they're like that different. I think like, we don't know how they do it under the hood. But my guess is that the whole reasoning effort thing is essentially a token budget that the model has been like RL to like, you would hope that you get different behavior. So the model, when it's told it has a short thinking budget, you would hope that uses slightly different strategies that are better versus if it has a high budget. It's more willing to like, do lots of math calculations, for example. But I think conceptually, it's really just about the model has some amount of room that can bank in tokens and.

• 12:29Speaker 02

  Yeah.

• 12:29Speaker 03

  It's trying to do that well, hopefully.

• 12:31Alessio Fanelli?

  Do you think we're gonna have this as hyper parameters for like much longer? Or do you think this is kind of like, you know, as we're early in this like reasoning models, more of the stuff is exposed and then it gets moved away from the user?

• 12:44Speaker 03

  I think in chat interfaces, it probably won't stick around. Like, I don't think we're always going to have the dropdown of like 04 mini and 04 mini high. That feels silly. I do think it's a thing that developers want, especially because once you've kind of built around a certain model, like a lot of these providers are hoping you stick with the one model and are not switching all the time. You do need a knob to control costs and also latency. And so that is one kind of useful knob to expose developers for controlling this like quality versus cost and latency.

• 13:16Speaker 02

  Awesome. Cool on all of that. I think the elephant in the room, let's talk about it, is this controversy around Opus or Opus, right? Snitching on you.

• 13:27Speaker 03

  Yeah, I mean, so.

• 13:30Speaker 02

  So for those, for those out of the loop, let's, let's recap. Because I feel like you're, you're closer to this than, than I am. Like, I learned about it from you.

• 13:37Speaker 03

  Sure, yeah. So this was someone who wrote, I'm not going to name him, like, because I know he doesn't want to like have all this attention on me. He deleted the tweet. Of course he did. It was essentially like going through different things that like people found during safety stress testing of Claude. And so this is not like, what's Claude going to do for you? I think people took this out of context like pretty badly. And so there's a fair point there that it's like, people are really reading into the one sentence but for them they should. But this is the thing Anthropic does a lot is they really stress test their models. They try to put their models in situations where they can really see like, what could an adversary get the model to do? Or what does the model do if it's in a situation where there's no right answer?

• 14:21Speaker 03

  So like, I think a lot of the kind of headline Anthropic like safety results, especially related to reward hacking and kind of deviation and alignment faking are all things to me that seem like a rock and a hard place situation where the model has two objectives it's given that are conflicting with each other. And it has to pick one. And no matter which one it picks, it's going to sound terrible. Like, it's either following the user's instructions or it's following like common norms. And once you kind of accept either of those, it's going to do the thing that is aligned with that set of like guidelines. So in the case of like, if your model's goal is to be like maximally helpful to the user, then it would help a user like build a bomb.

• 15:03Speaker 03

  Models' goals to be maximally helpful to society and a user's asking me to build a bomb. It's gonna be like, no, that's bad. I have to do something to stop this. Like, you kind of have to pick a goal.

• 15:14Speaker 03

  And like, maybe the right answer is the model just defers and is like, nope, I'm gonna stop talking. But people also get mad when you tell them like that the model will stop talking to you or like refuse to do anything. Like, there's just no, it's, there's no, it's no way to kind of win and make everybody happy. But I do think like, like they report this because they think it's important to have people understand the safety implications of these models and to understand like, okay, how bad would it be if someone was trying to use this? Could this like meaningfully help someone commit crime or violence or whatever? And so like, that's what they have like their safety framework for. And the things that happen in these like blog posts and threads and papers about like the model trying these things,

• 15:56Speaker 03

  they're kind of putting these models in a scenario that elicits these things. Like, it's the sort of thing that you would imagine a very smart human might also do in those situations. Like, let's say you are told like, accomplish some vague, unspecified goal at any cost. And you really like want to accomplish that goal. Think like game shows, like Survivor, I think is a good example of like, or Lord of the Flies, any of these like kind of canonical situations of people who have put in a weird spot and have to go do stuff and figure it out how to do it. They're kind of crafting these environments for the models and just looking at it and seeing what happens. And so like, I think it is a little silly to overanalyze behaviors in either direction of like, oh, the model is reporting you to the police

• 16:40Speaker 03

  or the model's gonna go help you find uranium on the dark web. Like, well, these models can kind of do, there's no, like, they're, the base model in general of LLM is not artificially constrained in any way. Like, with the right prompt, it'll do whatever up to its intelligence limit. And so like, the question is just how do you constrain the space from all possibilities down to like a more reasonable set? And like, that's hard. So.

• 17:03Speaker 02

  Okay, you actually gave a serious answer, which I totally respect. I was smart looking for shitposts.

• 17:09Speaker 03

  I mean.

• 17:10Speaker 02

  Like you're treating this as though like, yep, this is how, this is what the problem actually is, which is like totally fine. And yeah, I mean, that's what you are as a researcher, right?

• 17:20Speaker 03

  Yeah, I mean, I think tweeting is fun. Like it's cathartic to like, just kind of like get a post out. So like, when I saw the one about the uranium thing, I was like, let me tweet. So the tweet was like, we found that Claude can go search the dark web to look for like uranium. And I was like, here are the top 10 things that builders are using in their agentic rag applications with the new groundbreaking Claude 4. And it was just like silly, both making fun of like LinkedIn, like thread posters as well as just like the funniness of the scenario that they were talking about.

• 17:55Speaker 02

  Yeah, this is it.

• 17:56Alessio Fanelli?

  Does any of this make you think differently about what tools to give a LLM? You know, I know they deleted the tweet, but it's basically like, well, before if you're putting all these MCPs, like, yeah, you have email access and all of this. And now it's like, well, maybe I don't want to give email access all the time. If you're gonna snitch on me with the email access.

• 18:14Speaker 03

  I mean, I think coding with these models, especially like Claude 3, I did a fair amount. Like for a few weeks, I was doing a lot of Claude code with 3.7, mostly for kind of random side projects. I never really got to the point where I found it was helpful for a thing that was like a large existing codebase. But if it's like, hey, I want to like cook something up in a few hours for fun, pretty good at that. But these become messy and they become hard to maintain. And you get to a point where it's like, nothing is working. I just got to like dig in and fix it all myself. And so I think part of that is that the models have access to like a terminal and you can do a lot of stuff in a terminal. MCP is kind of a way of constraining the action space. So like in like canonical RL, people talk about like states, actions, rewards, policies as like the things that are like the moving parts.

• 19:00Speaker 03

  Models generally are trained in like old school RL with like a very fixed action space of like, what are the keys on the video game I can hit? But with a LLM, it's like text. Text is like kind of unbounded in what you can do with it in a terminal. There's not much you can't do in a terminal. And so if you're training models.

• 19:18Speaker 03

  Oh, I got a lot of flack for this one.

• 19:20Speaker 02

  I'm just showing this. Wait, flack? Why?

• 19:23Speaker 03

  People were, it was both people who were like, the notation is stupid and bad, but RL is really simple, or like RL is like complicated. And it's like, everyone has a different opinion on what RL means.

• 19:35Speaker 03

  And I was trying to just like, be like, hey, it's actually kind of complicated. And I wasn't picking this of like, oh, the definition of the NMDP is complicated. I was like, no, there's just like a lot of moving parts.

• 19:46Speaker 03

  And to think about it, to like, do anything, especially if you want to change any like, part of the system, like, here's a question, hypothetical, like, what happens if you have two LNs learning together? How do you reason about that? How do you like, think about that? Is this going to be a stable system or not a stable system? What if they're like, kind of cooperative, but kind of not cooperative? And then they're training to work together, but also want to back step each other? Like, this is kind of the environment people are finding themselves all in the, all the time out in the real world. But if you want to make AIs do this, you have to like, translate this into code and math. And the more complex your goals are with this thing, the more complex the math gets. And RL is like one math language that kind of exposes these primitives.

• 20:32Speaker 03

  But like, I think a lot of people are like, oh, I can follow the equations. That means I understand it. And it's like, well, sure, but like, also there's, it's like this, I don't know, N-body problem thing, where you can freeze it and look at it. It's like, oh, how does one thing moving affect everything else? And what are the cascading ripple effects?

• 20:50Speaker 02

  Wow, you just brought three-body problem into this. Amazing.

• 20:54Speaker 03

  Like, as in like the physics bird and not the show?

• 20:57Speaker 02

  No, no, no. I mean, actually, how, actually very like impossible to model. Like, I guess you can like, simulate it, but like,

• 21:06Speaker 03

  It's chaotic. It's like, it's sensitive to initial conditions. So like, you can't really say like, this is one of those things that like, like, why does no one predict the weather a year out? There's no, I don't think anyone has anything that's like, good at long-term weather forecasting, beyond like, I don't know, climate trials, but no one can predict whether it's going to rain in Seattle on a given day in a year. Even if you think like, like, the system's predetermined, like, we, it's all clouds bumping off each other and whatnot, and mountain ranges, we kind of know how these things work.

• 21:36Speaker 02

  So the butterflies are flapping their wings. I mean, like you gotta

• 21:39Speaker 03

  Exactly.

• 21:39Speaker 02

  Let it play out. Like butterflies. If we had no butterflies, we could predict it.

• 21:45Speaker 03

  Right. And so it's very sensitive to butterflies.

• 21:50Speaker 02

  Interesting. Okay. So I guess we can, we can sort of round it out unless there's any more of the controversy. I think there is. And like, I think that the system card is actually very good. They probably went too hard on it compared to like normal system cards. But uh, and it's a little bit confusing whether this is marketing or are they just like, no, we really super care about safety. And part of this is like Apollo just being Apollo, um, you know, pushing the frontier of red teaming, right? So they're going to report the things because it's extremely good Apollo marketing.

• 22:19Speaker 03

  Yeah.

• 22:21Speaker 03

  I think they're really, they're seem to still be like trying to be creative with their kind of consumer marketing. Like it feels like people in the AI world like love Claude or have grown tighter Claude, but still had a phase where they were using it a ton. But it hasn't really broken out to general people in the way. And it feels like a lot of their marketing that I've seen is like a little confusing. Like it feels like they've done a really good job at crafting a brand image that appeals to a segment of the population. Who has certain considerations that they really like, that a model has a deep personality or whatever. People, the sorts of people who I think also really like GPT four point five, many of them like really loved like Claude three opus.

• 23:05Speaker 03

  The big model smell, like a lot of people just don't care.

• 23:09Speaker 02

  Yeah, I just wanted to use it as a tool.

• 23:12Speaker 03

  Yeah. Trying to figure out how to like appeal to that audience. The LM sis sycophanti four oh, those, the people who love those models, different crowd. And it's a, it's a larger crowd. Um, and that's a tough problem to solve.

• 23:27Speaker 02

  What's your quick take on LM arena getting a hundred million dollars?

• 23:29Speaker 03

  Well, see, like, I imagine that they partner with B labs in different capacities to

• 23:35Speaker 02

  Probably making a lot of money. Yeah.

• 23:37Speaker 03

  Like. I'm not, I'm not in the business of trying to point the finger at like saying they definitely did this, but if I was a company that was able to raise at that kind of valuation and I had just had a long public partnership with Meta, eventually public partnership for a thing, which we've kind of seen was Meta had the ability to do a lot more back and forth than a lot of other webs did, I would imagine that there's some compensation going on there or access to data. And so like, I think being an eval company puts you in a really hard spot. Some people are talking about this on Twitter, like just that to be an eval company, you kind of have to sell to the labs, but selling to the labs doesn't really like kind of wreck your evals.

• 24:22Speaker 02

  Because your customer, yeah, yeah. So in finance, we would, I mean, you know, you are from Mooy Stanley. So this is the credit rating agencies, like literally your customer is the one that you're supposed to govern, but they're also your customers. So then you have to be like nice to them or they'll just go to the next one.

• 24:41Speaker 03

  Yeah. I mean, I do think that like the best source of evals going forward is probably going to be academia. And so this is the thing that I tell people who are like, starting a PhD, it's just like, find things that are like cheap to work on as a PhD student because you cannot go pre-train a foundation model really on your own, but you can build a really good, really clever eval. And like, we're churning through evals at the time, we saturate them, we always need more. It's not the kind of thing that is ever going to like end. And so that's the task of translating like vibes of what is good or bad about a model into kind of very precise scientific questions, I think is an important problem. It's a problem

• 25:22Speaker 03

  that you can get by a lot more with like brain power rather than dumping capital into it. You need to like pay for the API costs, but like, that is generally the kind of thing that either you can get covered within academic grants or like industry sponsors or the kind of thing that's just like, there's versions of these things that are like small sample size that get you on, get on the radar, or you kind of pick and choose which models you can afford to eval. But it's like an accessible field of research and it's One that like the incentives of academia I think are quite good for, which is like, write a splashy paper. That says something interesting about the broader field, rather than, oh, we want to make this one look like the winner.

• 26:01Speaker 02

  Yeah, I think a lot of grad students though don't have taste. I don't know how how better to put it. It just.

• 26:06Speaker 03

  That's fair. Yeah.

• 26:07Speaker 02

  But you go to enough academic conferences and I'm like, why did you work on this, man? Like, you're so, you're so smart. You're capable of better. So how do you teach taste?

• 26:15Speaker 03

  I think, I mean, I can tell how I did it originally, which is like, I think you always want to be thinking pretty far ahead and you want to be like, making kind of. Educated bets about what the world looks like in the state, in the theaters, like. You have to say like, what are the questions that no one's even talking about? And this is like not an easy thing to do. You have to like really convince yourself that you're kind of right about the way things at least might go. Like when I was like in, I finished up undergrad like late, like they finished in 2019 and then went right into grad school.

• 26:49Speaker 03

  But like towards the end of the 2010s, like we'd had like AlphaGo and DeepMind, uh, doing all this multi-agent RL stuff that was like really cool. Um, then it was like, okay. This stuff kind of works like AI is like going somewhere. Multi-agent systems are kind of going somewhere, still very early stages, but what's going to happen once this gets there? And it seemed like, okay, these things are all going to be like continually learning in parallel as this big multiplayer game, basically. And if you look at the math, the math was kind of like undercooked and there's like some really hard open questions that are still open questions in multi-agent learning theory. And so like that was my focus. It was, which was like, how do I like learn about this? How do I learn to think about this stuff better?

• 27:32Speaker 03

  And at some point I kind of got tired of proving theorems and it was like, okay, let's just go build a thing. But I think like. You want to think about like, whether you're doing theory or experiments, like.

• 27:45Speaker 03

  You have to lay out a few different conditional statements to get to the point where. You're really doing interesting research that's beyond like just living through that people are like obviously going to be working on in parallel. You want to be jumping ahead of the curve a little bit.

• 28:03Speaker 03

  I think my lot, I don't know, this isn't like, I wasn't the first person to do this, but like, it was pretty clear to me like after ON and before R1 that like RL was gonna work and that that was going to intersect with agents where the solution was gonna be like RL with tool use. That seemed like the way, the direction things were gonna go. And so that was like, I don't think that was a very risky research bet, but it was like a research bet that seemed to work out.

• 28:29Speaker 02

  Yeah. Speaking of which, you just published the paper. Now, now I have the full context is that you were an advisor on this and one of your grad students was doing the work, something like that?

• 28:39Speaker 03

  Yeah. So it was me with, Cilyon was my intern. This was kind of the last major thing I was working on at Morgan Stanley. And this kind of was in parallel with the verifiers as the repo that I've been building out. Major updates that coming very soon, by the way, I'm very excited about some stuff. But it kind of was something I really started in earnest, like January, kind of in the follow up to, I'd had the GRPO demo thing go viral. And I was like, oh, wait, there's something to this format reward thing.

• 29:11Speaker 02

  It was literally like a GitHub Gist, right? Or something?

• 29:15Speaker 03

  This is like a proper repo.

• 29:16Speaker 02

  No, no, no, the GRPO.

• 29:18Speaker 03

  The other one, yeah, yeah. The other one was like just a Gist. This one is like repo for like multi-turn tool use RL with GRPO. And so like, in some ways, the paper is like, it's the first paper that's really like, actually, there's been a couple of the papers that people have used the repo for, but it's one where like a lot of the stuff from the original, like GRPO demo Gist gets kind of extended to the multi-turn RL tool use setting. And so there's a lot of experiments here about like, okay, how do you actually get models to use tools? How do you incentivize tool use? Because something we'd see is that if you set these models up to use tools. They just won't. Like if you say, hey, here's a question, you have access to these tools,

• 30:03Speaker 03

  do as many rounds of tool calling as you want and then submit your answer. They'll just submit their answer because they like are, especially for like small models, like they aren't already trained to use tools. They don't really want to because they don't necessarily have that instinct. And they're pretty bad at like function calling and format instruction following. And so what you would see is like, when they use a tool, they would like mess up the JSON. And then they'd be like, oh, that didn't work and it threw me, it got me out of focus. And it would be more likely following that that the model would just like go off the rails. Because they would get like an error message from the parser. And so the safe option for the models is just to like stay in this basin of like, just do think then respond. Same with like normal formatting awards too. Like if you want models to use thinking tokens,

• 30:48Speaker 03

  you kind of have to like incentivize that. You're, you have to either do a little bit of like SFT warmup or you have to like reward them for doing it. Otherwise, they will not follow it a hundred percent of the time on format alone versus like a model like R1, like a hundred percent of the time it is going to use its think tokens. You are not going to ever see R1 just like talk normally without the thinking section. And so like you kind of do have to decide what you want the model to do. Like this is a little bit like a user facing question of like what, what behavior of the model should the default be? And if you have, if you want it to do a certain thing, if you want it to be a total use agent model, like it does help considerably to like actually have this incorporated into the reward.

• 31:31Speaker 03

  The kind of key trick in the paper to get around this problem. So, okay, one kind of reward these models would do is like, they would do like a dummy tool call where they would like, learn to ask this, use the same Google search every time and ignore it. So like, the, some questions would be like, okay, here's some like MLU style question. Go figure out the answer, use web search. And if you start rewarding them for like tool use, they will use the tool, but they don't really want to like have to, they want to like be very safe with it. And a lot of these questions, like they do kind of know a lot of the answers already. And I think calibrating the right difficulty of your questions for RL is like an important problem that we're still kind of figuring out. But They would like do silly versions of tool use where they aren't actually using the

• 32:14Speaker 03

  tool to assist in their reasoning. They're using it to get the reward. And so we kind of had to do a credit assignment thing of like, okay, did the tool result in information? And so for for these experiments we were doing, the trick was like, okay, does the like some string matching thing involving the ground truth answer and the returned search results from Wikipedia? So did the model actually search a thing that retrieved useful information for a question? And so this is like, but the framework is more general than just that. It's that, um, once you have a way to do intermediate evaluation, if you can evaluate like the quality of an intermediary state, now you can kind of rewrite the GRPO advantage calculation to take this into account. Because I think

• 32:57Speaker 03

  this is less of a problem in like PPO, you know, PPO is like the old school RL and it also is what people use for RLHF. But in the context of GRPO, GRPO is like great for like leaning heavy on highly parallel inference compute. It's more memory efficient for the actual training process. It's much easier to do in a distributed fashion because you have less gradient syncing and less model weight copies. It's kind of like DPO and steroids, I think, is one way to think about it. But it's also gets around a lot of the pitfalls of DPO, both in that it's like online by default as well as that you have this large set rather than just a pair of completions. So you do get like some intermediate credit assignment a little bit via this group comparison. But

• 33:40Speaker 03

  for tool use, it seems to be far enough out of distribution of small models, especially doing incorporating this turn level. So the way that I've been thinking about it is like, in like canonical RL, the state action are like things that you do many rounds of like, take an action, go to a new state, take an action, go to a new state. And for a while, people thought about LLM RL is like, oh, each token's action and the new sequence is a new state. And you can kind of do that. But you can also think of each turn as an action. Yeah, that's more likely. Where the state is the response you get back from the tool call. Um, and now you like have a different way of designing your RL algorithms to take into account credit assignment, which is that like, and it also is like a little more flexible from a reward

• 34:25Speaker 03

  perspective. So like, Feels like people are moving in the direction of model-based rewards where you either LM is a judge where the judge sees the correct answer or it has questions it's supposed to verify as properties of the response. Just because that's much more flexible than like trying to write these little parsers like writing a math parser to check if a math question is right is like not that easy actually. Because there's so many edge cases and you want to handle like latex support and markdown and like equivalent fractions and it's like just like let a model do that. Don't, don't have a 2000 line Python script that does that.

• 34:59Speaker 02

  Sorry, and let me, let me clarify. Math parser to verify that the math is right and you have a latex parser inside it?

• 35:06Speaker 03

  So yeah, so like a lot of models naturally will like think in latex because they've been trained on a lot of archive like tech.

• 35:11Speaker 02

  I didn't know that. Yeah, I guess.

• 35:15Speaker 03

  If you're doing like an R1 and people are like, oh, math is easy to verify. The easy to verify still is usually like this very long piece of code that has to handle lots of annoying edge cases and even then it's like 98%.

• 35:27Speaker 02

  Yeah.

• 35:29Speaker 02

  Hmm. Okay.

• 35:30Speaker 03

  Because like it's a free form response that is like, there's not only one way to write an equation like if you have two valid mathematical expressions that are equivalent, but they're also like symbolic. Like you need to verify the two symbolic expressions are correct, one of which might be written as code, one of which might be written as latex. One of which might be like written as word, like you can't do it if it's words with these like literal parts. They try to cover a lot of these cases. That's also why you'd see models put boxed around their final answer a lot is that it's, it's one hack is that it's much easier to kind of verify the right piece of the information if you know exactly where it's going to live. Rather than like the model saying the answer to the question is four. Then you have to like parse away the answer to the question is and just throw that out and so it's like

• 36:17Speaker 03

  deterministic rewards are like nice if you can get them to work, but they're also really painful and they're pretty hard to generalize across domains. Like for math, the easiest is when the final answer is an integer and lives in the same spot. Like there's a box where that's going to be an integer. And so this is one of the reasons like everyone used GSM AK for so long is because it's like mostly integers. I think it, Amy is all integers. It's super easy to verify these things and deparse them.

• 36:41Speaker 03

  But as you go to and multiple choice too, multiple choice is super easy to verify, but anything that's a little bit more flexible. deterministic like rule-based rewards start to break down.

• 36:52Speaker 02

  Yeah.

• 36:53Speaker 03

  And but the model based direction seems to be pretty promising. And I think underexplored for like, what if you use an L M as a judge in your R L loop? I think kind of going back to like Anthropic has been talking about this for a long time via Constitutional AI. In that case, it was less about the L M judging and giving a direct like reward to the model and more about training a reward model that was doing like token level advantage estimates, which is the PPO way of doing it.

• 37:19Speaker 03

  But it seems like you can kind of do that for GRPO too and other flavors of R L where you can incorporate.

• 37:24Speaker 02

  Full reward model?

• 37:26Speaker 03

  The reward model can basically be an L M where like it's fine tuned to like be more calibrated maybe and to have the right kind of range of responses.

• 37:34Speaker 02

  Yeah.

• 37:35Speaker 03

  But you could also have it be a reason or you could have it be something that is able to do tool calling. There's, there's no reason why the full power of L M's can't be offloaded or can't be also given to the process of evaluating whether or not an answer is correct or satisfies a certain set of criteria. And so I think like, and that's the direction I'm like most excited about is like really pushing on kind of beyond deterministic, rule based rewards into like these more flexible things. And I think you want to do this both at like a, so okay, that paradigm is not going to work super well with a turn level, with token level rewards, but I think it does work with turn level rewards of like, can the L M verify like whether a certain search query was useful? Sure.

• 38:21Speaker 03

  Like, there's a lot of these questions that are pretty granular that L M's can like basically nail all the time. If it's a good enough L M.

• 38:28Speaker 02

  Yeah.

• 38:28Speaker 03

  And you can incorporate that into R L with that sort of.

• 38:33Speaker 02

  Awesome. I think that was all the, you know, topics that we had prepped. Lassio, I think, I think you're, you're also pretty good on that. Obviously it'll, it'll take, it'll take some time to figure out Cloud 4. Anything you want to plug? We already talked about your talk, I guess, coming up.

• 38:47Speaker 03

  Sure. Yeah, I'll be an engineer on.

• 38:51Speaker 02

  Do it in fourth?

• 38:53Speaker 03

  In a couple of weeks, yeah.

• 38:54Speaker 02

  Yeah.

• 38:54Speaker 03

  Coming up.

• 38:56Speaker 02

  Your track is particularly hype.

• 38:57Speaker 03

  Thing. Yeah, mine's gonna be, that's gonna be a lot of fun. I'm also collaborating with Kyle Corbett from OpenPipe to do a course, which is both of us like have our open source projects that we like are agentic RL focused and kind of we've been friends for a while and are trying to do something that's a little more structured as like a way of kind of getting information out into the world for people who I think we're especially thinking about like kind of practical use cases for agents and helping people, giving people an outlet to learn more about like how the stuff works and yeah, more coming soon about that.

• 39:30Speaker 02

  Awesome. Well, I think that's it. Thanks for coming on, bro. Yeah, thanks for coming on at very short notice. I'm glad we can make this happen. We'll do part two with Calo and do a full Prime Intellect thing whenever you guys are ready.

• 39:41Speaker 03

  Awesome. That'll be fun. Great.

• 39:43Speaker 02

  Awesome.