Everything You Wanted to Know About LLM Post-Training, with Nathan Lambert of Allen Institute for AI

In this episode of The Cognitive Revolution, we dive deep into frontier post-training techniques for large language models with Nathan Lambert from the Allen Institute for AI. Nathan discusses the groundbreaking Tulu 3 release, which matches Meta's post-training performance using the LlAMA base model. We explore supervised fine-tuning, preference-based reinforcement learning, and the innovative reinforcement learning from verifiable reward technique. Nathan provides unprecedented insights into the practical aspects of model development, compute requirements, and data generation strategies. This technically rich conversation illuminates previously opaque aspects of LLM development, achieved by a small team of 10-15 people. Join us for one of our most detailed and valuable discussions on state-of-the-art AI model development. Check out Nathan's Lambert newsletter: https://www.natolambert.com https://www.interconnects.ai Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess SPONSORS: Incogni: Take your personal data back with Incogni! Use code REVOLUTION at the link below and get 60% off an annual plan: https://incogni.com/revolution Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers13. OCI powers industry leaders with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before December 31, 2024 at https://oracle.com/cognitive 80,000 Hours: 80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/cognitiverevolution to accelerate your career and contribute to solving pressing AI-related issues. RECOMMENDED PODCAST: Unpack Pricing - Dive into the dark arts of SaaS pricing with Metronome CEO Scott Woody and tech leaders. Learn how strategic pricing drives explosive revenue growth in today's biggest companies like Snowflake, Cockroach Labs, Dropbox and more. Apple: https://podcasts.apple.com/us/podcast/id1765716600 Spotify: https://open.spotify.com/show/38DK3W1Fq1xxQalhDSueFg CHAPTERS: (00:00:00) Teaser (00:00:59) Sponsors: Incogni (00:02:20) About the Episode (00:05:56) Introducing AI2 (00:09:56) Tulu: Deep Dive (Part 1) (00:17:43) Sponsors: Shopify | Oracle Cloud Infrastructure (OCI) (00:20:38) Open vs. Closed Recipes (00:29:48) Compute & Value (Part 1) (00:34:22) Sponsors: 80,000 Hours | Notion (00:37:02) Compute & Value (Part 2) (00:42:41) Model Weight Evolution (00:53:16) DPO vs. PPO (01:06:36) Project Trajectory (01:20:39) Synthetic Data & LLM Judge (01:27:39) Verifiable RL (01:38:17) Advice for Practitioners (01:44:01) Open Source vs. Closed (01:49:18) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  It's probably not worth the effort to spend all your time on preference tuning when you can just be making better data and better pipelines, which is what 2.3 is about, inspired by the transition we're seeing with the Llama Report, with Chatbot Arena, it's like turned into a hockey stick again, where we have these incremental scores and the OpenAI and Google are like skyrocketing their scores. And the philosophy is like, how do we try to understand what the open groups should be doing? And where there are hills to climb when you're increasing the complexity substantially of post-training? If you can have humans and LLMs do preference data, which do you send to humans versus LLMs? And that, I think, solves a lot of the problems, which is like, there are definitely things that we want humans giving the answer on, but there are a lot of mechanical tasks that we can outsource to LLMs.

• 0:44Speaker 00

  There's a lot more post-training that is not really touched. And I think that the opportunity is high. Because the big thing is how do you develop character? Character is something that you don't have a valuation for in your models. And our models, if you compare them to Claude, will not have as consistent of a character.

• 1:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Nathan Lambert, author of the popular Interconnects newsletter and machine learning researcher at the Allen Institute for AI, which today is releasing TULU 3, one of the most comprehensive open source efforts to diffuse the understanding and practice of frontier post-training techniques for large language models that we have seen to date. By systematically working to match Meta's post-training performance using the same Llama base model and sharing all of their findings and data publicly, Nathan and the team at the Allen Institute have illuminated what has historically been one of the most opaque aspects of large language model development. And this conversation represents one of the most detailed discussions of this topic

• 1:43Nathan Labenz?

  that you can find anywhere online today. We cover the full spectrum of post-training techniques, including supervised fine-tuning, multiple flavors of preference-based reinforcement learning, and a new technique called reinforcement learning from verifiable reward, which rewards the model for accurately answering questions with objectively correct ground-truth answers. At each step, we dig into the practical details that make these techniques work, the associated compute requirements, data generation strategies, and the value derived from each, as well as the experimental designs that are used to measure performance while exploring the vast space of possible training recipes.

• 2:22Nathan Labenz?

  We even explore some fascinating emergent behaviors that echo the frontier reasoning capabilities that we've recently seen from OpenAI's O1. Nathan's frank discussion of both the technical and organizational challenges of this work, including in a few moments where he acknowledges aspects that are not yet well-understood, provides a super useful window into what it takes to develop state-of-the-art models. And the fact that they ultimately succeeded in matching Llama performance with a team of just 10 to 15 people makes their approach one to study closely. Now, how long this level of open development can continue into future generations of models remains, in my mind, an open question. Even with billionaire estate backing, human-generated preference data and annotations are cost prohibitive for the Allen Institute.

• 3:07Nathan Labenz?

  And the synthetic data generation techniques used in this project may or may not be available going forward if frontier developers follow OpenAI's lead and choose not to release their O1-style reasoning traces. That said, Nathan expects that the community will ultimately figure something out. And just before publishing, we've seen Chinese AGI company DeepSeek announce a new O1-style model called DeepThink, which does seem to show its work and is reportedly going to be open-sourced in the near future. A development that could shift the question from whether or not such open reinforcement learning projects can continue to whether or not they should. And which should definitely cause anyone who thinks that Western companies can maintain a comfortable lead over Chinese companies from here all the way to AGI to stop and

• 3:50Nathan Labenz?

  rethink their assumptions. In any case, this episode is one of the highest-leverage pieces of content we've produced on this show to date. It's full of concrete, practical insights that were previously scattered throughout the literature or simply hidden behind closed doors. And I am really grateful to Nathan for such an open and technically detailed conversation. If you're finding value in the show, we appreciate it when listeners take a moment to share it online, post a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. And your feedback is always welcome via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy this super deep dive into the frontiers of large language

• 4:31Nathan Labenz?

  model post-training with Nathan Lambert of the Allen Institute for AI. Nathan Lambert from the Allen Institute, creator of TULU. Welcome to the Cognitive Revolution.

• 4:42Speaker 00

  Yeah, thanks for having me. The Nathan Squared Pod has happened.

• 4:46Nathan Labenz?

  I can't believe it's taken this long, but I'm excited that the day is finally here. So a lot of ground to cover. I'm just gonna fire questions at you rapid fire and let's see how far we can get. For starters, tell us about the Allen Institute. You guys have put out a lot of stuff. I would love to just get a little context on kind of the philosophy behind it, the, you know, the funding, the, you know, the GPU. Are you GPU rich or GPU poor? And kind of, you know, how are you looking to put a dent in the universe?

• 5:18Speaker 00

  Yeah, so Allen Institute, I'm new blood here. I'm, I joined it just over a year ago. The institute is actually 10 years old. It was founded, it's funded by Paul Allen, which is not surprising for people in the tech scene and knowing the Allen Institute is in Seattle, given the name. And the original CEO, Orin, I don't know how to pronounce his last name, also was a professor at UW, a lot of UW ties, I think was buds with Paul Allen. And this kind of grew out of that to just kind of do AI research for the common good, make things very open, very hybrid academic industry. That is kind of the story of the first decade, roughly, is not quite the first 10 years. And now there's like Orin left, the process of leaving started a few years ago.

• 6:00Speaker 00

  And there's the new CEO, Ali Farhadi, also professor at UW, was at Apple before. Most prominent work from his group, I think, is the YOLO line of work in computer vision. And it's kind of bringing in new blood to make this transition to build open language models. I think with a lot of, a lot of institutions realize that in order to be credible in AI, you need to have some amount of cred in the language modeling space. And right now for AI research, it is to be like, look, we can do this too. And it is a narrowing of scope to try to release great models and build models. I think OLMO is the start of this, which was, it was a very long story in 2023 that I was not a part of most of. I joined late in the year and launched in early 2024.

• 6:43Speaker 00

  And then 2024 has been the year of like doubling down on this and scaling up projects that you can do when you have more people. I think we'll talk about the post-training side of things. And there's plenty on pre-training and stuff in Val's as well. But it's really like what the academic projects in open source AI look like when you can bring more people in. And I think. Throughout the year, a lot of our projects have been getting bigger and we've seen like Momo, the vision model, which is primarily on the like vision team, but gets involved with the language pre-training team and other things. It's like these projects getting bigger and trying to increase the scope of research while doing as much as we can in the open. So no constraints on what we can say. At the end of the day, nonprofits are about telling a good story and stories are what can actually like change how policy

• 7:25Speaker 00

  happens and how people view the world. So at the end of the day, the Allen Institute needs to build good models so we can communicate why open source has benefits. And it's like a kind of reductionist and it's a little cynical to view nonprofits as storytelling orgs, but when your money mostly comes from a tech billionaire's estate, like you gotta stay some of it as it is and but the momentum is good. So it's fun to be in the space and see things continuing for the time being.

• 7:54Nathan Labenz?

  I think stories are honestly a super important part of life in general and the AI moment in particular. It feels to me like we are, you know, summoning this crazy new alien intelligence and really lack in most corners a positive vision for what we want to do with it, you know, what impact we want it to have on life. And there's a lot of like very general talk about, you know, won't it be great when we have everything we want? And it's like, yeah, could we put a little more detail on that? So I do appreciate the importance of storytelling in this space. You mentioned pre-training and post-training.

• 8:39Nathan Labenz?

  The project that you've just put out today as we record this, you know, a few days in advance, but we'll try to publish it right around the time of recording is called TULU. It is really a deep dive into post-training techniques and

• 8:55Nathan Labenz?

  Give us a little bit of context for why this project, what the goals of it are. And then I really just want to go deep on the actual methodology, what you've learned, and try to develop my own intuitions for the state of the art in post-training.

• 9:08Speaker 00

  Yeah, so I think this story fits well with what we were talking about with AI2. So AI2's open post-training recipes have been under this TULU brand for over a year, almost a year and a half. I think TULU1 was back when Open Assistant was new, was like, what can we do by mixing all these popular data sets that people are putting out in 2023? All these data sets and small models, like, how do you systematically mix instruction data sets to combine them with open resources? And TULU2 was around when DPO was very popular, and that was when we showed that you can scale DPO to 70 billion parameters and continue getting the benefits from preference fine-tuning and open recipes. Then we went away and did TULU 2.5, which was trying to answer the PPO versus DPO question,

• 9:52Speaker 00

  which the TLDR was if you tune your parameters right, PPO is probably better a bit, but it's probably not worth the effort to spend all your time on preference tuning when you can just be making better data and better pipelines, which is what TULU3 is about, which is inspired by the transition we're seeing with the Llama Report with Chatbot Arena. It's turned into a hockey stick again, where we have these incremental scores, and OpenAI and Google are skyrocketing their scores again. Nematron paper, Apple had a paper, which these post-training recipes are just much more complicated than big SFT step, do DPO on some chat data and be done. And the philosophy is we know these labs are doing it. How do we try to understand what the open groups should be doing and where there are

• 10:36Speaker 00

  hills to climb when you're increasing the complexity substantially of post-training?

• 10:42Speaker 00

  Llama 3 has hundreds of peoples on it and a large amount on post-training. We have 10 to 20 on ours. And it's like, what could academics actually do to do modern post-training? And curate new data, use new algorithms, sequence things together, but like kind of move beyond this paradigm of like just increasing like alpaca eval or vibe scores and try to get really specific on improving math, improving IF eval. Code is something we did a bit, but it wasn't the biggest focus. I think our valves for code were mostly saturated pretty early on, but kind of shifting this post-training into a much bigger domain of academic progress and kind of.

• 11:18Speaker 00

  Honestly, there's been a slow in how often people are releasing models and data sets and kind of like reboot that. I think the DPO era was a very fun one for seeing open post-training models. It was it was bonkers. Like every week there was a new like DPO model that was state of the art on something. But that was like last December, last January, and it feels much slower now. It's much more big players dominating the AI narrative and it's trying to reset that.

• 11:44Nathan Labenz?

  Yeah. So one of the things that I noticed in the materials that you shared is that the goal of the project was to beat Llama 3.1. And you're starting, if I understand correctly, with a couple of different base models, but one of the base models is Llama 3 base, basically, right? So you're essentially saying, okay, if we take the same pre-trained version that you folks at Meta started with, can we match you or exceed you, hopefully with. Post out of post-training to post-training, apples to apples basis.

• 12:18Nathan Labenz?

  They obviously put out a big report and, you know, shared a lot about what they did, but I believe they don't share much or any of the post-training data. Give us a little bit more. I'm like, what do they share and not share?

• 12:30Speaker 00

  I would say they share an outline of their method. They don't have a lot of, like, I might have hyperparameters, but not really clear without noting their code base and things like this. So they share a high level outline of like, here is our approach. And here are the types of things that we do, but not the specific things that we do and definitely not the data. So the biggest differentiator for our post-training is that we release the data. I think we built like three to six new data sets just for this project and we're using those with the combination of already existing data sets and we will re-solve them at once. But really, what you're saying is right. It's like the development model here was we had a set of evaluations we cared about that we tried to cover things like factuality, knowledge, math, code, instruction following, safety.

• 13:10Speaker 00

  And the goal, the primary development target was Lama 3 AP where we effectively trained about 1,000 AP models to try to get these scores to be higher at AP and then validate them at 70B. And we're also adding like OMO models, which is our, like makes sense. This is our main thing, which is like trying to like make sure the recipe translates. So for that reason, it's like kind of a basic goal is like, okay, we like, in order to know you're in the right ballpark, you need to beat these people that are known to be pretty good. I think I've talked to, like I've talked, the fun thing about LlAMA base models and instruct is that the closed labs can also

• 13:49Speaker 00

  validate their pipelines on it. So like the closed labs can take the LlAMA 3 base and then use their post-training and compare it to the instruct model that Lama also released. So hearing from some people in industry, they're also like, yeah, Lama 3 instruct is like 3.1 instruct. The latest stuff was like pretty cracked. Like it was a pretty good post-train. I think obviously like OpenAI's infrastructure isn't going to work as well on LlAMA base models, but the 3.1 models particularly actually were very good. I think it's clear that there were some ways that they were rushed or trying to focus on systems rather than the novelty. It was mostly SFT and it was DPO for the Lama 3.1 instruct. And I know they are expanding into more things now.

• 14:29Speaker 00

  And we're kind of, by being smaller and moving fast, I think we're actually going to do some, we're doing some things that I bet will look pretty similar in the Lama 4 instruct or like a Lama 3.5 instruct type thing. But that's kind of like the theme is we want to beat their numbers. We're doing similar things. We have the advantage of being smaller. We can be a little bit more clever, maybe. But that was the development target and it really is like. I can look at the days, I don't have the dates in mind. It's like, if we started this project four months ago, it's like, oh, a few months in, we passed Llama 3 instruct. And then like a month and a half later, we passed Llama 3.1 instruct to 8B. And then like a couple of weeks later, it's 70B, and it's like just metrononic. And it's like, every time, it's like, oh, wow, this is actually happening.

• 15:13Speaker 00

  So it's fun, it's fun to see, and we're very excited to see what people build on it. And I don't, like, we might have a discussion later on, which is like, fine tuning from base versus fine tuning from instruct, which is another kind of, there's another fork. And I think fine tuning from base is very academic and very important to post-training, to understand how to do this from base models. And fine tuning from instruct is a very important area, especially for new companies and domain-specific things. Because if Llama is going to put out a very strong instruct model, you don't have to do the whole 2LU 3 suite to make it good at your specific domain. It's like, we're doing part of this, but eventually, there's going to need to be research there. I think on the day of recording, Nexus Flow, which is a startup that came from,

• 15:58Speaker 00

  like, some of the people are from Berkeley Nest, the group that released the Starling models. They released another fine tune on instruct, and we saw a Nematron fine tune on Llama instruct to go very viral recently. So there is some messaging that, like, we're not doing this fine tune on instruct, but it is another area. I just think in terms of like fundamental, long-term ecosystem building, the from base is still the thing that needs the most transparency.

• 16:23Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:28Nathan Labenz?

  So let's get into the real nitty-gritty here. I have a general sense, I think most of our listeners will have a general sense of, you know, certainly, pre-training, understand what that is, you know, web scale data, next token prediction, okay, cool. Then you've got your typical recipe, there may be some exceptions or caveats to this, but typically, next comes instruction tuning, and then comes some sort of preference tuning. And that's where like reinforcement learning from human feedback fits in. And then there's different algorithms, and you've mentioned PPO and DPO, that can be used to actually do that final stage of preference tuning.

• 17:08Nathan Labenz?

  That's the baseline understanding. Give us the next level of understanding. You know, what's most important to understand after that?

• 17:16Speaker 00

  Yeah, so the, because I'm gonna jump to the real nuanced thing, which you might not even have in your questions. It's like, what are open recipes doing and what are closed companies doing? In the open, we have the benefit of being able to train on outputs from very strong models like GPT-4, which is something that we did. There's some amount of legal uncertainty there, but we definitely put it, the wording in our datasets is like, look, you're understanding, you're training on outputs from these models. But that definitely gives an advantage in some way. Llama, in order to get a model like GPT-4, needs to build Llama 405B. They need to do instruction tuning over their various base models and all of these things to kind of get this distillation model to work from.

• 17:56Speaker 00

  So they are trying to do a similar thing, which is a lot of the instructions are written by a very strong model. So a lot of math and code instruction data is now written by for Llama, Llama 405B, for OpenAI, GPT models. And in that way, we are doing something similar, but in the open, you can kind of take shortcuts, which is like, we don't need to do this whole pre-training. We just go right to the model, which is best for SFT. I think that stage actually looks very similar. I think we don't have as good of control over which prompts we are using. I think the closed labs will do a lot of filtering and controlling of the distribution. We do a lot of mixing, but it's mostly on taking the levels of known datasets and seeing what their performance is on downstream evals.

• 18:41Speaker 00

  It's largely a process of adding in a dataset to instruction tuning for a specific capability and making sure it doesn't degrade other performance. So if you have like a math eval, which is like the all capital math eval, which you can improve this, but a lot of it comes down to the formatting. So some math datasets will have different formatting in the completion that will make it harder for your model to learn from that, or it could affect something like GSMAK or coding or stuff like this. So these are the kind of things that you're doing. And then also at the end, you're probably going to try to include some more general chat data. We included 100,000 multilingual samples from Koher's Aya, because we know Chatbot Arena is, has a decent amount of multilingual, even though multilingual wasn't in our eval suite. And there are some other safety things which are more borderline, which is just like,

• 19:26Speaker 00

  how should a model behave? And this is kind of what happens in SFT is like, there's some, there's some art to it in squashing second order effects. But I think largely that's Really similar to what the closed labs are doing. And then at preference tuning, it takes a kind of big turn, which is the closed labs use humans for their preference data. We do not have the money, so we use all a LLM as a judge to collect our preference data. I like to quote John Schulman saying on this, which is the pithy way to say it is like, "Human preference data is high noise but low bias, and LLM preference data is low noise but high bias." We do not know exactly what bias we are getting, but largely we can still see pipelines of,

• 20:06Speaker 00

  we collect our own preference data from various model generations, language models label it, the preference data is good. The biggest change here is from data sets like UltraFeedback, which exists on Hugging Face. We essentially redid their pipeline with completions from the models we had trained at SFT, and that gives a meaningful, like, low percentage improvement over just using a random preference data on the Hugging Face hub. So it's higher effort. You have to go through the effort of generating completions of LLM, making sure that's all right, getting diversity, doing LLM as a judge. But that type of thing gives us, like, we give, across our experience, time and time again, this, like, on-policy idea of preferences, even with LLM as a judge, was better. So I do think that that's kind of what, if you look at Meta's system diagram in their paper, they show this.

• 20:51Speaker 00

  They're like, "We take a new model, we pass it in, we get new preference data, and we train on it." The difference is that they do multiple iterations, which could be based on how they get data, could be based on how their timeline is and stuff like this. I do think multiple iterations is something we're seeing again and again, and we didn't look at, but just kind of checking the box of like, okay, on-policy data does work for preference tuning. It's not a huge paradigm shift, but it's a whole different approach than people have to do. We've seen some of this in kind of like on-policy and new, like, online DPO algorithm variants that everyone kind of gets, like, spiralized. It's like, "Oh, God, we have some other star PO algorithm I can't pay attention to." But the algorithms were going in that direction, where they're doing more of, like, generation from the model and labeling.

• 21:33Speaker 00

  The way that we did it, I think, is a bit more segmented, which is kind of like closer to what Llama is doing. This, at a high level, is similar to what Llama 3.1 does. I've heard that they weren't doing any fancy verifiable RL, any O1 stuff on theirs.

• 21:51Speaker 00

  What we kind of added to this, I think I can, specifically this is in the paper, like you can look at the acknowledgement section of the paper and you'll see like one name and a bunch of normal grant garble and gobble. It's not in the draft I sent you, but like there's one name there and he told us to just do RL on verifiable outputs. So this is the stage three that we did, which is essentially taking the training set from things like math, GSM 8K, and it's actually like IFAL is verifiable because you like count if the constraint, IFAL as a summary is a bunch of, is an evaluation where there are constraints on prompts like respond, make sure your response has X word, make sure your response has X paragraphs, and these are things that are verifiable

• 22:36Speaker 00

  in Python code. So across these things like math, instruction following, we have a system where you have these prompts with constraints and then we used RL to just give a reward if the constraint was satisfied. And we have seen on multiple models that essentially can improve GSM 8K, you can improve math, IFAL is a little bit trickier, there's a bunch of RL details there. But we essentially do this at the last stage to be able to, if our DPO like worsened our math scores, we can bring it back up. And with our pipeline where the DPO data is very trained to our models, we actually get most of the math improvement there and the last RL stage is pretty minor. But if you take some old RL model off the pugging base, you can apply this like RL verifiable rewards thing to

• 23:19Speaker 00

  it and get like a 15 point boost in GSM 8K without a ton of degradation in other valves. So it's like, we're really just scratching the surface there, but the winds of AI, you can see that it is going this direction. There's murmurs that a lot of big labs do stuff like this. You definitely think they're doing it for code. We have an added code to code interpreter. O1 is described as a large scale RL system specializing in reasoning tasks. It's like, huh, they're probably doing something like this. I think. One of our models just we left running RL for this math really long and it started doing this like, let me check my answer again, like redoing redoing chain of thought within the chain of thought. It was like literally the thing that OpenAI was showing us where it's like, wait, let me check that.

• 24:02Speaker 00

  And this is definitely not O1, but I think we've seen other papers in the space coming out. There's like Vine DPO is one that's really Vine PPO is very similar. Let me get their name right. So it's easier for people. People have talked about Quiet Star. That's a bit more complicated. I think Trice is also one, which is also a bit hard to understand, but motivated by the same thing. So the literature is going in that direction, but we kind of showed like, look, you can do this in your preference tuning. You don't have to just do a math model. Like you can just add on these types of RL at the end and it doesn't blow the model up. So to kind of summarize, it's like, as we go, we've unlocked, we have these three stages, SFT, DPO and RL. And each of them, we're kind of taking more alpha from what industry is doing.

• 24:47Speaker 00

  Like SFT is known, it works. DPO, there's some new tricks that we needed to uncover uncover parameters, different settings. We use this like a linked normalized setting, but that's mostly known, but good information. But the RL stuff is like, look, this is like real, we're beating Lama numbers. We used RL like it's still mind blowing to me that RL is becoming more relevant. I think like 2023 is like, oh, where's RLHF going? But now it's not even just RLHF. We didn't even have a reward model in this. We can go do the technical setup later, but there's no reward model. It's just an RL value function and it's fine. So that's a summary of the training. It's like probably somewhat dense if you haven't heard these terms before, but also informative.

• 25:31Nathan Labenz?

  Yeah, well, I like that. Let's, we're going to unpack it a little bit, but that's a great initial overview. So I'll maybe just come at this from a couple of different angles to try to understand the overall landscape better. For starters, like. How much compute and how much value goes into and comes out of each of these stages? Maybe compute and data, like, you know, just relative amounts and relative value that you get from each one?

• 26:02Speaker 00

  Yeah, so this is like a general purpose instruction model, which I have to caveat with it. Because I think if you have different domains, you have much different compute. I think SFT is something we talked about a lot. The data set largely kept growing. We didn't do a lot of sub sampling results because we're like, we're searching for high numbers. Our final mix is about a million prompts. Most of them are single turn. This model will not be as good at Llama as at multi-turn. We're not a meta AI shop. We don't need that as much. It's, we don't have the valves for it. But it's about a million prompts at SFT. If you're using, I can give really specific throughput numbers. If you're giving, using 32 H100s, you can train a 8B model in about a day on our code.

• 26:47Speaker 00

  Our code is not super optimized because it relies on transformers. I think you can get about a 40% speed up if you're using kind of really specific code, which is something we might do in the future to only do like Olmo and Llama architectures. Because if you have, if you support fewer architectures, you make it much faster. So that's like a million prompts.

• 27:07Nathan Labenz?

  So that was 32, 32 H100s is like a 24 hour train cycle to train on a million prompts?

• 27:15Speaker 00

  Yes, for SFT.

• 27:17Nathan Labenz?

  And I think it's about like- At market prices, that's what, two, let's say, two bucks? Prices have come down, as I understand, two bucks per H100 per hour? So 60 bucks an hour, 1500 bucks kind of compute cost?

• 27:32Speaker 00

  I wasn't expecting this number, but yeah, yeah, yeah. I think it's reasonable, let's ballpark that, like ballpark a thousand dollars. Because I do think we could probably get 90% of the performance with half the prompts. I think, looking at our mix, we have like 30 plus percent math.

• 27:48Speaker 00

  We were going hard on math, trying to get these numbers across. We ended up, we did like general math eval and then we have subsets for specific subsets of math, like intermediate algebra where we weren't as good. So we like looked at the eval and saw which micro subset was not as good and like tried to make data specifically for that. So like, we definitely, you don't need all of this math eval. So that's a good, that's a good rule of thumb. DPO and preference tuning, we have had, we haven't been able to show as strongly that scaling helps, like getting more prompts in and keeps improving evals. I think it'll be like order of a couple hundred thousand prompts. It's not as big. DPO does use more compute, especially at 70B. I think it's because you need the reference and policy

• 28:33Speaker 00

  model. Still, we've run these jobs and like, they take like six to 12 hours if you're using somewhere between like 16 or 32 GPUs, much faster than SFT, just because the dataset is much, much smaller. I think at 70B, we added the normal, we like changed our forward pass from the default like hugging face slash DPO implementation to make it a little bit more efficient. We do caching reference log probes. So you don't have to store both 70B models in memory at the same time. Because if you don't do these optimizations, it starts to look a lot more like PPO where you need 128 GPUs or more to do at 70B, which is like

• 29:18Speaker 00

  quickly, you can see how PPO kind of balloons what is going on. And it can take longer if you're trying to really get the best absolute scores. So I do think like SFT is by far the biggest compute because we have the most tokens. DPO, I would say ballpark a quarter of what we talked about for SFT, would be a couple hundred bucks. And RL is probably like, especially at 70B, can be almost similar to SFT if we really run it for a long time. But you can get most of the benefits probably in a similar amount of compute as DPO. So the RL curves look remarkably similar to like kind of old school RL tasks where at the beginning they get the most improvement and then it's kind of like level and bouncing around and like maybe going up a little bit. So if you do like one epoch,

• 30:03Speaker 00

  which is this first improvement, you're going to save a lot of the money. But we're like, oh, we're trying to get the best numbers. Let's let it run for a few more days and see what we're doing.

• 30:12Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 30:17Nathan Labenz?

  How about the eval improvement through that process? I guess if you have a base model, you have to basically few-shot. If you go to instruction tuning, then you could just do zero-shot or few-shot.

• 30:31Speaker 00

  Yeah, that eval setting is crazy. There's a lot of iteration on it. I would say that for performance, roughly, I think we get like 90% of our performance at SFT, and then the last 10% of mix of DPO and RL. And then like alpaca evals slash vibes things, you get the most from DPO and RL rather than SFT. But at this point, like our SFT mix beats on our eval suite, the Llama 3.1 numbers. But it's very focused on our evals. And I think the preference tuning kind of softens it a bit to be a nicer model to talk to. The eval suite is complicated. I don't know all the details off the top of my head. It's good to look at the paper, but we tried to do... There's this whole distinction between pre-training evals and post-training evals.

• 31:15Speaker 00

  And I do agree that most post-training evals should be like using a chat template. So you're generating from the model to generate tokens. A lot of them are CO2 chain of thought. And there's kind of a distribution over the number of in-context shots. I think there's some that are zero-shot, there's some that are eight-shot, kind of depending on the domain. And probably the trickiest thing to manage is answer extraction for reasoning, which is like, does the answer appear in the format that eval expects? Particularly on math, Llama 3.1 uses what's called the Minerva format in a specific prompt. And we use what is called a flex format, which essentially if the default writing is like the answer is in boxed, you also allow like boxed and the answer is colon and like one other thing.

• 31:59Speaker 00

  And this is mostly to be fair to like Quen. So Quen is a fun example where their 72B model gets a score of like 6 with Llama 3.1 setting, but a score of 74 with our setting. So it's like, okay, like we're like competing with Quen instruct in a way. Like we have to have a setting that is not just... What Llama does, and I mean, all of other labs are doing things like this, which is tailoring their training to eval. And it's, it's hard to know what they trained on. We did a lot of work to decontaminate all of our training data sets on the evals that we're doing for development. We also have an unseen eval suite. So we do a few methods to check for exact match and overlap on all of the training data sets that we used

• 32:40Speaker 00

  throughout it, which is some of the final data sets. And we're also going to release decontaminated versions of like other data sets along the way. So popular names like, like OpenInstruct, Nvidia's daring anteater data set has some contamination on math. For example, like the hugging face numina math T R R, T I R, which is tool integrated reasoning, which was for a math competition, has math contamination. So we had to remove this. It's like, they were using their model for a Kaggle competition, not for like fair evaluation. And it really goes to show, it's like, one, we're releasing the data, but two, we're showing like how easy it is to have contamination. So it's like, yes, we need more people to release it. So we don't know if

• 33:21Speaker 00

  any of the models we're comparing against trained on test, either on our development set or unseen set. It's just like, we can't know. We found a lot of contamination out there, but we don't know what other people are doing. And it's kind of, I think there was probably a snarky paragraph in the paper, which is like, we don't know which of any of these models trained on the evals. It's like, okay, we're sure the best.

• 33:44Nathan Labenz?

  When you're finding contamination, you're finding it in data sets that other people have open sourced, or you're using some sort of diagnostic to detect it in the model?

• 33:54Speaker 00

  Um, yeah.

• 33:55Nathan Labenz?

  Because we've seen some techniques like that.

• 33:56Speaker 00

  We're looking at data sets. People are working on model diagnostic, but we're looking at data sets. And like, there was this weird thing. Well, we'll talk about data sets. We're essentially, the biggest thing we're doing is exact prompt match. So if a certain data set, training data set has more than like a 2% exact prompt match with one of our test sets, which is like, that's the percentage of the test set in the prompts exactly, we consider that contaminated and remove them. The other thing is like, can you detect if models trained on certain test sets? So I, earlier in this year, my big project was Reward Bench, which is like trying to build an ecosystem for evaluating reward models. Now it is going, there's lots of academic papers on it. But one of the weirdest things we found is recently substantial contamination on

• 34:40Speaker 00

  Rewardbench prompts, which was taken mostly from other test sets, was generated by Lama Instruct with the Magpie method. So Magpie is a synthetic data method that manipulates the chat template to get the model to generate prompts in distribution of what it was trained on. So this is the type of weird head scratching you need to do to do like, is a model trained on something? You can't prove it, but if you have the weights, you can get the model to generate-ish prompts from its distribution, and then you can check those on test sets. Which is really funny, because Magpie is meant as kind of like a training architecture for generating new data. But when you then realize that that might be what you need to do for evals, I told the author and he's like, oh, cool, another use case for my paper.

• 35:25Speaker 00

  Which is really funny, but I do think that whole ecosystem is going to continue to grow a lot. Because you need it to have some source of truth, I think. Eval's are only growing in value and cost, and you need to be able to audit the models to assess this across industry. So there's going to be some mix of new startups, academic research, regulation, undecdomination. The whole thing is kind of a fair game.

• 35:53Nathan Labenz?

  Yeah, interesting. Okay. Let's go back to the post-training stages, and I want to get you to try to help me develop my intuition for how the model and how the model weights are evolving through that post-training process. I have a pretty nice little story that I tell people about the pre-training process where I'm like, given some text, the model's job is to predict what comes next. There is a ground truth as to what comes next. And so it outputs a distribution, and then the question basically amounts to, how can you, with the chain rule,

• 36:34Nathan Labenz?

  how can you tweak all the little numbers that are the weights in this model to just nudge yourself so that you're a little closer to correct? And we just do that a ton of times, and obviously there's batching and other complications. But the atomic unit is you make one prediction, you get a score, and you nudge yourself to be closer to having made the right prediction.

• 36:56Nathan Labenz?

  That gets a little more complicated at various, in various phases of the post-training process, right? Instruction tuning, from what I understand mostly, is similar, right?

• 37:07Speaker 00

  It's mostly the same. You have to add new to, it's the same, but you add in new tokens. So you add like the tokens to help constrain the model, which is like user or some other, like EOS token. You add these things in, in a specific format. So you have to learn a little bit of new tokens, but otherwise it's exactly the same.

• 37:24Nathan Labenz?

  So that's kind of weird, a little bit counterintuitive in as much as, at least for some sort of instruction tasks, and this is, I guess, you know, a natural segue to the preference training. You think like, what you really want is to be telling it what's good and what's not good, but with the instruction tuning, you're still basically just saying, this is exactly the ground truth tokens you should have predicted, and all the weight adjustments are just specifically toward that exact string.

• 37:51Speaker 00

  Yeah, it's almost like, for simple things, yeah, I don't think you need a lot of SFT data, because you're just like manipulating the weights to be more focused on this type of formatting. Like, you need to do enough so that this is the way the model outputs. But for specific tasks, like math, I think you can make a lot bigger gains, which is just like, the model has not seen a lot of this, and you're really trying to like, do the same thing that you do during pre-training, which is like, teach the model to do chain of thought math. Like, you need to have that somewhere to kind of kickstart things. I think pre-trained data probably has some of it, but I think that that's a balance that we'll see evolve, which is like, what is the basic amount that you need to do some instruction following? And then like, how much more do you need for specific capabilities?

• 38:32Speaker 00

  I think. At some point, a good thing for that, I did many times, I think this project is good for the readers, like, go look at the Lama report and look at the percentage of instances per domain at instruction tuning. And at instruction tuning, they have way higher math coding reasoning than preference tuning. So I think like, those are the domains where the model needs like, just more flops to understand the basic capabilities in SFT. But like, at preference tuning, it's much more of... It becomes a contrastive loss function, either through DPO, which is like pairs in comparing them. If you dig into the DPO math, it's like actually some weird double negative where it's like decreasing the probability of the negative of the rejected response rather it like does

• 39:16Speaker 00

  that more than increasing the probability of the chosen. There's weird oddities in the actual DPO math there. If you go really deep, PPO is almost more intuitive where you you generate new samples, you have a value model that assigns attribution to each of the tokens where higher number is good. And then it tries to increase the likelihood of things that it sees as being good. And in this case, it's guided by a reward model. In our case, it's guided by like, is the answer right? And it becomes much more flexible. I think DPO is somewhat restricted to the generations that you give it, but RL in that way is flexible, where I think it just can kind of change the behavior of the model more. I think when

• 39:58Speaker 00

  we apply math, we see that it takes different reasoning steps. We see this kind of different types of changes than you see at SFT and DPO. I do think there is going to be a lot more understanding of like what these stages actually do. And it's kind of funny because like SFT is where most of the performance gains come on evaluations from chat to capabilities. But it just does really feel like when you do these and when you look at the loss functions, that there's kind of more that you can change about the weights doing RL or doing this kind of direct preference optimization optimization sort of thing. And that's something that we've heard for a while. I think leaders in RLHF have said like RL is just a more flexible loss function. There's a lot more scale you can do. I think that was like the framing

• 40:42Speaker 00

  that somebody like Jason Way at OpenAI gave in a talk. It's like, look, like RL has a lot more legroom because you can just the loss function is very different and we haven't explored it. And things like O1 and kind of our own experiments make me more optimistic in this. It's just like, it's very weird. You can do a lot of weird things. Changes the model, notably the val scores don't go down that much and it will take a lot more intuition, both on the individual level and the academic level to like understand these parameter changes. Cause it's so like is fundamental to the fine tuning domain where it's like, RL is doing something very different than the previous loss function. In some ways, it's remarkable that you can just do SFT and pre-training and then you can just throw RL at it.

• 41:25Speaker 00

  And the KL regularization is enough to just like hold it in place, but like you just change loss functions 99% of the way there. And it doesn't break everything. It's like this like, that type of thing makes me give a lot of sympathy to the like whole Dario, whatever mindset of like, they just want to learn. It's like, there's something complicated going on that I don't have deep enough intuitions of deep learning to grapple with.

• 41:49Nathan Labenz?

  Yeah. Okay. That's really good. Let, let me,

• 41:54Nathan Labenz?

  just go back to a couple of points for quick wild ups. First, the KL regularization, that is basically a way of just anchoring to the earlier distribution, right? So your loss function has multiple terms and one of them is like, subject to getting better in the way that we want you to get better, which we'll come back to in a second. Also, like, don't change too much. Is that basically the right intuition there?

• 42:20Speaker 00

  That's what it is. You essentially, look at the log probes of the original model and the new model. And you make sure that the difference in probability is not too big between your RL model and your reference. And I think the way that we phrase it, it's like you have a KL budget. So you can like only change the model so much. And like, once you reach your budget, you normally don't expect the model to keep changing if you're doing value updates. Because like, it can't like, it can't make substantial changes. It's just moving around in the same neighborhood. Which I think is a nice framing. I wish more DPO, like DPO is different where it's like, A controlled KL distance through their beta parameter throughout like the number of epochs that they're doing, whereas RL is like, where doing online RL is a little bit more open-ended

• 43:05Speaker 00

  on the KL side of things. But I do think this in general and post-training, showing more plots of like, performance versus the amount of KL that you spend is very nice. I think historically, this just sounds like total random numbers to people not looking at this. It's like PPO KL will be like 10 to 20 scale type thing. This is when you're doing the full thing on chat, for example, or like GSM 8K, like really basic math KL is like one. So it's like, we're really, really not changing the model very much doing this RL for math compared to what you would do if you were doing PPO for everything. But if you also compare that to like best event sampling or something, best event sampling over a reward model also has way lower KL

• 43:45Speaker 00

  spend than doing this full like PPO online RL thing. And I don't know the intuitions off the top of my head for where DPO falls. But it's like all of these preference, that's like kind of a way to measure how much you're changing the model in preference tuning. It's like, what is the KL difference across a controlled set of prompts? There's a problem where we're in all of these numbers that I'm saying it's on a different set of training prompts. So it's almost like we need to have like a standard, like these are the prompts that we about, these 100 prompts are what we evaluate our KL distances on across different domains to really see how much the model is moving in general. But I think that's the additions. That's a good way that people can look at it.

• 44:24Nathan Labenz?

  That one that you're anchoring to is the original base model, or it's the instruction tuned model?

• 44:28Speaker 00

  Instruction tuned.

• 44:29Nathan Labenz?

  And that just stays, no matter how many like iterations you're doing?

• 44:33Speaker 00

  The reference model stays the same.

• 44:34Nathan Labenz?

  The entire time.

• 44:35Speaker 00

  There are people that, there are experiments where you can do like, do one round of PPO with your reference as the SFT model. And then you can like, start another one with your reference as the first PPO model or DPO. So people do do things like that. It's just not as popular. There's definitely some papers there, which is like moving, like resetting your DPO reference to give you more ability to learn. I don't remember the names off the top of my head, but that is the sort of idea that people fiddle with.

• 45:05Nathan Labenz?

  And these KL divergences, they are, the penalty gets, it's like a square function, right? So the penalty gets bigger, the more you sort of drift. And that's kind of where this like budget notion comes in is like, you sort of have a, a limit as to how far you can move before that term, I guess, starts to dominate, basically.

• 45:30Speaker 00

  This is like, what is, there's an approximation that you use in a lot of KL. In a lot of RL, you use an approximate rather than a full KL, where the approximate, you're essentially just, I have, I pulled it up. I have the silly like RLHF book that I'm working on, which is mostly like notes on fundamentals of RLHF. And like the regularization is this. And it's like, there's a difference between what a lot of people do in these implementations as an approximate KL. I think that, like John Scholman has a blog post on this that everyone references, where the approximate is you just subtract the log probes of a generation for the policy model. You do that minus the reference log probes, which is a different function than the full KL. I don't know if it's a change. It seems like

• 46:14Speaker 00

  that's a change between square and not. Just kind of thinking about it off the top of my head, without having the exact KL equation.

• 46:22Nathan Labenz?

  Interesting. Okay.

• 46:24Speaker 00

  So there's.

• 46:24Nathan Labenz?

  So I mean, presumably all of these things like could work and probably have, you know, work to some degree and have some pros and cons. But yeah, that, that sharp bottom versus kind of flat bottom is an interesting distinction. So I still want to get a little bit better intuition on how these multi-token evaluation and multi-token comparisons work.

• 46:48Nathan Labenz?

  And maybe we could break this down also by PPO, DPO. I'm getting like as a user, you know, if we start with the human experience, I either am presented to generations and asked to say which one I like more, or I might be presented one and asked to rate it one to seven, or maybe I presented two and asked to rate them one to seven or whatever. So that's kind of the, the, the signal is derived from that, right? Then we, in a lot of these schemes, train a reward model on that so that we don't have to always have a human doing the evaluation. So now we have a reward model that attempts to score in a way that the human would score, but it's still giving you this sort of,

• 47:33Nathan Labenz?

  this one or that one is better, or that, you know, here's a one to seven kind of rating of these outputs. The critical distinction there, I think, that I, that I'm still kind of struggling to develop like a clean story I can tell, you know, my friends who aren't so obsessed with this as I am, or certainly not as you are, is how does that score translate to updates when it has to go through this multi-token generation? Because I used to have, you know, with the pre-training and the instruction tuning, I have this simple, you predict one token, you get, you know, you can score that one token, you can update on that. Everything else is sort of aggregation of that. Here, the score applies to this whole generation. And so, you know, there could be like weird forks in the road,

• 48:16Nathan Labenz?

  or, you know, even if just like you add an extra token, now like everything is kind of off by one token. How do I compare base generation to the, you know, the current policy generation, given that I can't do it on like a token by token basis anymore?

• 48:34Speaker 00

  So the RL math is essentially you will give it a label based on the whole trajectory and then the value model, like... If you're doing PPO, the value model will take a generation and it will output a value per token. Then you will have the label from the reward model, which is like the reward from your environment. And that is used to update the value model to kind of have these per token updates. And then the policy is actually going, like in PPO, you're actually going to be taking attribution for every token in your batch and doing updates based on what they think will be a long-term better generation in PPO.

• 49:15Speaker 00

  That's very different than DPO. I don't have like a per token just understanding of what is happening in DPO because it's, if you think about the loss function, it's doing essentially all of the tokens are kind of grouped into chosen or rejected. And it's trying to increase the margin between chosen and rejected log probes across the sequence. But I think early in the DPO days, there were questions on like length issues because what you're looking at is the sum of the log probes. So then if you're like all negative, like they're all like negative numbers because it's a log of a probability. They're all less than ones. They all become negative numbers and you're summing them up.

• 49:56Speaker 00

  And it's like, what does that do? Because you're therefore like increasing the margin between these two negative numbers is what the loss is doing. So I don't exactly have as clear of an intuition there, but I you kind of look at the losses and see how those are different, whereas I think a bit more like per token attribution in RL versus DPO, and they're both substantially different than SFT.

• 50:20Nathan Labenz?

  So how does the, in the PPO side, how does the, like, what's the original ground truth that translates a generation level score to token level value?

• 50:35Speaker 00

  So it depends on your training.

• 50:37Nathan Labenz?

  That's often glossed over, I feel like.

• 50:38Speaker 00

  It depends on your training setup. So some setups, you'll start the value model from nothing. So then it'll take a few hundred steps to take the scores from the reward model to kind of warm up the value model. So you can kind of see if your value model is a random init, you can see this where your like value model loss has to converge before your policy will start changing notably. There's other ways where you can init a value model from a reward model or an SFT model, which is what we do on this like verifiable outputs to kind of make learning a little bit cleaner. And I actually don't know how that changes the value model exactly. I think that essentially the, it must, there must be some mapping between

• 51:20Speaker 00

  like the log probes and value that it is doing, which is why you would warm start with the model. But I, I don't know exactly what that mapping does, but essentially it takes steps to learn that this, and it's, it's Bellman updates. It backlogs from this final reward is how the value, value model learned. So it takes time to get to the earlier tokens, which is what the warm up is doing.

• 51:42Nathan Labenz?

  Fascinating. Is there a intuition or even just an observation as to kind of what, what sort of different tokens get what kind of value? Like presumably something like the answer is would have like low value and then the actual answer would have high value?

• 52:00Speaker 00

  Or the answer is might have high value because it knows that the answer normally follows it or something like this. I think it's, it's probably not easy for us to build intuitions on this because the, the feature space is so big. It's like, like how are you going to build intuitions over what? Like in, in O1's case, it's like, is weight a high value token? But if it's like too high value, it'd only just be like, wait, wait, wait, and never answer. So there's definitely some like weird things that I don't know how clear it is to make, to get intuitions out of it.

• 52:31Nathan Labenz?

  Okay.

• 52:31Speaker 00

  It's like, it's all, all the tokens are conditioned on the previous tokens. It's like the value is conditioned on everything that has come before it. So it's like not just the token. It's like the token in context.

• 52:42Nathan Labenz?

  Hmm. Have you looked at the, and this is a bit of a digression, but have you looked at the Entropics project? I wonder if there's a-

• 52:50Speaker 00

  I have it in detail, but I do think it's a good direction.

• 52:54Nathan Labenz?

  Yeah, it seems like there's some connection here where basically, I mean, and I haven't studied that in depth. And I don't think it's like, you know, a total game changer necessarily, but it does feel like at least the seed of something that could be a big deal.

• 53:12Speaker 00

  Inference time compute is very related to RL, where it's like having a good model from RL that can attribute value is very useful to spending more on inference. And we'll see all of these things continue to proliferate. And I put Entropics more in the inference time compute thing. But there's very similar intuitions. Like if you sample more within a bat, if you sample more completions from a prompt in RL, you're essentially like exploring more to potentially find a high value completion. And if you're doing inference time compute, a lot of it comes down to how do you sample from the language model and encourage it and like reweight

• 53:53Speaker 00

  or interrupt it or change the sampling to find the right completion. And that's where like multi-color tree search type things come in. It's like search over reasoning steps or all of these like Q star, whatever type of things you want to go down that direction. It is related, but I think it's, you can draw a boundary between training and inference and it's, it's okay.

• 54:16Nathan Labenz?

  Yeah, the connection that I'm, the thing that's bringing it to mind for me is just this sense of looking at the logits on a token by token level and sort of inferring from there, you know, like, is this going well? Is it not going well?

• 54:32Speaker 00

  Is that what Entropics is doing?

• 54:34Nathan Labenz?

  Basically, I mean, it's it has kind of multiple modes where if it's if the if the model is like confident, you know, for multiple tokens in a row, and I should say I'm a little bit out of school on this because I've only studied a little bit. But if it's like high confidence, you know, it just kind of proceeds. If the logic starts to indicate low confidence, then it will like manually inject a follow up question in some cases.

• 55:02Speaker 00

  Yeah.

• 55:02Nathan Labenz?

  If that doesn't work, it'll like double back and try again.

• 55:06Speaker 00

  Well, I would say that I do think that something like this is what like O1 could be doing. But just like, we see how easy it is for RL to go off the rails. Like if you run RL for a long time on something like I F eval, which is these constraints, it literally does exactly the constraints. It's like, please include this word in your answer and make your response 100 words or longer. It literally just will print the word 100 times and then stop. It's like, it's like stupid stuff like this. And it's like, well, maybe if you can do RL plus like have some more fine grained control over the generation. And you're like, oh, it kind of you squint. It's like, you just have to figure out how to put these things together.

• 55:46Nathan Labenz?

  So one more thing on DPO versus PPO before we move on. So PPO has this reward model, just make sure I'm understanding correctly. PPO has the reward model. There's then this process of translating the reward score.

• 56:02Speaker 00

  The right intuition is that the reward model is trained to do PPO. But in the RL framing, the reward model is actually the environment in a way. So the environment in RL is supposed to be what returns the reward. So the reward model is a very constrained environment and your actions, your inputs to the environment are prompts and your actions are completions. So it's like a totally broken RL environment, but the reward model is not part of like the training update. It's more of an isolated thing that we just happen to train.

• 56:32Nathan Labenz?

  Gotcha. And then the to compare against DPO, there is in PPO, there's this process of.

• 56:40Nathan Labenz?

  Mapping or translating a score to token wise value, which then goes into the backprop versus in DPO, there is no reward model, right? And just contrasting two outputs and some fancy math, which nobody seems to have a great intuition for, maybe the original authors.

• 57:00Speaker 00

  I've talked to them a lot. I feel like I had more of an intuition for this like six months ago when I was really in vogue, but I do think like it's still changing the tokens individually. Like it's like it is in that way, doing something similar, but it's not doing it through a value model. It's only mediated through the loss, which looks at the sums of log problems. So it's kind of removes that intermediate step.

• 57:25Nathan Labenz?

  Yeah. So that's simpler, requires less compute, requires less GPUs. And just for whatever reason, seems to consistently fall slightly short of.

• 57:38Speaker 00

  Yeah. I mean, in our setup, we tried to throw PPO in at the end and we haven't gotten PPO to beat our DPO setup again. So it's like, okay, we went through this big process where we saw PPO is better. And I still think like, if we kept hammering at it, we could probably make PPO better. But just experimentation time and data is so much more important that it's just like, we're not going to take four times the compute and experimental time to, it's not worth it. At this point, like, oh, our PPO is not as good. Maybe it's like half that we don't really know how to train a reward model because we haven't been focusing on it as much, but also half of just like, it's harder. So I do think that it's funny. It's such a great silly debate. It just never goes away.

• 58:20Nathan Labenz?

  Yeah, but it's a great, very good zoom out corrective for you to give me there too, because as much as all this stuff is easy to go down a rabbit hole on. Data quality matters more than which algorithm you choose and probably matters more than anything. Maybe you could give some intuitions there.

• 58:39Speaker 00

  Yeah, it's like we baseline 2.2 on our version 2.3 and it's like 2.2 versus 2.5, which is DPO to PPO is like 1%. And then on our valves, like 2.2 to 2.3 is like 14%, which is all just like data curation and process and stuff like this. So it's like, okay, like there's, there's your number. If you want to go off and be a algorithmic, like academic, like your most cases, you're going to be fighting 1 to 2% versus making really specific data where you care about, where you can get like 10x. I'm not surprised. That's probably what industry does too. Like these post-training teams are like, you're the person generating Python code data. You're a full-time Python code instruction and preferences and prompts and filtering lead.

• 59:24Speaker 00

  And you do it and you generate crazy good data on one really, really specific thing that may or may not be in the actual valve suite. It's like, that's like even we aren't doing, like we have people on data, but it's not like they're spending a month on this one really specific thing. And you have 12 people doing that. It's like, no academic lab will ever do exactly what the closed labs are doing in that level of depth.

• 59:50Nathan Labenz?

  Yeah, the, I mean, it's super resource intensive.

• 59:55Nathan Labenz?

  I guess you said you trained a thousand instances of 8B in this whole project, right?

• 1:00:02Speaker 00

  If you look at our valve leaderboard, we have like a thousand plus models. So not all of them are 8B, some of them are tests, but like, I think it's ballpark for, if you want to get this level of results, that's probably about the process that you will need to go through. Like you could be twice as good and then it's 500, but that's not that big of a change.

• 1:00:23Nathan Labenz?

  Yeah, so I just want to get a little bit of like a procedural understanding for what this, you know, the overall trajectory of the project is, and maybe we can contrast it against something that I do and that I've like talked about in previous episodes, which is just fine tuning a model for a specific task. I've got a whole, you know, kind of lecture and how to on that. And I tell people typically start with 10 gold standard examples. The quality matters most, like just staple your pants to the chair and work on those 10 examples until they're the absolute best you can make them. That's small enough, you could probably do few shot in a lot of cases, maybe you'll want to fine tune, depending on whatever.

• 1:01:08Nathan Labenz?

  And then I typically tell people, expect three rounds of iteration where you're going to do that fine tuning, you're going to find some weaknesses in it, you're going to then come back and augment the data. If it's like generally not good enough, you need to like 10x the data is usually what I tell people. If it's an edge case that you just hadn't considered before, you can maybe just kind of patch with, you know, give it five or 10 examples of that situation and how to handle it correctly. And, you know, three rounds, maybe more, if there's like a lot of different edge cases you want to handle over time typically should get you there if you're doing one task.

• 1:01:43Nathan Labenz?

  Obviously, the huge difference here is you're doing

• 1:01:45Speaker 00

  I'm going to turn into a whole 15 minute discussion because partially I'm also reflecting on it. And it's very interesting is that, I mean, this goes all the way up to like the CEO is like, I'll sit down. It's like, how do we manage the fact that we have extremely excited students, a lot of AI too is collaborators with UW, which is like total best of the best, super motivated students want to do something, but they can't do these whole projects. And then you have like people like me or slightly like people like me that are my same age. So I'm like a junior, mid seniority research scientist, did a PhD, have some experience, but I'm also like not a professor. I have more visibility so that ends up being like, I end up being effectively more senior because I have distribution and I have done things.

• 1:02:29Speaker 00

  But you have also these people that are like research scientists, full time. They have some projects, they can focus on this and do a lot of it. It's like, how do you balance these incentives is very hard. So I would say that for this project,

• 1:02:43Speaker 00

  In a academic sense, I would be last author, but in an industry sense, I'm first author because I just have to do all sorts of random things. It's just like incredible, like just hold everything in your head and try to keep track of everything at once. And there's admin help and we have meeting structure to help with this, which is like the philosophy section of the paper is like, what are we trying to do and are we on track going in this direction? Because what it starts with is like, I would say there's like four to seven leads that have done a substantial amount of like first author level work in a normal academic setting, which is like making a lot of data, running a ton of experiments, building the whole eval setup. There's like all these things, which is like, at the beginning, we kind of sit there, it's

• 1:03:25Speaker 00

  like, what do people want to do? And we haven't talked about this, but like, one of the things we really wanted to see is like Lama 2 and Lama 3 did rejection sampling. It's like, can we make rejection sampling work? And it's like two people and all they're doing is like, can we make the numbers go up with rejection sampling? And this is like a negative result we have, which is like, we did a lot of things. There's some interesting things, like if you're using an 8B model, the generations are less good than the instructions we start with. So for like generating new instructions, then we do SFT on them. It like kind of makes sense that it would make the model worse. But it's like, we had these two people doing instruction tuning. Some people started with like, let's compare these DPO alternatives, DPO, like norm,

• 1:04:06Speaker 00

  norm, DPO, whatever all these things are, figure out what these settings are. And then some people that are like, where are the open data sets? Can we get them? Let's start training on open data sets again. Where are we falling short of Lama? So it kind of started as like this algorithmic phase. And then you get the bearing of like, okay, where are we not doing well with Lama? Are things just not working? Let's kill them. And then it kind of shifts into the second phase of like, let's try to get really specific data. Let's kill some of our long tail projects and we're working on specific data. And then you continue to mix, you continue to make your Val Suite more stable. When we start the project, the Val Suite is not that stable. It's like, huh, like we're getting crushed by Lama on this thing, but like,

• 1:04:50Speaker 00

  The formatting is totally bizarre. It's like isoterritoric because we tried to recreate the entire llama eval suite in our codebase. So there's like some evolution there early in the project as you converge on evals. But then it becomes much more exploitative, which is like we build new data for specific capabilities. So people are kind of heads down, like building math data, building IF eval data. And then there's probably a trade off of like some people are working on SFT data and SFT mixing. And some people are working on DPO and DPO mixing. And then like soon this whole like does this RL thing work, which takes like two people to start doing this RL thing. Kind of on their own, but like part of this project. And then as the weeks and months go by, like you try to finalize SFT. So like SFT is finalized.

• 1:05:33Speaker 00

  The data mix was probably like a month ago. Then we have to, we did more decontamination or like, oh, we have to turn it again. Oh, our 70B hyper parameters are wrong. Oh, we have to turn it again. Oh, we have to try model merging. We have to train it a few more times. The quick note on model merging, it's like it's a safe bet to merge SFT by running multiple seeds on the same data set. But it could often be that just by running on multiple seeds, one of your seeds for SFT is actually going to be the best. So if you want to get a good SFT model, you want to just train it three to five times across a few random seeds. Which is like pretty funny. It's just like even more compute. And then like it for the last month has been mostly like final touches.

• 1:06:15Speaker 00

  And then full on DPO, which is like we have to generate, we have to do this on policy thing, which takes a lot of API credits. And a lot of just hands on, which is like you take your SFT model, you run completions. And then those are like the person that just owns on policy preference data. So you give them prompts and models and they do like the LLM to make generations. And then they use the OpenAI API to do LLM as a judge. And then they're like, here's your preference data set. And then you have, we have different models. We have 8B, 70B, we have all those models. So it's like, this is the person in the team that's just owning, like they make preference data. And you have to do this. And then those people, that goes to a training person, training person does DPO, they pass it to RL. So you kind of just have these people that naturally, to zoom out, you have people that naturally converge to different areas.

• 1:06:59Speaker 00

  And this is probably like 10 to 15 people that are actively involved most of the time. At this size, we did not need strict delegation in management. If you go to the llama size, 100% chance you need delegation in management and rules who who makes what decision when. Effectively, I'm like softly that person, which is like making the call on what SFT mix is final. But there's definitely an interesting transition organizationally, which is like you cannot scale it more because it becomes a mess if you're more than 10 to 15 contributors on one of these language modeling processes. But then there becomes a big cost if you add in managers because then you have to do a lot more of like delegation of decision making and stuff like this.

• 1:07:43Speaker 00

  So in some ways, you can tell by how I was describing it. A lot of this is like somewhat chaotic and free form and just relying on people being in the weeds, in the details, and very happy to communicate with the various people that they know needing it. And a lot of that is autonomous. It's just like, um, I can't be over Valentina's shoulder being like, get your DPO model to Hamish to do RL on. It's like, they just do that. And a lot of those processes are messy. Like, we describe our stand-up meetings as chaos. And it is very funny because we just sit down, we write our updates, and then we do 50 minutes of just like chaotic technical updates based on what the heck people are doing. It's like the just information overload in a lot of ways.

• 1:08:28Speaker 00

  So, I don't know. I think that accurately reflects the chaos, but there is like some kind of cadence to it of like, what is your goal? Are you on target? How do you like make decisions to kill sub areas of the project? Like, kill rejection sampling, kill long context, kill multi-turn? Like, these are just things that it's like, as you get, like, as you get better numbers, you know you need to get the model out. So you just have to keep reducing entropy and reducing entropy. So it kind of goes through this phase of like, you explore in the middle and then you get momentum and you collapse onto a final model. And We can do that much faster than Llama can like because getting Llama out is a bigger issue for them legally,

• 1:09:09Speaker 00

  strategically and stuff like that. So they do a lot bigger of an investment cycle to get these models out. I think even more so than like OpenAI. It's like OpenAI and Google and stuff releasing these new API models every couple of months. Whereas Llama is like, you get like one or two Lamas a year and they drop the weights and it's final. So it's like kind of interesting to think about the distributions and there's, I mean, there's a lot of policy discussions on like weights being final and out there, but like that type of stuff is also informing the development cycle, which I haven't talked about at length.

• 1:09:43Nathan Labenz?

  Yeah, that's, it sounds fun.

• 1:09:46Speaker 00

  It is, that's a lot.

• 1:09:47Nathan Labenz?

  Multiple follow-up questions. So when you are doing the day-to-day experiments, what sorts of, it kind of reminds me of, I was a chemistry research assistant as an undergrad in reaction development. And it had, there were some commonalities where it was like, you know, we throw a bunch of reagents in, we kind of run the reaction, then we come see, you know, come back a couple of days later and measure how well it works. And at times we were surprised and it was a, you know, high dimensional space. So you could kind of, more acid, less acid. Let's, let's do an experiment with, you know, varying amounts and then vary the, you know, the solvent and just all these different things that you could vary.

• 1:10:28Nathan Labenz?

  It sounds kind of similar where I'm imagining that like the thousand things were actually like a hundred experiments of ten variations each.

• 1:10:39Speaker 00

  Yeah. So this is the tension. It's like, how do we be

• 1:10:43Speaker 00

  scientific at the different stages? So different things we can run ablations on and some of them like mixing are just like, here is our process. It's like, we did this. It's like, it's very much unacademic. But in like the RL side, it's like, oh, we can compare different initializations with value model. You can compare different regularization, these specific hyperparameters. So there's definitely both of that, which is like, you need people that can operate in this chaos, which is just like intuitively, like where do we go? But you also need people who are very scientific. It's like, yes, no, does this work based on X clear results? So it kind of, it has the full spectrum there and I. I mean, I agree. I like being I've my background is in micro electromechanical systems and other kind of EE stuff before kind of shifting into AI. So there is that

• 1:11:28Speaker 00

  kind of messy, like in the lab nature of like you're just trying to build this thing or you're like trying in that case, it's like you're doing a reaction. It's like it works or it doesn't in some cases.

• 1:11:41Nathan Labenz?

  So you're literally just like, I guess maybe better way to ask the question is how much juice are you finding in different dimensions of exploration? Like you could vary the mix of the instruction fine tuning, or you could vary some hyperparameters, or you know, what are the sort of classes of things?

• 1:12:01Speaker 00

  But I think like most of it's data. It's like you kind of find hyperparameters that work and they don't really change. But within that, I think there's still a very high level of juice. I think we're doing a general model, but I still think we could fit more evals into our mix and improve performance without substantially changing the size. I think that like the amount like you were talking about this with your recipe, like the amount of data that you need to target a specific eval is actually not very high. And like, there's a lot more that you can do. With that fact, there's a lot more post-training that is not really touched. And I think that the opportunity is high. It's mostly about setting yourself to have an eval feedback cycle. And we I talked about

• 1:12:46Speaker 00

  killing these different capabilities, and that's because we didn't have evals that we liked. It's like, I'm 100% sure that we could improve them, but it's just much easier in a distributed environment where you have a source of truth, which is your evaluation.

• 1:13:01Speaker 00

  Because the big thing is like, how do you develop character? I think character is something that you don't have a valuation for in your models. And our models, if you compare them to Claude, will not have as consistent of a character. And I especially think for these models that are going to many users, character is important, and I find it very fun. But it's like, I don't know how to motivate the right people to do a four month, like, let's like, let's make like, almost like, like chef's kiss emoji, like really right spot on, like, then the CEO is gonna be like, dude, what the heck are you doing? Like, what, what is happening? So in that regard, like, that is a split that like academics works with the valves, but anthropic, like, Mandaskal, like

• 1:13:40Nathan Labenz?

  The vibes matter, yeah.

• 1:13:42Speaker 00

  has a lot of like final say is like, this is what Claude is supposed to be.

• 1:13:47Nathan Labenz?

  Yeah, that's that's really fascinating in multiple respects. Okay, let's talk about the synthetic data and LLM as judge situation. I get, obviously, it's super resource intensive to create this stuff, you know, with humans doing it.

• 1:14:05Speaker 00

  Much cheaper than humans. Yeah.

• 1:14:06Nathan Labenz?

  Yeah, 1% typically is kind of my.

• 1:14:10Speaker 00

  Yeah, what do we spend on API?

• 1:14:12Nathan Labenz?

  Maybe even less at this point.

• 1:14:14Speaker 00

  It's far enough in that's fine. Like, everyone cringes here when we look at our OpenAI bills, but I would guess it's over 50 grand we spent on LLM as a judge credits? Or like about for this project, which if you're doing that with humans, it would be millions.

• 1:14:30Nathan Labenz?

  Yeah.

• 1:14:31Speaker 00

  And that would still kind of hurts. It still hurts that money just goes to OpenAI and we're doing open source research.

• 1:14:38Nathan Labenz?

  Yeah. I mean, that's, um, that's like 10,000 million tokens? Because it's like, you know, a couple.

• 1:14:50Speaker 00

  Yeah, the LLM as a judge is weird because you throw most of it away. So it generates a bunch of tokens and you take one, which is the answer. So like does COT or something and then you take the token. So I don't know. It's hard to attribute exactly where they're going, but yeah.

• 1:15:03Nathan Labenz?

  Yeah, it sounds like you may have, if my back of the envelope math is right, that's.

• 1:15:07Speaker 00

  It's still way less than the GPUs, is the thing. Whereas like all the people I work with is like, we are dealing with the world where our GPU, I mean, you had tons of them says I didn't answer the question. It's like effectively order of a few thousand H100s, not all H100s. There are some other compute. And it's like, if you live in the world where that is the resource that your project is targeting, it's like, we are targeting big impact projects where we have $10 million plus assets. It's like per year, like per GPUs. It's like this is not, like the $50,000 API bill is like, oh, we're really pushing it big this month. Like, I'm like, this doesn't matter. Which, that makes it very odd, which is like, a lot of these people are students and I wish I could pay them more in that context. And it's like, I also am not paid frontier lab, but it's like all these, this

• 1:15:52Speaker 00

  is why all the compensation stuff is so wonky in the labs when they're spending so much on GPUs. It's like their headcount for the, what they're paying their key researchers like, doesn't matter. But then, which is like, so bizarre and hilarious, but like, I guess good for the people that are making the millions of bucks. Like, sure, it doesn't hurt them.

• 1:16:13Nathan Labenz?

  So just staying on the synthetic data for a second and kind of LLM as a judge. How would you describe the, I have some like, you know, I'm an AI enthusiast, but when it comes to using LLM as a judge, I have a little bit of like intuitive misgiving where I'm like, are we too quick to delegate, you know, to the AIs, the decision on what is good for an AI to do?

• 1:16:41Speaker 00

  So there is a project that I'm not an author on, but here, I really like the framing. I think, I'll do the framing and I'll comment on results, which is a bigger problem than just the paper. The framing is if you can have humans and LLMs do preference data, which do you send to humans versus LLMs? And that I think solves a lot of the problems, which is like, there are definitely things that we want humans giving the answer on, but there are a lot of mechanical tasks that we can outsource LLMs. This paper is probably one of many things that will come. I'm sure labs are doing stuff like this as well. The problem with it is like, there's a lot of papers that like this, that it's like, why can't we have academic papers that show that humans are important?

• 1:17:22Speaker 00

  So the same thing is like, here we like, look at our GPO results on like a human controlled data set versus human preferences versus LLM preferences. And they're like, LLM preferences scores on the valves are higher. And it's like, what are we missing? And it's like, I always look at the paper and I'm like, everything seems fine, but I disagree with your final conclusion because I feel like it's not going deep enough, but I don't know how to do the experiment to go deeper. So that is kind of me reiterating in a different way, the thing that you were feeling where it just doesn't, it doesn't seem like as an open research area, we have gotten to the bottom of what human or like what preference data is doing to the models and like why we can just not use humans.

• 1:18:03Speaker 00

  So I got like, I'm like, oh, this sucks, but I do think eventually we will continue to chip away at that understanding, which goes back to this, like humans, high noise, low bias, machines, low noise, high bias. Like eventually we will understand what these biases and noises represent, but we do not. And I'm. It's fine. I think RLHF has shifted more away from this like safety or like what is a preference area that's not discussed as much now, which I think when you're talking about normative relations and kind of these sociological things in quote unquote preference tuning,

• 1:18:44Speaker 00

  it is more important to be in touch with these types of like human versus machine biases. If it's literally like make math number go up, I'm like, okay, like it's fine to not have as clear of a guidance there. I think like the thing that I've wanted to see is you give instructions to annotators and I would like to see how well the model reflects the instructions given. If you collect preference data with different instructions, how does that change the final model? It is a very complex attribution, but it would be nice to see.

• 1:19:16Nathan Labenz?

  But the sort of layman's takeaway at the moment is at least like the best frontier models do outperform humans as measured by the preference data that they create gives you downstream better eval scores than the human preference data.

• 1:19:38Speaker 00

  Yeah, I think the caveat is we don't have the same human preference pipelines that the labs do. This is like whatever pipelines we have access to. But yeah, that's the, that's the thing clearly.

• 1:19:49Nathan Labenz?

  But they've, I mean, of course, that comes from the fact that they originally did this with humans and put a ton of time, energy, resources, blood, sweat, tears into it and did a good enough job that now it's like really hard to replicate what they did for humans. You know, who knows? Their human effort might have been a little bit better than the current AI effort, but there's, you know, it's just really hard to replicate the, the quality of the human mobilization that they had to, to get there in the first place. I'm speculating obviously there, but.

• 1:20:23Speaker 00

  That's why I kind of describe RLHF in some ways has forked between academic and industry. Like industry cares much more about Chatbot Arena and user retention than academia does. And that will probably widen and that's fine. I don't think we will ever have the ability to hill climb on Chatbot Arena because we don't have 100 million users. It's fine. I would like to know what they're doing, but it's like, I don't, that's, it'll take time.

• 1:20:53Nathan Labenz?

  Okay, another thing that I wanted to go back to that really caught my ear and then we went a different direction at first, but you know, might in some ways be the most important observation that you mentioned was, if I recall correctly, it was like in that final stage where you're doing the reinforcement learning from, what's it called, verifiable?

• 1:21:14Speaker 00

  Reinforcement learning with verifiable rewards. It might get changed in the paper, but that's the name that we're going with right now.

• 1:21:20Nathan Labenz?

  But this is essentially, did you get the math problem correct? Or like, does your code work? I guess you didn't have code specifically, but anything, I mean, this obviously seems like a huge trend, just for cost and scalability reasons. It's kind of why I've been, I wonder if you would agree with this characterization. I was just telling somebody the other day, like, we probably should expect superhuman performance on things like code before too long, because the objective, you know, and fast feedback is such that like, there's not really a limit at what a human programmer can do. Whereas like, superhuman poetry, first of all, is ill-defined and second of all, like, there's not, you know, it's, it's going

• 1:22:02Nathan Labenz?

  to be a noisy and slow signal, kind of indefinitely.

• 1:22:06Speaker 00

  Yeah. Yeah, I agree.

• 1:22:09Nathan Labenz?

  Okay, so now, in doing that process, I believe, it was, it was within this reinforcement learning from verifiable reward process that you started to observe a one-like reasoning, where it would like, double back or check its results again?

• 1:22:27Speaker 00

  This was specifically-

• 1:22:28Nathan Labenz?

  This is an emergent phenomenon?

• 1:22:30Speaker 00

  Yeah, yeah, it's like to one math domain. But to be clear, this is like, we trained it for way longer than it's practical. So like, at this point of training, normal evaluations have totally tanked. So the model is like, not as good at normal things. It'll still converge to good math answers in this one specific eval that we're training on. But it's like, its whole chain of thought process kind of like got borked. And it was, it's just funny when you see the same keyword that everyone was viral about with OpenAI O1, where it's like, wait, let me check that. And I was just like, this is so funny. But I'm, in the same way, it's like, not that surprising. It's-

• 1:23:06Speaker 00

  It seems like an RL thing to do. The other examples when O1 came out is like, oh, look, it changed to French and then it changed back. And that's just like, to me, it's just like, models doing really funky things is like, not surprising that it's the RL part of it. Because like, how, like, none of our loss functions, otherwise, are encouraging such, like, weird shit. But also, it's like an N of one. So it's like, it's much more just for fun and for piecing things together over the long term than definitively like that model is anything like what O1 is doing. I do think that the same type of training approach probably applies where they have some, they have some sort of verifier, and then they do a lot of RL. They do this on many more domains at once, and probably with

• 1:23:51Speaker 00

  a mix of deterministic and learned verifiers. And they probably do way more RL training than we are doing with some tricks. But I do think that it's not unreasonable to be excited about that. And I don't think, there's good reasons that there's some smoke and mirrors about O1 where they made it seem more complicated than it is. It's a breakthrough, and I'm sure they did a lot of really interesting novel things and hacks to get there. But it's,

• 1:24:19Speaker 00

  we've seen all the labs release the same things and again and again. Like, there's going to be O1 equivalents from Anthropic and Google within five months, or whatever you want to say.

• 1:24:30Nathan Labenz?

  Okay, that is really interesting, you know, kind of informed speculation as to what they are doing. But I just want to make sure I'm understanding this weight observation correctly. This is a purely emergent phenomenon in the sense that this was not something that you, like, gave it example data to learn from?

• 1:24:51Speaker 00

  I find it very unlikely that some of, one of our, like, we didn't train, we didn't do SFT on, like, O1 Chain of Thoughts or anything. We can't get O1 Chain of Thoughts. We didn't do any, like, intermediate edits to Chain of Thoughts or anything like this. People that I think are smart have said that they've seen llama do the same behavior. If you like really crank the temperature up and do things. So it's not in that regard, it's just like not that different. I mean, even like the stupid reflection 70B model is in principle, a similar idea. So like, there are a lot of ways to induce this behavior. This is a very, very, very open ended one that was with the RL loss on some verifier.

• 1:25:31Speaker 00

  So like, that's why I was like, ah, this is so, it's like, we stumble upon O1 like behavior without even meaning it. It's like, OpenAI deployed this thing to 100 million users. It has to be like a somewhat stable recipe. It's not like they had some cracked checkpoint that did this and they're like, we're going to deploy this to everyone. Like they could definitely do this behavior and it's, and get it many times. I'm interested in that.

• 1:25:56Nathan Labenz?

  Yeah, that's, I think that's a super fascinating little tidbit. Of course, all these things are kind of in there or like, you know, on the verge of being in there. But what does wonder, I think a lot about the Grokking results and, you know, it seems to make sense to me to understand behaviors as kind of on a spectrum from, on the one end, like fully stochastic parrot, you know, uh, just raw correlations between tokens. And then on the other end, like a fully Grokkt, actual algorithm that has been, you know, traced into the weights through enough time. And of course, for any given behavior, you can't really, it's not, not at all trivial.

• 1:26:41Nathan Labenz?

  And maybe, you know, borderline impossible to tell which is which. But when you start to see these things from such a simple signal as like, you got this problem right, or you got this problem wrong. And you start to see these like qualitative behavior changes. And especially when it does, I mean, as the Grokking did too, right? That happened in what would have traditionally been considered like an extreme overtraining regime. It does start to at least paint a suggestive picture that there's like potentially some sort of, I won't go as far as Grokking for this, but like a phase change kind of happening where it's like entering into a regime where. You know, it is, I think it seems fair to say that it's like beginning to actually reason.

• 1:27:26Speaker 00

  It's like the weird RL expressive behavior, I would say, whereas like no longer can, it's a generation fundamentally shifts to do like weird stuff. Whereas like most internet text is very, just going forward where this like RL behavior is just much more cyclic and strange. But just why it's sad, we can't see the O1 reasoning traces for general use. So once we have something like that, it will, making these analogies will be much more compelling. Like if we could run O1 on our same prompts and you look at way more of them, I think it'd be a lot easier to say like, yeah, this is a very similar behavior.

• 1:28:08Speaker 00

  We just have to, you have to then scale the training regime to be stable, do this in every domain and like do it repeatedly.

• 1:28:17Nathan Labenz?

  Yeah, which obviously is not trivial, but I mean, this is something that you observed.

• 1:28:21Speaker 00

  I mean, it's totally.

• 1:28:21Nathan Labenz?

  Without any intent to see it.

• 1:28:23Speaker 00

  Well, it's like, yeah, you're just like one of the RL, two RL leads, like Hamish and Costa were just poking around generations. When you do RLHF and RL, it's like, you need to look at the generations to make sure the model's still working. It's just like a normal thing and they're like, oh, this is a really.

• 1:28:36Nathan Labenz?

  Look at your data, yeah.

• 1:28:37Speaker 00

  Oh, and I'll tweet this, this is so silly. The O1 tin hack community will love this, but like it's, and that was, it's not like we're searching over the generations for the word weight or something. Like I, we found a couple of them. I'm sure there's many more if we look for it.

• 1:28:54Nathan Labenz?

  Yeah, there's no substitute for digging into the data that is, can't, can't repeat that mantra enough, probably.

• 1:29:04Speaker 00

  Yeah.

• 1:29:05Nathan Labenz?

  If you were to describe everything that you learned and take away all of the process of the experimentation and the learning. You know, like, what's the sort of idealized final process? It's, is it as simple now if I was gonna just redo your thing on a different base model, whatever? Do I have basically one day of 32 H100s for supervised fine tuning and another day for the next phase and another day?

• 1:29:29Speaker 00

  I think you need to do a few mixes. You need to do a few informed mixes based on the base model and like what capabilities it needs more or less help with at kind of each stage. But you can probably start from the same superset and then do a few experiments with more or less and various behaviors. So probably like a few cycles per base model to make sure things look right if you really want to get the best performance. You can take these data sets off the, just like kind of from the shelf and use them and you'll probably get like 80 to 95% of the performance on a given base model. So depending on how much you care, it's pretty fine. And I would say similar at both SFT, DPO and RL. The interesting thing with RL is that we don't quite know

• 1:30:14Speaker 00

  how the ceiling is defined per base model. So if you do RL on like a less good SFT model, we have found that like on Llama, for example, on 8B, like RL for GSM AK will always saturate at 87 to 88 GSM AK. That's just like a fundamental limit of the base model. No matter what we do at SFT or DPO, we could then get the GSM AK to something like 85 reliably without too much degradation. On different OMO base models, like we take the OMO model for July, it's like we bumped it from 60 to 75. And it's like, we don't know what defines these kind of saturation limits, but as you get better at having more training stages, a lot of these kind of like just take it off the shelf and train will look different.

• 1:30:58Speaker 00

  So it's like if we know we can really boost GSM AK, it's like we don't need that SFT training data. And maybe we can use that SFT budget in a different way. But that type of sophistication is something that we haven't explored a lot of. So that's why it's like, it's kind of, in my mind, it's like the, to reflect on this and try to think about what you do next. It's like there's a very different thought process. If you like really know you can recover certain abilities at different times, where it's not just like arg max at every stage. And that takes a lot more experimentation that we haven't done.

• 1:31:31Nathan Labenz?

  So most people that are doing stuff in the world are, well, I guess, I don't know this for sure, but my general understanding of like most actual, you know, industry projects are that people are not trying to create general purpose chatbots, right? They're trying to create something that fits a specific need in a specific context, you know, and kind of maxing like one or possibly a few different tasks. What advice would you give to people who are like, okay, how do I map this onto my probably much simpler situation? Um, maybe I, you know, should I do like one model per task and just keep it simple? Should I, if I have five different tasks that I want to do and they're like kind

• 1:32:15Nathan Labenz?

  of related, should I try to do them all in one mixed data fine tune?

• 1:32:23Speaker 00

  Um, it depends on the interface you use. If you really can always just choose the right model, you can do one, one per. I think there's a lot of interest in general interfaces right now. Some of this will change in the RL stage. I think I'm gonna give it, I'm gonna, I'm signed up for some NERIP's talk on like AI engineers and I'm gonna try to figure out a worldview on like how to come up with these RL verifiers for different tasks. Cause I do think if you have a verifier and the distribution matches your task, like this RL stuff will just kind of work. And it's kind of, it would be really interesting to see more engineering-y and less research-y people try to adapt this and just take it. And we have early days of what we call like LLM gem or like

• 1:33:05Speaker 00

  an open source repo where you can like add different constraints and then just do RL on it and take the model. And like you could add these domain, you're essentially adding domains for RL and language models. So I think that's kind of the newest frontier where some of these like SFT, single domain, few domain, it's like. You could try it and you can see and it's not that interesting, but like if we can unlock RL with verifiers for so many different niches, it'll be really like the next moment in this kind of like fine tuning, specific language models narrative.

• 1:33:37Nathan Labenz?

  And I guess the the general purpose answer there would be LLM as judge, right? I mean, you can get more concrete where possible, but I, you know, I always think about my

• 1:33:51Nathan Labenz?

  company, Waymark, we do video creation for small business and there is partially a ground truth, but honestly, we don't really have that much trouble with the the concrete, like the objectively verifiable stuff mostly works out of the box. That's like making sure you're delivering the right amount of content and the right structure and so on. And then the real question is like, what is a good script for a video? And we, you know, if I'm applying your lessons learned to that, I basically would just say,

• 1:34:24Speaker 00

  Yeah, verify.

• 1:34:24Nathan Labenz?

  LLM's judge it and make a preference set and go with that.

• 1:34:27Speaker 00

  Yeah, we have, I don't, I don't know if I have the energy for the whole multimodal discussion, but there are definitely different guidelines as you go multimodal. I think preferences are more powerful in things like images, audio, video, because our intuitions, especially human preferences are just intuitively much more expressive than in text. So a lot of things they're saying are very like text and in that way like capability in a very narrow sense.

• 1:34:53Nathan Labenz?

  Yeah, that makes sense. I mean, I wouldn't expect in the images, I feel like you do sometimes get some pretty good feedback.

• 1:34:59Speaker 00

  You could probably do like detectors of different types of objects. If you like are saying like make sure if they have a prompt, you can do a detector to make sure that certain noun or entities in the prompt are in the image. And I bet you could have a like, that is a way of doing precise instruction following for images. And I wouldn't be surprised if it's already done.

• 1:35:19Nathan Labenz?

  Yeah, that's interesting. I mean, I find the Momo model quite interesting for its pointing. Seems like Claude has now kind of got a pointing function too. But that is a cool. I don't need to dig into that too much today.

• 1:35:34Speaker 00

  How it works for robotics. Because you think of the robotics task, it's like

• 1:35:39Speaker 00

  it bridges the gap between VLM and planner in a very nice way, where the VLM could be asked a question of like, how do I, like, what tool do I need to do X? And then it can just point at it rather than saying the answer. Which like, and then the planner knows to get things that are pointed at or something. And that is interesting. And some of my robotics friends are excited about it.

• 1:36:04Nathan Labenz?

  Oh, one more in the weeds question, then one kind of zoom out.

• 1:36:08Nathan Labenz?

  On policy, we've mentioned a couple of times, the importance of on policy data. My intuition for that is just like, you want to be working from where the model is now.

• 1:36:21Speaker 00

  Yeah.

• 1:36:22Nathan Labenz?

  And if you, I guess if you don't do that, then it just doesn't work as well. Can you give a little more color on that?

• 1:36:29Speaker 00

  Yeah, it's, I mean, it's very related to this kind of PPO DPO debate where a lot of the proponents of PPO is that you're scoring and updating the model based on things it is generating itself, rather than completions you got from elsewhere. And that's really the thing. It's just like, it seems like it's a bit better learning signal if within these batches and these pairwise comparisons, the the tokens you are looking at are closer to the log probes of the current model. Which I think makes a lot of sense, because what you do at like, say, DPO is you compute the log probes of the token in a completion, and that updates your weight. So it's like, okay, like, it makes sense if the model's log probes are not in like a weird space it hasn't seen before. And it's nice to see that it backed up by experiments.

• 1:37:13Nathan Labenz?

  Yeah. Okay, cool. That makes sense. So finally, the big zoom out, I guess, is like, O1 is obviously out there now. We don't get to see the reasoning traces. What do you think this implies for the future of?

• 1:37:30Nathan Labenz?

  Proprietary aka closed versus open like one of the striking slides that in the presentation that you sent me was, you know, basically very directly saying, we can't do this as an open organization if we don't have the frontier models to do all these generations, do these scoring, like, there's just not a substitute for that in the open world. Now, we don't even have the O1 traces. So, does this suggest like an end of an era of open source catching up? Or what do you think is going to be the future there?

• 1:38:08Speaker 00

  I think it'll end up being fine. I think that it takes time and it will be different. But there's so, like, there's so much interest. I mean, I'm trying to like hedge, but I do feel like it's pretty likely that I end up doing a project in this. But there's just so much interest in it. And it's both exciting and new. In some ways, we have less of a disadvantage in time, because we're seeing it and we're not like waiting, like GPT2, which entered like GPT4 when people got serious, or entering at O1. But I just think that it is time and time again, we see that people are very motivated in this area and it'll probably be trained differently than O1 was, but people will figure out how to elicit

• 1:38:50Speaker 00

  the same behavior. It's like, once you have an existence proof, it will help. We still have law with 405B, which is very powerful, open weight model for reasoning and stuff like this. It's mostly just, I think, takes the iteration on building entirely new infrastructure that the community converges on. Like, the fine tuning infrastructure is not going to be like, well, part of it will be used in building O1, but I suspect there's going to be a hole. That's like, what is the transformers library for O1-like models? Like, there's something different there that I think is simultaneously exciting and maturing for the ecosystem, which is like, look, we need to approach these. Systems and training in an entirely different way, and there will be a bigger spectrum of resources that are done to do this. Like, again, there's the pessimistic case of like,

• 1:39:35Speaker 00

  we need that. We don't know all the steps along the way, but I I'm pretty sure we'll get something close to it with the amount of excitement that we see.

• 1:39:44Nathan Labenz?

  And what do you think that looks like? What's your interview? Is it sort of, is it like elaborate prompting to kind of generate these synthetic traces? I could imagine you take a four or five and you like chain four prompts together and say, now look at it a different way. Now look at it a different way and kind of stitch those into some bootstrap bubble.

• 1:40:03Speaker 00

  Yeah, I feel like I'm losing the energy to go through this whole thing. This is the post that I plan on writing is like my plan for reproducing O one. I mean, you need to generate some seed data that looks like it. You probably need, you have a language model, look at chain of thought traces and modify and continue them. And then you kind of do that. And you need to figure out how to seed a model with those and then kind of have the right domain to do some sort of RL on it on certain verifications. So it's kind of like, get some initial data that looks good, get a model that can do it a little bit. And then you have to the feedback loop of actually like having things that can verify and having the update function reinforce that behavior is, I think, the

• 1:40:45Speaker 00

  hardest one. And we'll start to see efforts on generating this data and generating verifiers. And then this putting it together thing, which is where there can be a lot of variability and performance. I think we already see online, there's people like open like O one reproduction. And I'm like, still like, I'm like, I don't even, I don't even need to look yet. Like, until 2025, like, I don't even need to like, take them seriously. Like, all the ones that are already out are like, not that serious, unless it's like Google and traffic or something.

• 1:41:14Nathan Labenz?

  Yeah. All right. Well, we'll check back in with you in a few months, perhaps.

• 1:41:18Speaker 00

  I need to wrap my head around this. I'm in the different type of post-training environment for a while. And then that's kind of the next thing to explore along with some like better taxonomies of agents and stuff like this, which is fun, but it's a big mental adjustment.

• 1:41:35Nathan Labenz?

  Yeah, it's a target-rich environment. That's for sure.

• 1:41:38Speaker 00

  Yeah.

• 1:41:39Nathan Labenz?

  Cool. Well, this has been fantastic. Any other closing thoughts or calls to actions you want to share before we break?

• 1:41:45Speaker 00

  I'll post a lot of these things on my blog, Interconnects. I already have another O1 blog post in the queue. I don't know when I will send it, but that's to come. And it's fun. And I'll see a lot of people around at NERIPs. If you're listening, I'll be there, which is exciting. So thanks for having me. This is a rapid fire. You exhausted me. By the end of the book, I was like, oh, man, I'm cooked.

• 1:42:11Nathan Labenz?

  Yeah, well, you've been working hard and I certainly pushed you down a bunch of different little dark corners. So thanks for doing it with me. I do think a lot of alpha in catching up with what you've been up to. So really appreciate the time and energy. And with that, I will say, Nathan Lambert, thank you for being part of the cognitive revolution.

• 1:42:30Speaker 00

  Yeah, thanks for having me.

• 1:42:33Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.