Pod Insights a16z Podcast DeepSeek, Reasoning Models, and the Future of LLMs About this episode Beyond the speculation and hand-wringing over the Chinese AI model called DeepSeek lies a much more concrete technological reality: Reasoning models are the future of large language models. Beyond DeepSeek, other AI labs have committed to their development, and reasoning models currently top popular LLM leaderboards.
In this video, a16z partners Guido Appenzeller and Marco Mascorro demystify how DeepSeek works and explain what the era of reasoning models means for artificial intelligence.
Transcript 0:04 Speaker 1 Hey everyone. So DeepSeq R1 last month really captured the zeitgeist. It's a brand new reasoning model from China. Very high performant. It showed up very high in the rankings. But I think it also caused a lot of fear and confusion. And that's not a good thing. No?
0:18 Speaker 0 Yeah, that's not ideal. So one of the good things about DeepSeq is basically they open first their weights, their techniques and how they build these models. And they've been doing that for a while. So this is a good insight on basically how these reasoning models work. And I think looking forward,
0:32 Speaker 1 from now on, pretty much any state-of-the-art model will use some of those techniques. And we've seen this already, you know, from models from OpenAI and models from Google that are structurally very, very similar. And this has huge implications on how much compute we need, how much GPU power we need for inference as well as for training.
0:48 Speaker 0 So here's our breakdown on DeepSeq. Some of the models they've been building, like Deepseq math, B3 and B2, and now R1.
0:56 Speaker 1 All right. So if you look at the slide here that shows the current ranking of the best AI models that we have today, you'll see that pretty much the whole top of the of the rankings has been taken over by reasoning models. This is super impressive, right? We had sort of classic LLMs all built roughly in the same style, you know, occupy the top spots for a long time. And then it changed very, very quickly. What actually happened here, right? And what happened is that we have a new type of model. It's called a reasoning model. And you know, we've got a little example here of how the output of a reasoning model differs. On the left side, we have GPT-4.0 mini. Uh asked the question about orbital mechanics, you know, that's uh it's a good reasoning question. And uh it pretty much directly gives you the one-shot answer of how this particular question works. On the right side, we have a distilled version of Deep Seek R1.
1:42 Speaker 1 So this is a very, very small model. It can't actually answer this directly from memory, but what it does, it starts reasoning. And if you read the text, right, it really starts to hustle. It's trying to think, it's trying to theorize it, questions itself, and you know, over time hopes that it's going to arrive at the right answer, which in this case it actually eventually did. So it's very impressive that with these small models, we can actually get to these very high, high quality results. So, Michael, this is how we used to do training, is that right?
2:10 Speaker 0 Yeah, that's right. And that has been a very efficient way, very simple, in the sense of like you gather a lot of text data from the internet, and you have pairs of questions and answers, and then you train the model in a very straightforward way. Aaron Powell
2:22 Speaker 1 If you look through the steps, right? Pre-training that's typically done on very large computer infrastructure. That's where you know you need the large 10,000 H100s or more, and you pretty much want to train on all the data you can possibly find, which typically means we you take the internet, because today the largest repository of knowledge that we as mankind have is the internet. And then there's two more steps. Uh first one is supervised fine tuning, where we take examples that were generated by humans that teach the model how to behave. So for example, if I just take a pre-training model, I would ask it uh what's the capital of France? An answer could be.
2:56 Speaker 0 Well, if it's not fine-tuned at all, it would just keep uh going forward, like with more questions like what is the capital of whatever other city, right? And we just keep going. So the purpose of adding these supervised fine-tuning data is basically making the model have answers straight into what you you the correct answer is, right? So these base models, if you don't do this supervised fine-tuning, they would just keep going in an out of complete mode, which is not normally what you want. So that's why you add that next step of supervised fine-tuning, which is kind of like this format of instructions or questions and answers. And that's how you make the model being able to answer what is capital for answer. So we
3:31 Speaker 1 essentially teach the model how to interface with the human. If somebody asks you a question, you should give an answer. You shouldn't reply with more questions. Uh that's that's not very useful. And then the last one is the enforcement learning with human feedback. So how is that different?
3:42 Speaker 0 So that's a that's a really good point. So the RLHF part comes in after you do the SFT, which is basically you have traces of answers or questions and answers, and then a human evaluates them and gives a score on which answers are the best ones. You train a lot of model basically based on this preference data, uh, and that's how you do the RLHF phase. Now, the reality is like you can get to really good models purely with like SFT data, so like just having the question and answers, being trained on on formats like credit or like Stack Overflow, which already has that perfect format where you have a voting mechanism. Humans vote for the best answer there, so it just goes up in the ranking the way Reddit does it or the way Stack Overflow does it. And that's kind of a way to have the data be higher quality when you train this model.
4:29 Speaker 1 That's right. They're not just correct, but they're the best. Correct.
4:32 Speaker 0 Exactly.
4:33 Speaker 1 The first one, pre-training, is done in a fully automated way. All I need is lots and lots of data and do an X token prediction. The other two require humans, right? SFT, we require human-generated examples and RLHF. There needs to be a human in the loop there. You in the loop subware to basically get this right. How are reasoning models different? And uh we look specifically here at Deepseek R1. That's not necessarily because they were the first, we're not sure they are, but uh they currently have probably been the most open about the methodology that published source code and they publish the model weights. So it's a model we just know the most about. So let's step through what they did. And if you'll if you look back uh look at the timeline here, the first thing they did actually was they trained a math model, uh Deep Seq math, which was uh very, very good math model, very, very good uh reasoning for very specific tasks.
5:19 Speaker 1 And so what what was the innovation?
5:21 Speaker 0 Yeah, so so here's the thing with Deepseq. I think like uh this is this is uh a team that we we've seen multiple models coming from a long time, right? Since late 2023, I think. Where basically the way I look at it is R1 was kind of like having all these innovations been together in one single model, going from the multi-head latent attention that they did, the MLA, uh, which I think was released in B2, and then Deepseq math. This is when they introduced their GRPO algorithm for RL training. And basically R1 was kind of like the compilation of all these techniques that they were doing working on already from other models. So that's kind of like the the part that was uh kind of surprising. It was like it's not like a one single innovation that that happened, it was like just accumulation of all these very nice things that they did over the last couple of years.
6:07 Speaker 1 Deepseq math is is really impressive. And I think for the first time, they actually trained the model by learning from itself. This is a new approach. It can train by by experimenting by itself. It basically it takes reasoning chains to solve fairly hard problems. And if it's a math or coding problem, the nice thing is we can verify the end if the solution is correct or not, right? So if I give you a very complex math equation, and uh you know, you you do it step by step, you give me the result. I can plug in the result into the equation, see if it was correct or not. Right. And basically, is to say like if that's correct, then please learn from what you did. If it's incorrect, then don't.
6:41 Speaker 0 And I think the big thing in R1 or generally with these reasoning models is we were doing before there was a human in the loop always, right? Like when we have this SFT training and these other techniques that we're doing after, like RLHF. And having R1, I think was or having this reasoning model was basically the first time we started seeing some indications that you can improve the quality of the models by just doing RL on top of it. And and the idea. So Deepseed came with B3 in December, and then they released uh just a few weeks later uh R1. And basically in between there was this model that was called R10, which was basically they took a B3 base model, which was the model they released in December, and they applied directly reinforcement learning on top of it on verifiable domains.
7:27 Speaker 0 So that is math, that is puzzles that they had, code. So all these domains you can verify, right? And you can have uh what they call a rule-based verifier, which basically is not another LLM, it's just basically looking for is the answer correct or not. Like you can just basically compute that kind of sorry and knows like this is correct or not. And it gives an answer, uh a score to the response that the model is giving. And you do that over many, many samples. But this was like R10, which in a way was a very interesting model because it is it showed that it improved in some reasoning benchmarks and math benchmarks, but eventually didn't do really well on other things, right? Like it was switching between languages. I think that was one of the things that they observed. The answers were not very readable for a human sometimes, like it would repeat the last token many times.
8:17 Speaker 0 So that's when R1 comes. So they tried to fix in a way R1 zero, but turns out that R10 was actually very helpful in the process of making R1. We
8:26 Speaker 1 talk about Deep Seek math. Let's talk about Deep Seq V3. So Deepseq V3, I think was a very good model. It was when it came out still is. It's a classic LLM, although I think they took a little bit of reasoning from a precursor of Deep Seek R1 and applied it back. So it might have a little bit of reasoning in there. But if you look at the rankings here, right, it's it's definitely one of the best models when it came out that was out there with the MOE architecture, I think it's a very modern architecture, right? It gives you very good inference characteristics.
8:50 Speaker 0 The other thing is like talking about MOE. So Deepseek has been doing MOEs for a long time and or a few years now. But the nice thing about B3 is the number of experts, right? We talk about this to 156 experts, which is a large, a relative large number of experts in terms of at least open source models that we've seen out there.
9:09 Speaker 1 That's right. It's much more higher.
9:10 Speaker 0 So it's a much higher and the ratio of the experts that activate the re-inference is also pretty high. Like they activate a few experts versus the total number of experts, which is 256. So they had this nice ratio of activations versus model size, which I think is it's pretty unique. And the reason I say that is it's training an MOE is not very straightforward. It's it's a little bit more complex than just training a full dense model. So I think that was some of the innovations they did or the know-how that they had was being able to train uh large expert model.
9:42 Speaker 1 And so and this deep seek v3. So that was the starting point for everything else they did. So let's talk about R1. So here's a rough flow chart to see what the different steps are for training R1. So the first thing they did was basically they trained this thing called Deep Seag R10. Which is great at reasoning, but has a lot of other problems.
10:03 Speaker 0 That's a fair statement. And
10:05 Speaker 1 honestly, everything else is just them going back to Deep Seek V V3, so their classic model and figuring out ways how they can leverage the better reasoning capabilities in G Seq R10, but at the same time make that model better behaved and easier for humans to consume. Right. Is that is that the right over here? That's right, that's right. So let's talk about how Deepseq R1 was actually trained. And this diagram here may look a little bit intimidating, but this gives you an overview of the training process. So on the left side, we have Deepseq V3, which is their classically trained LLM. So this is not a reasoning model, only with a little bit of a little bit of reasoning, but it looks more like the models before reasoning models
10:42 Speaker 0 aware of thing.
10:43 Speaker 1 And then basically they took that and trained using the same approach that previously done the math model. They use that to train uh the first deep seek R10. And we'll go in details of how exactly that was done. Uh and Deep Seag R10 is actually very, very good reasoning model. It's arguably better in reasoning than the final Deep Seek R1. Um unfortunately, it is not very nice to work with because it switches languages uh uh you know randomly, it may start in Chinese and then talk in English and then switch back to Chinese. It you know, may have gibberish in its answer and so on. In the end, it produces the right answer, but it's not something that a human would normally want to use. And so I think it's fair to say the entire rest of the training process just trying to get this really badly behaved model to become a nicely behaved model that humans like to use.
11:24 Speaker 0 That's right. So basically the way they did that, trying to fix R10 is it added a couple more phases in the post-training. Um that included two supervised fine-tuning phases and two reinforcement learning phases. And these are reinforcement learning phases that were a large-scale reinforcement learning phases.
11:38 Speaker 1 Uh super. And then every time basically they took outputs of that and went back to the B3 basically. They always went back to the
11:43 Speaker 0 base and start training
11:44 Speaker 1 again. The net result of that is Deep Seek I1. All right, let's jump to that into it with a little bit more detail. So what we got here is a prompt that they use for training, right? Which basically instructs the model specifically to show its reasoning step by step, right? And then the F an evaluation function, which actually makes sure that that the model actually does that. So it doesn't get any brownie points, so it doesn't get uh reinforced on things where it does not show reasoning or the result is incorrect,
12:08 Speaker 0 right?
12:09 Speaker 1 And as they were running this, I mean this is to me is an amazing graph, right? So they this is sort of plots the the length of the response over the training steps. And you can see that very quickly the answers get longer and longer and longer, right? And not just by a little bit, but you know, we're starting out here with something that's like maybe a thousand or so um uh tokens, a little under a thousand tokens or a little under a thousand words, so that would be two pages, right? Which that's already a lot uh pretty good reasoning for a complex problem. But at the end, there's at something like 10,000 tokens, so 10,000 words. So that's was that 20 pages? Yeah, there's an amazing amount of reasoning.
12:40 Speaker 0 Right. That's right. So a very interesting thing too is when you see these reasoning models, there's these new tokens that appear, right? Like the thinking tokens. And it's basically just a token that appears when the model is doing its reasoning steps or reasoning tokens, and then it when it finishes reasoning, it has like an end-of-token thinking token basically. And then it gives you what it thinks is the right response by the end. So basically apply this pipeline, and they saw these few things that were interesting in R10, right? So they kind of used some of these traces actually in R1. When they decided to basically fix these R10 models, like how do we improve now, or how do we make it usable for humans, right? Like this is great on reasoning, but how do we improve it so that people can use it? And this is where the interesting things came later.
13:26 Speaker 1 This is obviously the aha slide, where we see what they're doing not only makes the answers longer, but it actually works. So we're suddenly seeing replies that uh, you know, like if you look at the score here, they can solve much, much more complex problems that previously weren't solvable. This I think is really impressive. So basically R10, that was the quantum leap in in model performance.
13:46 Speaker 0 Yeah, that's right.
13:47 Speaker 1 Here, this is a super interesting example. This is an aha moment. Have you ever seen this in the model before?
13:51 Speaker 0 Yeah, this is this is the aha moment and the deep seek team just called it the aha moment and the in the model, but also the aha moment for them at least. When they saw like those tokens, basically the reasoning was like, hey, I'm making a mistake here, maybe let me fix this, right? So it's kind of like its own reflection. And this is very clearly very important during the RL training process. So I
14:09 Speaker 1 mean, this is this is a little bit how humans learn, right? It's like here's a problem, you try to solve it and you learn from how you solve it, right? That's right. Turns out that we can now do a similar thing with models. So the net result of this, I have now Deep Seag R10. It's very good at reasoning. It's sometimes in the middle of the sentence switches the language from English to Chinese or even something else. It's uh, you know, uh generates very strange answers that are hard for humans to read. So how did they get from there to an actual fully working model that is that is easily consumable by humans?
14:36 Speaker 0 Yeah, so that so that was uh one of the beauties that they did was basically they wanted to fix R10. And the way they did it is they took V3 again, the base model. Uh but instead of going straight into RL, the way they did it with R10, is they decided to step in between. Yeah. And they called that the cold start data. Yeah. And this cold start data was a few thousand. I don't we didn't say the specific number of how many traces they use. But basically they use an amount of data that was coming out of R10. These were traces that were coming out of one zero. But these were basically selected, just making sure they are in the right language, they were correct. They selected some of those. And they basically train the model B3 in a normal SFT way using those traces that were coming from R10. They call this the cold start data, which was that process.
15:21 Speaker 0 And it's basically to align the model in a few ways, like uh so that it gives responses in a consistent way, maybe more like a longer with more chain of thought. That was kind of like the initial kind of like the idea of why they wanted to add this cold start data in that. And then they added RL after that.
15:38 Speaker 1 They they generate the cold star data. They you know, they they use it to initialize the model, and then sort of they did the same trick that they did before, which is basically you train uh a model that gives you feedback uh on what you what you do and use that to basically rank answers that are generated by the model to basically improve what comes out of the model, right? So all of this again, we're using the the data that they came out of this R R10 model, but now sort of apply it back to the original model and to get a model that's better behaved. But
16:04 Speaker 0 it turns out they still weren't happy with that. So they use that again to generate
16:06 Speaker 1 traces, so to generate reasoning, reasoning chains. I think it was 600,000. There were
16:11 Speaker 0 600,000. That's
16:12 Speaker 1 a lot. That's a lot. If you think about it, let's assume I want to generate 600,000 traces of complex reasoning things with humans.
16:19 Speaker 0 Yeah.
16:19 Speaker 1 You know, for some of the math problems, you definitely want somebody with a graduate degree. If I mean we saw that the average answer was 20 pages, how much do I have to pay uh uh a math PhD to generate 20 pages of text, right? If this is uh I don't know, 100 is probably not enough. Assume it was 100,000, then this will be 60 million dollars worth of results.
16:38 Speaker 0 Right.
16:38 Speaker 1 But now I don't need the humans anymore, but instead I can generate those
16:43 Speaker 0 with a computer,
16:43 Speaker 1 right? That is amazing, right? Right.
16:45 Speaker 0 So 600,000 verifiable, it's kind of like domain math, puzzles, code. And then they took 200,000 that were more in the side of creative thinking, creative writing, storytelling.
16:56 Speaker 1 There's rare from humans.
16:57 Speaker 0 And those those came from humans. Basically, uh they came from some of them came from B3. It's very like out of the model B3. And then some of them came, seems like from the data set that they use for training B3 by itself. So they selected these 200,000 traces. Then they applied another SFT run. So basically R1 was multi-stage training, where included two phases of supervised fine-tuning and two phases of reinforcement learning, being the first phase that they used in SFT was the call-star data that we talked about. Second step was the RL phase on verifiable domains based on rule-based math and so on. The third step was basically the SFT on these 800,000 samples, so 600 from math and code, and then the 200,000 from creative thinking and and other fields that are more on the human preference rather than like verifiable domains.
17:49 Speaker 0 And then the final stage was the reinforcement learning again, basically in all domains.
17:54 Speaker 1 Yeah, that makes sense. And but it's interesting. They said they spent, I think, five and a half million dollars on training v3. So going from v3 to our one is probably pretty cheap, is my guess, right? I mean, it's mostly inference what you're doing. That's a little bit of
18:06 Speaker 0 SFT, right?
18:07 Speaker 1 That's right. I mean, in terms of number of operations, it seems to be two orders of magnitude smaller or so than I think what the pre-training is. But the V3, um, they they quoted 5.5 million dollars, I think, you know, at market rates uh to train it. Does that sound sound reasonable?
18:22 Speaker 0 That sounds reasonable. I mean, we we've seen other models from other uh companies and labs, and what they disclose is is it's not something crazy. So I think it's we
18:30 Speaker 1 ran the cost for some Lama models last year, and I think we ended up with you know a little over three million dollars, right? And and I mean all of these numbers are probably we can only we can only reverse engineer them within a factor of two or so. But these are precise numbers, but they they they roughly make sense, right? Right. There's one big caveat here, which is the final test run is often not the majority of the money that you spend, right? Uh you need many test runs that don't work well. You're highly paid PhDs that that do the uh do the training, uh probably highly paid. They they need sort of infrastructure to experiment on for daytrust. And so of your total compute budget, that final run is probably a very small fraction.
19:07 Speaker 0 Yeah, that's right. I think a lot of comes from experimentation. I mean, coming up with this composition of having these multi-stage training going from B3 to SFT and then RL and SFT and then RL again. This obviously coming up to that process probably require a lot of experimentation. Yeah. Um also there's a lot of fail runs all the time, right? It's I mean, we're just calculating the final run, but like at the same time, it's like how many fail runs do you have, like how many failed experiments you have. So I think that's where a lot of the cost comes.
19:35 Speaker 1 The cost of an aircraft is not the final assembly. It's not the test platform.
19:39 Speaker 0 That's correct. So there's a lot of cost that comes into that. Selecting the right traces, maybe there was a lot of data set uh shuffling to figure out what is the best data set and which one shows you the best results. So there's a lot of experimentation that obviously we don't see in terms of the cost, but obviously that requires uh so
19:55 Speaker 1 Deep Seek also did a number of other interesting innovations as part of how they built the model. There were some some. Change the attention mechanism, right?
20:02 Speaker 0 Well, they they so they could we've used in the past uh multi-head attention. Yeah. But they didn't implement implementation MLA, multi-head latent attention, which basically it's it's a memo, it's an optimization on uh on the multi-head attention with the KB vectors already just in latent space. So they're more efficient during training and inference. And they came up with this in DeepSeekBit 2. And that being said, they to this had a few implement implications, which was if you wanted to now to make the context longer, say they start from A to 32 or 128K. So the normally there's this thing called rope embeddings, rotational positional embeddings, which basically uh what you do, you you add an additional vector that has additional positions going forward, like if you want to extend it. The problem is with MLA, it kind of loses that vector in position.
20:48 Speaker 0 So they needed to re-implement rope, and they came up with this thing called the couple rope. Yeah, yeah, um, which they also created in in one of the previous models they did. Then they they also implemented uh GRPO, which is basically an improvement of PPO, uh, which is this RL RL algorithm for training, and they implemented that in Deep Seek math.
21:07 Speaker 1 I mean, uh overall, I think it's fair to say they they combined really uh a whole string of best of pre-techniques here, right? So very well built, I think really really amazing job.
21:16 Speaker 0 And it's very interesting to see that a lot of the optimizations are mostly in the compute efficiency side. So it's like how do we make the KVQ vectors more efficient? How do we train in mixed precision? So they train in FP8. But basically all the optimizations they're doing, um, even in the RL are in GRPO is basically a more efficient way of sampling.
21:33 Speaker 1 I think it's the classic story of AI, right? Compute efficiency really matters. Yeah. And if you're, you know, in China it probably matters uh even a little bit more. So we have this new set of reasoning models. What what are the consequences of that? Right. So I mean, if I look back, if you would have asked me in Q3 last year, I would have said, look, we're seeing model performance top out, right? I mean, so if I look at the the big you know top-tier LLMs from you know the pretty much all of the all of the model providers, uh, we're seeing their test scores compress into a narrower, narrower band. Uh you know, we're seeing open source catching up, right? Everybody is now densely clustered, where you know, 18 months ago, uh 18 months earlier, you know, GPD4 was pretty much alone at the very top tier. So how does this change things?
22:17 Speaker 0 A lot of the compute is probably gonna go into test time inference or test time compute as people.
22:22 Speaker 1 Okay, so that's the first thing. So now that we basically generate what was it, uh 10,000 word reasoning points, we need a lot more inference. We
22:28 Speaker 0 need more
22:28 Speaker 1 inference. And if we've gotten 20 times longer than our answer. So very roughly, if we all if everybody would switch tomorrow from whatever they have today to a reasoning model, we would need 20 times more inference. That's crazy.
22:39 Speaker 0 That's great. Yeah, yeah, that's crazy. And and that has an implication, obviously, in like how do we build the systems because normally we've been optimizing for a lot of training, but now also like we're we're adding this test-time compute that is very important, right? So a lot of that workload is going into that space now.
22:54 Speaker 1 What does it mean on the training side? I mean, we were sort of tapped out on training because you know we can overtrain, but but nevertheless, with Chinchilla, we know that there's sort of a ratio of the amount of training data we have and the model size. The amount of training data in the past was all human-generated data. There's only so much on the on the planet. And you know, I mean, the internet is the largest repository. So I think just every what we saw is everybody was training on the internet. Uh, this is the same data set, everyone was using generative transformer. So everybody was roughly getting a model of the same quality, right? And throwing more compute at it did not help that much because if you don't find more data, then their compute isn't worth a whole lot, right? Right. And but that's different now.
23:29 Speaker 0 That's different. And also the the quality of data matters, right? I think they believe,
23:33 Speaker 1 yes.
23:33 Speaker 0 The pre-training is like if you skip scaling data and compute, you would get uh huge improvements. But we definitely saw that plateau at some point, right? And we needed to come up with a different way of of doing this reasoning on these models. And this is where RL comes in and the chain of thoughts. And so basically that's kind of like one of the changes, I think. The other one is well, it's these models are open source now. I think I'm seeing a lot of innovation in the space where people are taking these models and
23:59 Speaker 1 that's great for the industry. It is great
24:01 Speaker 0 for the AI industry. So I think that's another thing where you have now a reasoning model that is basically a pretty good quality model on pair with some of the open AI models. So that is being a big change in the industry as well. Yeah,
24:14 Speaker 1 that's that makes sense. So I think here's a slide from uh Jason Wang, CEO of NVIDIA, where I think the way he uh he framed it was to say like, look, we've we've had our first sort of curve uh of model performance, right? Where we're starting to top out a little bit, but now we have new ones on top, right? So I think overall this means we you can probably now train better models. We need more GPUs for training because like for the self-reasoning and self-improvement, there's no clear peak with hum the like they had before with the where we need human generated data, right? So we don't know yet whether it's gonna top out yet. We need more for inference. And and overall, we'll we'll have much better models, right? And this is really exciting, right? I think overall this probably means that the AI is gonna accelerate a little bit for a while again.
24:55 Speaker 0 Definitely
24:55 Speaker 1 uh purely at infrastructure level, and then the applications on top that really verticalize this and you know create more interesting use cases, they'll accelerate it further. So I think that's uh this bodes very well for for AI overall.
25:05 Speaker 0 Definitely. Another another interesting thing that that came out of the paper from R1 was for these model models, the llamas, the 70Bs, the 7Bs, 13bs. What is it better? Is it better to use RL, the same RL time? On the 70V models, or can you do some sort of distillation from a bigger model? So turn time in their experiments, they took Lama C V and some of these other QN models, uh, and they basically apply RL straight the same way they did it with R1 on these base models. And it turns out that it improved in some fields, but it was not as significant improvement. So where they saw a significant improvement was doing distillation from R1. So basically generating a lot of question and answers and long chain of thoughts. And basically they just train on these traces. And that was more efficient that showed much improvement in these models.
25:53 Speaker 0 And I thought it was quite interesting. So it turns out the distillation actually ended up being very effective uh for these model models rather than applying the RL one, the RL of the enforcement learning pipeline in these models.
26:06 Speaker 1 They actually run great on your laptop, right? And of course, we mean downloading the distilled models. So there's a sort of a lot of controversy around the you know, Deep Seek, do you want to send your data to China? And it probably depends on what you want to do, how how comfortable you feel with that. But but uh independently of that, the model weights are open. So you can just download them on your laptop, run them with some open source package, like OLAMA, for example, on the MacBook works great. And now we have a fairly powerful reasoning model, you know, running locally on your laptop. It's actually pretty amazing.
26:32 Speaker 0 It's pretty amazing. Like you can do all these sort of innovations where quantizing the model to run it on a very small device. Yeah, exactly. Um or maybe you use uh you use a a 7B model, but you have very from a very good destillation from from R1, which suddenly turns out to be a much better and more efficient, probably for a smaller device. Yeah. So you have these sort of new innovations that are pretty exciting.
26:52 Speaker 1 Super. Well, I hope this was interesting. Uh you know, I think this is a big step forward for for AI again, and some some more exciting developments to come. Thanks.
Previous How to spot an AI Deepfake Next Agent Experience: Building an Open Web for the AI Era Page updated August 25, 2026 · transcript and summary are machine-generated
