⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

Watch the full show on YouTube: https://www.youtube.com/watch?v=wH8YKCE82Qo

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by swyx from Cognition. First time I introduce you at the new role.

• 0:13Speaker 01

  We haven't figured out the official title yet. But yes, we're recording this the day after we announced the Series C, and it's very good. Alessio is also part of the Cognition family now. We all had a fun party at the house. But today's not about us. Today, we're talking about research, small L M, model training, all this stuff with Eli from Hugging Face. Welcome.

• 0:37Speaker 02

  Yeah, thanks a lot for the invitation. I'm very glad to be here and congrats again for this new job.

• 0:46Speaker 01

  So you, obviously, I think I actually first reached out about small L M. I think that, you know, this is small L M 3, one of the single best open source AI research papers I've read, I think probably in like a year, maybe two years. I don't know. Like, yeah, it's very, very good. But you know, I think like, maybe let's just get a high level. Like, who are you? What do you do at Hugging Face? All that stuff.

• 1:10Speaker 02

  Yeah, so I think like, most of the people know Hugging Face for like the hub or like the transformer library. And we also have like this research team where we basically try to explore in the open all the science subjects. It could be, we have a team working on data, obviously, with the fine web, the recently released fine PDF. And we also have this team working on pre-training and training models such as small L M. And we are basically a very small team of 30 people in average. And we try to do some cool stuff that the community we love and really use to build on top of it and to contribute to the open science, basically.

• 1:52Speaker 01

  Yeah, while you mentioned the Find PDF, why don't you plug Find PDF a little bit? I think a lot of people know Find Web.

• 1:59Speaker 02

  Yeah.

• 1:59Speaker 01

  What's good about Find PDF?

• 2:01Speaker 02

  Yeah, I think I have some...

• 2:04Speaker 01

  Oh, okay, it's already in your slides, okay.

• 2:07Speaker 02

  Yeah, it's already somewhere, I think.

• 2:09Speaker 01

  Oh, wow, that's very recent, okay.

• 2:12Speaker 02

  Yeah, that's, I mean, I updated it today. So, and so it's basically cool because I think we have a lot of high-quality data set and the field of data set people have been bringing some awesome stuff. NDR, Datology, they didn't really do it yet, but this is pretty cool as well.

• 2:32Speaker 02

  And one thing that was missing, A, two or so, of course, and one thing that was missing is like a new source of data and PDF was like greatly underexplored. So that's what our team did. And you can see like the performance is pretty impressive. Like if you take this curve

• 2:53Speaker 02

  like when mixed with like other web data sets, not PDF1, it basically have a very, very, very good performance compared to this like Nemo Transc2, which is a very recent release from NDR. It's pretty cool. And there is all the pipelines, so I didn't work on this. It's Ainek who mainly did a lot of great job on this. And then there is all this pipeline where he benchmark a lot of PDF extractor. And we are cooking like a blog post where I really explain basically all the steps of this pipeline.

• 3:30Speaker 01

  Yeah, yeah, what is this "refresh from internet" step? What is that? Is that like a very common thing?

• 3:37Speaker 02

  Yeah, I think this is, so like the way it worked is that... Yeah, so sometimes like I think the PDF from Common Call are not really well extracted, basically. So I think what like he did here is that he will fetch it from internet to maybe have like a better version of it if it's like more recent or I don't know.

• 3:58Speaker 01

  Yeah, yeah.

• 4:00Speaker 02

  And there's a lot of it.

• 4:02Speaker 01

  Yeah.

• 4:04Speaker 01

  So obviously, you get data sets. I actually previously talked with Hugo Llorisong of the vision team. And I'd have fixed it in Obelix was really, was really cool for vision as well. You are on the training side. What is this unified view of model training?

• 4:22Speaker 02

  Yeah, so my, yeah, I didn't say that was my job at HAI and Face is being the guy that's like keeping touch with what's like the pre-training innovation are. And like my view of modern training, basically you have like those different objectives that you that are somewhat orthogonal to each other and you can like explore in parallel. The first one is like the, is basically maximize the relevance of the raw information, meaning like maximize the, the quality and the diversity of the data. This is probably like the most important one. It's not really the one that people are often the most excited about, but I think it's pretty

• 5:03Speaker 02

  well known now that like there is this garbage in, garbage out thing that's it's basically super important to have very good data and like a very good diversity. But you can also optimize for other things and this is quite orthogonal to the data set. So you need to have a team that build a very good data set. You also need to have a team that do research on model architecture to basically have the like construct the best model possible under certain constraints and the constraints are like the efficiency. So you want a model that will be good at inference, that will like fit on a certain number GPU, that will have a certain like KD cache consumption, anything like that.

• 5:49Speaker 02

  So this is like the, the model architecture part.

• 5:53Speaker 02

  And when you are clicks like this data set and model arch, you can also maximize the, like the information you get from the data. So at each step, you basically want to maximize like how much you extract from the data. One way to do that is to do this local destination. There is other way to do that. It's also related to, to the. To the architecture part, but there is this multi-token prediction, which is very good, for example, for coding. And yeah, and basically, there is also other steps such as like maximize the quality of the gradient you get. Like when you you do your update step, you need to to maximize this this quality of the the the dates. So this is like different optimizer, uh, thing that will approximate the the action matrix, for example,

• 6:36Speaker 02

  or just like project into better space for learning. And when you have all of this, you you you want to have basically the the right hyperparameters to make sure that your uh, like your training is not exploding, because it happens quite a lot, especially when you scale that just like your gradient or your activation are doing like completely diverging. There is all this field where you want it to be stable. You want to understand uh how to scale from a a smaller model to a larger one. And do this thing called feature learning, where basically you you are at each step learning as much as possible from this gradient. So this is kind of my view of uh model training. There is, of course, some some connection between all of them, but.

• 7:19Alessio Fanelli?

  And and for these five steps, how do you think about which ones have the highest variability? I feel like some of them, there's kind of like a meta of this is how you get the best results. What are like the parts that are like still very not figured out, where like you feel like the most interesting research is being done?

• 7:36Speaker 02

  Oh, I think data set is like the the data is like

• 7:41Speaker 02

  is a bit unexplored because uh I think people don't find it very interesting. I mean, for for a bit of time, like people were a bit under exploring like the data sets, especially since it's super important. Uh, but we we've seen like recent paper and uh recent trend to build better and better data set. And so I'm I'm confident that like uh hopefully people in the open will. Somewhat catch up with like the the the big models. The rest of it, I think the model architecture, we we like the Quen, DeepSeek, Kimi architecture. We're already at a point where this is already not optimal, but this is like pretty, pretty advanced.

• 8:24Speaker 02

  And there is all those, all these things where basically you want to to maximize, like to to to do some new optimizer and to to really have some good stability and scaling feature. This is a bit under explored and especially like since Quen and DeepSeek paper doesn't really detail how how they do the stuff. I mean, we're left with a very few model providers, even the open that talk about all of this. And for example, a good, a good way to view that is that D C G three is still using the same Adam parameter than Lamatu. I mean, there is something wrong with it, I don't know if it's just me, but I feel that

• 9:07Speaker 02

  like the the the hyperparameters for Lamatu A B, for example, shouldn't be the optimal one for this mega DeepSeek model with a with a lot of of parameter.

• 9:21Speaker 01

  Yeah. Yeah, I I I think like a lot of, it's almost weird, like, I wonder if we've hit the peak big model craze. Because now, I do you expect, you know, 10 trillion model releases, you know, 100 trillion model releases? Probably not. I think we might have peaked at two.

• 9:42Speaker 02

  Yeah, I mean, I'm almost sure you should know, like, Claude, Claude, Okuz, like G P T five, like, what's the scale of those very big models? I'm not sure it's like that crazy compared to the Kimi one or Quen that will probably.

• 9:56Speaker 01

  Yeah, yeah, both are, yeah, both are 1 trillion models. Uh, yeah, I mean, all I will say is I have, I have no insider information, but I have a lot of rumors thrown at me. And it is in the range of like single digit trillions. Um, so, cool. I mean, and then the other, the other thing, you know, I want to, we probably won't spend that much time on optimizer, but I think like optimizer is actually like one of the low key more interesting trends this year as everyone moving to Muon. I don't know if you have a strong view on this, like, basically, like, all of these five, I don't know where you want to spend the most time.

• 10:30Speaker 02

  I really like all the optimizer parts, but that's like more like a personal. I mean, I will not probably talk about this a lot, but that's one of the things that I find cool because it's very recent, basically, and it's like using some, I mean, it's very recently that people start using something different than Adam. We know, for example, that like Google is using, I mean, we don't know it, like on like all paper, Google are using like Ada Factor, but recently before this K2, everyone was using Adam, AdamW. And recently, there was the GLM paper, there was this K2, of course,

• 11:15Speaker 02

  that is trying to do some new stuff with optimizer and I find it pretty cool because it's like very deeply influence the long process of the model. This is actually quite a big of a change, I think.

• 11:26Speaker 01

  Can you give an intuition on what's going on? Like, I think the basic insight of Adam, AdamW was mostly like momentum, right? Momentum matters, you trade off, you use more memory and like, but in exchange, you get much better learning, like, okay, fine. What is the core insight that everyone sort of like exploring?

• 11:43Speaker 02

  I think for, so I actually have some stuff about it somewhere. Somewhere along.

• 11:50Speaker 01

  Somewhere.

• 11:51Speaker 02

  Yeah.

• 11:52Speaker 01

  I love it. This is like just your notes of all the papers.

• 11:56Speaker 02

  Yeah, this is exactly my notes, it's very, I didn't publish them, to be honest, so.

• 12:01Speaker 01

  It's great.

• 12:02Speaker 02

  And this is like the new algorithm, which is very simple, to be honest, and like the key part is this like Newton Schults method, which is basically taking this like this matrix and making it better in a sense that it will be somewhat orthogonal, orthogonal, to be precise, I think. And this will have the property of like basically spreading the learning in more dimension than with this.

• 12:33Speaker 02

  Adam Optimizer. And I think this is why. It's what they say, yeah, I think.

• 12:41Speaker 01

  Yeah.

• 12:42Speaker 02

  Yeah, it's it's more stable. Uh, and also it encourages more to learn and explore new things. And this is pretty cool. And they they did this new clip optimizer, which is pretty cool. And actually, it's quite simple. They just track the this quantity, which is the max logit per head. And if it's this quantity is too large, they are basically saying, OK, that's that's not good. I rescale after the backward my my parameters by descendants. Uh, to to make sure that they they are they they scale down and they don't explode anymore. And this is actually it only have an impact in the on the early step. And after it stabilized and basically don't do. Like it doesn't have any impact.

• 13:26Speaker 02

  At this point, you can deactivate the. The QK clip and this will just work as well. But this is pretty cool. And they they find it by just looking at the dialog and they see this this crazy maximal attention logits. And they are like, OK, there is something wrong and I need to to contain those attention logits. And the method they choose is this QK clip stuff. There is actually different stuff to to do that. There is like this I don't Z loss, there is the the QK norm, there is a lot of stuff that's kind of want to patch the same thing. But this one is quite elegant, to be honest, because it's very simple.

• 14:04Speaker 01

  Yeah, yeah. Uh, cool. Uh, do do you, uh, just a side follow up? Uh, did you actually follow the sort of schedule for optimizer stuff from last year with Aaron DeFazio? Anything came out of that?

• 14:14Speaker 02

  I don't have a strong opinion. I tried it, uh, like a few days after it was released. Because I was, we were writing a paper about the the WSD schedule. And so this was kind of an alternative to it. And it didn't perform as well as like a simple Adam W. And it was more sensible to some to the beta one and beta two. But I think like with this kind of stuff, you you really need to have this this knowledge on how to optimize. Uh, those parameters for each optimizer to really get the best performance of each one. And so I don't want to to say something bad about it because basically.

• 14:55Speaker 02

  At nineteen percent, there is a ninety percent chance that it's just me that used the wrong hyperparameters. But yeah.

• 15:02Speaker 01

  Yeah, you're holding it wrong. Cool. I mean, I'm happy to zoom out. Sorry, I feel like I zoomed in on optimizers just because you're one of the few people that we have on the podcast that can talk about it.

• 15:13Speaker 02

  Yeah, and there is a lot more to that. There is MIO, but there is also shampoo. So there is a bunch of series of optimizer that passed time, but.

• 15:21Speaker 01

  If you, do you want to keep going? Yeah, I was.

• 15:23Speaker 02

  I don't, I, I'm not sure I have the, the slides for shampoo and stuff. I like this slide. This is from my previous set of slides. Basically, I was like, looking at what the lab are like using and I, I basically saw that every open model we are using Adam.

• 15:43Speaker 01

  Yeah.

• 15:44Speaker 02

  It was kind of crazy.

• 15:45Speaker 01

  It's pretty validated at this point. Yeah.

• 15:47Speaker 02

  Yeah, and since then, there is this new MIO trend. There is this recent paper talking about the different optimizer.

• 15:54Speaker 01

  And this is this week.

• 15:57Speaker 02

  Yeah, this is this week.

• 15:58Speaker 01

  Wow, these lights are super live. Oh shit. Okay.

• 16:04Speaker 02

  Yeah, this is this week and this is basically saying that, so there was a lot of papers that claim, claim some X speed up and things like that. And what they find is that the speed up is greatly, greatly exaggerated and mostly because often people like undertook the Adam W baseline. And this is like something which is pretty, pretty, like we, there is always this problem where people are doing new architecture, for example, they are under, under like the baseline of the transform is often under optimized.

• 16:43Speaker 02

  And so they, they basically say that when you increase the model size, the speed up is decreasing. But to be honest, even the author, I think, say that this point, they didn't really optimize their parameters. They just like do some, I would say that, do some extrapolation and like extrapolation can be a bit tricky. So this, maybe we can forget this one, but I think the, the, the speed up. Between one point two and one point three is pretty fair. And there is a lot of insights such as like if you don't do weight decay, you can get those kind of curve. And if you, for example, take a look at like a step uh uh ten, ten thousand, you you can conclude that uh the the one with uh weight decay equals zero is the best. But in fact, at the end, it's it's not at all.

• 17:29Speaker 02

  So early curve can be can be a lot like can mislead a lot the the interpretation. So it's yeah. And also, like, you you see here, if you you don't tune enough your hyperparameters, for example, uh this is the two value they they take for soap. And in one case, it's very bad. It even has like this spike, which is not great at this scale. And in the other scales, it's like beating the the other also. So yeah, it's pretty, it's pretty complicated to do out of that, but yeah.

• 17:58Alessio Fanelli?

  Is there any value in the early curve? Like, even if it's like, you know, not predictive of like the whole training, like seeing kind of like a faster slope in the beginning?

• 18:08Speaker 02

  Yeah, so what I look uh when I when I look at the early curve is mostly the spike. If there is some spike early on, it never a good sign for the for the for the rest of your training. There is some, for example, if you compare different data sets, the early curve can be can have can give some information. It also depends on the the schedule, your the learning life schedule you are doing. Uh, we we often use this W S D schedule, which means that basically you you you do this like. As I go, there is this stable phase. And why it's great is because you can basically decay whenever you want. And so you do one run, uh one big run with, for example, a fixed setup, and then you decay uh at different points.

• 18:52Speaker 02

  And then you can get uh this performance without doing like a different course design schedule for uh for all of them.

• 19:01Speaker 02

  But overall, especially when doing optimizer ablation, you never want to look at early curve and you want to wait for the long rate to have fully decay and like the ending to be to be to zero or whatever value.

• 19:21Speaker 01

  Wow, this actually is a really good paper. I like, there's a lot of.

• 19:25Speaker 02

  Yeah, it's a great one. And the people who are doing it, like the mine project at Stanford, they are really great and they are doing some amazing open source stuff. So I really encourage people to to check out what they do.

• 19:39Speaker 01

  Awesome.

• 19:40Speaker 02

  Yeah.

• 19:41Speaker 01

  Okay. Anything else on the optimizer world that's going on? It's like, interesting.

• 19:47Speaker 02

  Yeah, there is this this plot from the the Asian Solar AI paper about MIO. And the idea of this plot is basically saying that you can use MUP to, I mean, you can use some technique to to do hyperparameter sweep. And it's working with MIO, for instance. And there is this idea where you, so in this, in this paper, they only did one D hyperparameter sweep. So they fixed everything and they just changed the long rate, for example. But for instance, there is some connection between the long rate and the weight decay or long rate and batch size. But you always want to sweep together some of

• 20:29Speaker 02

  the hyperparameters. And this is what they they do here. For example, they they sweep both the the long rates and the weight decay together to to find the optimal one for for their training.

• 20:41Speaker 01

  Yeah.

• 20:41Speaker 02

  And I think it's optimal now.

• 20:44Speaker 01

  I just realized this is a four dimensional chart.

• 20:46Speaker 02

  No, three dimensional.

• 20:48Speaker 01

  It's three, but then, you see that? That's the fourth dimension. Yes, correct.

• 20:53Speaker 02

  Yeah.

• 20:55Speaker 01

  Very interesting. What else comes to mind for you? I I think like, you know, we're kind of on the model consumers side. And so there's some of these things like, you know, that get to the engineer side and we think are really interesting. Yeah, like like sparsity matters to us, like efficiency, you know, speed.

• 21:15Speaker 02

  I think the the M O E is definitely something that is interesting to understand.

• 21:20Speaker 02

  And it's definitely interesting to understand how powerful it is. So for instance, like the the first graph are basically saying that you get better performance for the the same number of flops. But there is this problem and it's basically that for example here you you get like

• 21:39Speaker 02

  a three X performance boost. But here you if you if you look at the training times, you only get like a two X performance boost. And this is mainly because M O E are faster to train, but this is depending a lot on your code base. And there is actually some very small detail that can completely make your M O E very bad. I think one is this.

• 22:09Speaker 02

  Is this one is like the this is from Quinn and they are basically saying that so I don't know if I I need to explain, but like the the load balancing is basically saying like you you have your different experts and you want all of the experts to get approximately the the same number of token at each batch or at the end, because if you only have like two experts that activates every time it basically like the the doors are sparse. Will will basically be be useless. So one of the the very important part is this load balancing. And what they find in Quinn. So there is different way to do load balancing. I can talk about this later, but

• 22:56Speaker 02

  what what they find is basically you need to compute those two statistics. Which is at each step, you look at how many token act actually selected by each expert. And you look at the probability that is affected by by the router basically. And when you compute this

• 23:18Speaker 02

  this statistic, you you are like doing that at the micro batch size level. So this mean that for instance, when you do training on multiple G P U's. Let's say you just replicate the the model on all the G P U's. You have this global batch.

• 23:35Speaker 02

  You have this global batch, so it means that you for example, you you give a hundred document to to your to your model. And if you have five GPU, you you give like twenty to each of them. Because you want to parallelize this process. And then after you you gather all the results together and and you do the backward pass. And the fact that there is all they the statistics compute at the the mini batch size is like pretty pretty important for the model. You can, I think, the the most important point is this one. So basically, this is saying that the model is not really activated for code. Like, there is no expert that are specifically activated for code. Same for math, same for literature.

• 24:18Speaker 02

  So this is like, um, all of this is like the the local batch. And if you do it at the global batch, meaning that you give, uh, at at each step, you you give like a north token, basically, so that the statistic is meaningful. You get this very good and, uh, like, this is basically a sign of of good health for them. We need to have all those activated, like, specific activate activation for each domain. And they find that, for example, in the the Megatron item base, this wasn't the case. This was done as the local batch, um, I think. So it basically means that everyone that is using Megatron at the time of the paper, uh, it wasn't even a paper, it was just a blog.

• 25:01Speaker 02

  But they they had this, they they were in this region, basically, where where the the MOE were not like very specialized. And this is super important for N N usage, because you you want to have like specialized experts, so that they really learn something different from each other. Even after, for example, you can do some pruning. And if you want to model only on code, for example, you can just prune out some some of the English literature experts and just keep the one that are activated for code. And then you can initialize your MOE with those and like you bootstrap them. So there is like a endless possibility here. And, uh, and yeah. But this is like a very small detail and this has a huge impact. Very crazy.

• 25:45Alessio Fanelli?

  You mentioned it's something that, you know, personally, I never trained a model. How easy is it to divide these skills? So when you say, you know, literature, math and code, what, can you maybe run people through the process of like, how do you kind of like funnel these things and different experts? And then is it just as easy as you said to like, I'll just delete the literature part? Or is there some sort of like code dependence between them?

• 26:09Speaker 02

  Yeah, there is some code dependence, definitely. So there is other way to basically limit this code dependence, but there is definitely some code dependence. And when you do printing, I don't know if people are doing that. I think that's interesting to explore. But like at least in the open, people are not really like printing. For example, the English data, or that was just some research she IDed that could be called in the future. But, uh, but yeah, but the specialization is actually done in the router layer of your Amui. And this is pretty important, basically, because this is like the, the, the step where you will learn, like each token will learn to which experts to go, depending on like the, the meaning of each part.

• 26:54Speaker 02

  And we can even have some fancy stuff. So this is like from two weeks ago. This long cat flash from the DoorDash of China paper that really is by the company that basically really is like a state of the art model. It's just pretty crazy.

• 27:10Alessio Fanelli?

  This is, this is the most recent presentation I've ever seen.

• 27:13Speaker 02

  Yeah.

• 27:15Alessio Fanelli?

  This is all like happening in the last two weeks.

• 27:17Speaker 02

  So yeah, I mean, I'm trying to, to stay in, uh, in touch with what's, what's happening. And that is this dual communication expert, which I find pretty cool. I think it's the first time I see it in like a big model. I mean, I'm pretty sure it's the first time. And like it allowed the, the model to basically say, this token doesn't, uh, like this token is like, uh, for example, very easy to predict. So I should not, uh, have, uh, like allocate a lot of flops to it. But this, uh, this token, which is for example, for, uh, to, to, to answer an M M U question, is very hard. So I will activate more of the, of the real experts. But those are like just residual connection, basically.

• 28:03Speaker 02

  And is the thought that I seen that? I mean, it's great, but I mean, they they have a very strong team because my guess is that it's pretty hard to to like balance it. And to make the the training stable and they also have this chain or so, which is, I mean, I'm not trying to understand why they do that. I think I understand, but it's it's pretty novel as well. It's basically replacing the shared expert with some some double attention layer. Yeah, I mean, it's it's pretty crazy. There is a lot of question for MOE, basically, do you want to to have shared expert or not?

• 28:39Speaker 02

  GPT OSS Coin 3 doesn't have shared expert, but like the the Coin 3 next, which is like a release, like a, I mean, they didn't release it yet, but they submit the the PR to Transformer.

• 28:52Alessio Fanelli?

  This is literally last night.

• 28:55Speaker 02

  This is this is not last night, this is like four hours ago, I think.

• 28:59Alessio Fanelli?

  Oh my.

• 29:00Speaker 02

  But.

• 29:00Alessio Fanelli?

  Oh my god.

• 29:01Speaker 02

  But yeah, and and they basically have shared expert this time. So this is interesting. I think shared expert are pretty important to, they basically act like as like the the knowledge that you don't want. Like you, you want your expert to be very specialized and you want the shared expert to be just like the basic stuff and anything like that. That's the intuition behind it. And I think DeepSeek is using that as well.

• 29:26Alessio Fanelli?

  What do you see as like the even more granular MOE future? You know, I I think it makes sense for people say, hey, it's like code and math and literature. Do you see a future in which the MOE model is like, you know, front end expert? Like, you know, if it's like a type script code base, it's like, oh, this expert, like the routes is an expert at, you know, the component is an expert at the test writing like. Is that something that makes sense? Are we bottlenecked by the data or what are people doing there?

• 29:57Speaker 02

  So my intuition, I don't really know, but my intuition is that, for instance, between math and, I don't know, all the history, like the Greek history, for instance, not really the Greek because it's too much, but let's say code and Greek history, there is not a lot in common, except just the fact that the language is the same. But between like, for example, I don't know, TypeScript and JavaScript or Python, and I don't know what language, there is more similarities. So I'm not sure if it's like, there is maybe more transfer learning, meaning that a model trained on like Python and JavaScript will benefit from the JavaScript data as well.

• 30:42Speaker 02

  I'm not sure if the king's fine, but for Python, Greek history.

• 30:46Speaker 01

  Yeah, I think like the question of semantic experts is always something that people are wanting to have intuitively, but the machine in machine learning terms, it never works out that way. I actually feel like people are moving towards more semantic experts. It's like somewhat started by Apple, because they want like sort of downloadable Laura's and stuff. I remember, oh, Gemini 3 nano, 3N, also had this, where it's like downloadable, offloadable weights. This is like a really severe.

• 31:18Speaker 02

  Gemma 3N.

• 31:19Speaker 01

  Gemma, Gemma, Gemma, yeah, yeah, yeah.

• 31:22Speaker 02

  Yeah.

• 31:23Speaker 01

  I see, I see some moves. I don't know how serious this is, this thing is, but like if you're, if you're going to the point of having a router for different models, like semantic experts at some point makes sense.

• 31:33Speaker 02

  Yeah, and what I like is, I think there is this recent trend where OpenAI like basically saying that they, they want like a router for a user to be, to, to, to have like some, and I think it's great for.

• 31:48Speaker 01

  You might as well.

• 31:50Speaker 02

  Yeah, and and this is kind of the same principle, but in baked inside the model. So I find it pretty cool because it's basically saying, uh, if my, if the request that I have is easy, I will just not allocate, and I will basically do inference with a few expert, which is the equivalent of GPT-5 menu for input. And if I have like a very strong question, I will basically activate all the experts. But they they do it at a scale where it's basically varying between, uh, things like there is the the variance like four billion parameters. So it's not a lot in GPT-5 menu and the the big one now don't have the same number of parameters.

• 32:33Speaker 02

  But this is, this is cool to see those behavior cooked and baked into the into the model.

• 32:40Speaker 01

  Yeah.

• 32:41Speaker 02

  Yeah, I don't know. I can talk about M O, uh, or other stuff, but you don't.

• 32:48Speaker 01

  Yeah, guide it however you like. Uh, you know, I think the other, the other trend, uh, what what definitely was more granular, um, um.

• 32:56Speaker 02

  Yeah.

• 32:57Speaker 01

  Uh, experts, uh, it was going up to like a hundred and twenty or something. And I don't know if you have a chart here. Okay, it looks like some kind of efficiency.

• 33:03Speaker 02

  Yeah.

• 33:04Speaker 01

  Yeah, go ahead.

• 33:05Speaker 02

  Exactly. And this is like basically, I think, so this is like the active number of parameter here and the total number of parameter here.

• 33:14Speaker 01

  Yeah, yeah.

• 33:14Speaker 02

  So this is basically the the M O is passing it. And you see that. So those are like the kind of old model and they are like not really great in terms of spacity. But if you look at the new one, uh, there are even the like this communicator is pretty, pretty insane as well. I think like the, the, the new plan, they would use will be somewhere. Yeah, I think I'm not wrong. If I read it well, so this will be.

• 33:41Speaker 01

  Quen Quenex or Quenext.

• 33:43Speaker 02

  And three.

• 33:45Speaker 02

  The next?

• 33:46Speaker 01

  Yeah.

• 33:47Speaker 02

  I think this could be.

• 33:48Speaker 01

  Yeah, I would say, from what I've heard, Gemini Flash is also pretty strong around this.

• 33:54Speaker 02

  Yeah, I mean, M O E, like, we're this kind of pioneer in this M O E field. Think M O E is from them as well, so. So.

• 34:03Speaker 01

  Yeah, of course, yeah.

• 34:04Speaker 02

  Everything is from him, so.

• 34:07Speaker 01

  Everything is from him. I think, like, what was the difficulty for being this sparse before, right?

• 34:13Speaker 02

  Yeah.

• 34:13Speaker 01

  Is it just stability or training?

• 34:15Speaker 02

  So this is definitely one thing. Also, I think, like, training codebase, you really need to have an optimized infrastructure and optimized training codebase to have this good sparsity. Because the way it works is that you basically are distributing the expert on different G P Us. And in the M O E layer, they basically need to sync before, because like the router is dispatching the way to the token to the different experts. And basically, the more granularity is, I mean, the more granularity is, the more this step can be costly in terms of communication. So this is one of the reasons why it wasn't very good.

• 34:58Speaker 02

  For example, we tried, we are training a M O E currently at Target Face. I mean, we'll train soon, start the training soon. And we tried with Megatron and we benchmarked, like, for example, the Mistral architecture with the Queens three, this one. And at the start, before any optimization, like the Mistral one, was very fast and had very good throughput. And the Queens three was very slow to train. But we, basically with the new kernel, the new, all this new stuff, it became feasible to train. And there is also this sparsity.

• 35:38Speaker 02

  Uh, this like it can basically make the train sometimes a bit less stable. And kind of a bit thing, I don't know if uh. Well, for example, this old MOE paper, they basically benchmark uh this number of experts. So sixty-four and thirty-two, and they didn't find like much difference between them. But if you look at like a more recent sparsity scaling law, you'll find that there is like this is basically a uh the the difference uh sparsity coefficient. And you find that there is actually quite some improvement and of course there is like diminishing return. But you you can increase the sparsity quite well, and it's very good at inference.

• 36:20Speaker 02

  I think that's the main argument that that inference is very good because you can have very fast inference because there is only a very small subset of the parameter that is activated.

• 36:30Speaker 01

  Yeah, I I think one of those things that's kind of interesting as a pattern is like MOEs as a as a production artifact, but not as a training target. So the training target, you you just have the dense model and then you distill to the MOE model. Yeah, I think like I I think this is the pattern for every frontier lab now. And I'm just not sure, like, you know, obviously, everyone's, why why do you just train the the MOE directly? Uh, it's because of stuff like this.

• 36:56Speaker 02

  Yeah, yeah, I mean, yeah. I think they also train MOE uh from the start, but maybe less sparse, you know. They basically do some more sparsity on the production model than for the the one that is used uh to distill uh from. Actually, this situation is like quite a big thing that people in the open are not really really doing yet, I think, but uh but yeah, I mean, I think that's it for for me.

• 37:27Speaker 01

  Nice, nice check. Uh, I think that the the other thing, you know, uh, especially you mentioned the G P two five router is the difference because like MOEs have routers, they went putting a router on top of another router. Like, it's ridiculous, right? Like, okay, what is the future of routers, right? Like, is there a fat router that is basically its own L L M? What does that look like? If you if you just extrapolate, like, what what does it look like in the future?

• 37:52Speaker 02

  Like the feature of the router?

• 37:54Speaker 01

  If if we're if we're going to the point of like putting a router on top of a router, isn't that the same job?

• 37:59Speaker 02

  Yeah, and I'm actually, I don't have it in here yet, but let me screenshot something quickly from another paper, but yeah, if I go back here. There is this, um, this NSA, which is like the, the new deep seek attention. And it's basically a router inside attention. So you can see it as like a router inside attention, basically. Because, so there is this compression part, there is this sliding window attention part, so yeah, let's say. And there is this, this part where you are basically selecting which attention block. It's interesting for the specific token, so you are routing the attention of the token to some specific block of the previous one.

• 38:44Speaker 02

  So you have also some routing inside attention happening with those new sparse architecture. There is also municipal that have a very similar, similar attention mechanism. But yeah, I think it's interesting because basically like, like the more router you have, the more parameter efficient your model is. I think that's the impression here, especially at inference, because you only activate a subset of it. And that's like the same logic for the, the, the model router, basically, because you, you are selecting the model with like fewer activated slots for, for 35 minutes compared to the big one.

• 39:21Speaker 02

  For M O E, you are basically saying, uh, I'm only selecting a fewer token or I'm only selecting subset of it. And, and you also have this same logic into attention, so no, yeah, but yeah, it's pretty cool. Yeah.

• 39:37Alessio Fanelli?

  Um, can we maybe talk about what comes out of the training? I think this is like, I, I don't know if there's any other part that maybe you think people don't cover enough or like.

• 39:48Speaker 01

  Definitely data set, you know, there's a, there's a big chunk there that we haven't covered, but yeah, up to you.

• 39:53Alessio Fanelli?

  Yeah, I get, yeah, I think in, I, I guess like in and out, so like, so this is the part that, you know, expert ML engineers like you do, and then there's maybe the two things that people who consume the models can do. One, they can build data sets, even if they don't know how to use it, because then they can go to some fine tuning platform or training platform. And then there's the other side of post-training. So yeah, maybe let's start on the data set side.

• 40:20Speaker 02

  Yeah, I mean. Yeah, I think to answer also on the question, I think like the the goal of of like uh ML researcher is basically to to cook the the best model possible so that the user don't have to do much with it.

• 40:35Speaker 01

  Uh. Yeah, out of the box, zero shot.

• 40:38Speaker 02

  Yeah. That that's, I think, pretty cool. And obviously, a big part of it is like the dataset. Uh, there was this recent paper, like the Beyond Web from the the Dataology.

• 40:50Speaker 01

  Yeah, we just had the Dataology episode, I think, last week.

• 40:52Speaker 02

  Yeah, I saw this, but it's pretty cool.

• 40:55Speaker 01

  Yeah. So this is super recent again. I've not used to talks being this recent.

• 40:59Speaker 02

  Yeah.

• 41:01Speaker 02

  Yeah, so I I mean, I've read it. I mean, I've read it fully, but I didn't spend that much time on it. But if if I had to to recap a bit the the paper, the the interesting plot are are this one. And so this is like, I think, the the the the interesting one is that you basically want to rephrase. Um, so this is about rephrasing. So basically, you have so when you like there is the the web that has it is is shit, to be honest, like there is a lot of very bad content in it. And there is one approach, and what we do in Fine Web, Edufi, for example, that is basically filtering some some educational content or some subset of the internet, which is often higher quality.

• 41:47Speaker 02

  And the question is like, what do you do with the lower quality data? I think that's all I see. That's not exactly what they say in the paper. That's uh more my thought. But I think an interesting result is this uh is this um is this graph. And basically, this is like the curve between low quality web and high quality web. So this is the web, basically. And if you only uh rephrase the low quality parts, you get this curve, uh the light blue one, which is kind of close to to the one where you also rephrase. I know, it's not that, but it's like, it gives you a a big boost. And I think, I'm not sure if they do the application here. But I'm not. Yeah, okay. So that's what they do.

• 42:31Speaker 02

  They they also try to to rephrase like in the the high quality subsets, and they also re-put uh the dataset they rephrase on. And there is only a small difference. But you could think it's because of the repeat. I'm not even sure it's because of that, to be honest. And the, the, the, the thing that you, you need to understand from it is like, uh, it's very good to rephrase the low quality data, uh, to have like just better, better benchmark and better performance. And, and yeah, and rephrasing is, is a bit, uh, is a bit tricky to do. Uh, I think a lot of it is making basically the right prompts. And one of the, the good prompt, I mean, the good style that the, the model learned,

• 43:15Speaker 02

  the model like, especially because the benchmark I constructed this way, is the conversational data. So the Q and A, for example, answer. Uh, so example, MMLU.

• 43:28Speaker 02

  So this is from small M to, uh, uh, training, so which is, uh, 1.7B, I think.

• 43:35Speaker 01

  Yeah.

• 43:36Speaker 02

  I know. I didn't think.

• 43:39Speaker 01

  Very small.

• 43:40Speaker 02

  And, uh, yeah, very small. And this is basically until like 6.5 trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA formats, the meaning that the, the, the model have to select, uh, which answer is, the model have to output, for example, the, the right answer is A or B or C or D. And this is random, like for 6 trillion tokens, random. So this is a big deal. And it only learn after like 6 trillion tokens to, to, to basically answer question in a QA format. And this is because we, we didn't do at the time, a lot of, uh, we didn't include a lot of QA, uh, Q and A, uh, data

• 44:25Speaker 02

  in our training set. And this is also from a block from Datology. And you see that their training sets, like the, the, their data set is, is very good at MMLU. Uh, and this is, this is like mainly because of the format. Like all of our dataset are random because they don't have like Q A formats. And I don't, uh, they don't disclose it, I think, but uh, maybe they do to be honest. But this like the fact that they gain accuracy on M M L U and this not random is because there is some Q and A uh formats uh inside of that set. So you see like, if you just look at the benchmark, uh, you could say, uh, for example, like this C L M is very bad or it's very bad quality data, I don't know. But with a very light rephrasing and a, uh, uh, let's say a style rephrasing.

• 45:10Speaker 02

  So you take the document from whatever that I said and you, you put it in Q A format. You would have like normal random performance on M M L U. But yeah, this is, this is very.

• 45:21Speaker 01

  You're very, very sure it's not training for tests, right? Like, uh.

• 45:24Speaker 02

  Yeah.

• 45:25Speaker 01

  This is obviously the, the, the problem.

• 45:28Speaker 02

  Yeah, there is obviously some contamination, but uh, I mean, thirty percent is not that much, to be honest. I mean, it's, it's great, but uh, it's not like hundred percent or something like that. But I'm not sure it's contamination. It's more like Q A, Q and A formats and uh, and that's what they, they also find here is that you basically have better performance where you, you put some conversational data uh in, into the, into the training. And this basically plateau at twenty percent, which is uh, it's good to know.

• 46:01Speaker 01

  So yeah. Okay. And, and you know, one of the reasons I, I obviously, I think we've been following the rephrasing, the web research for a while. I think one of the reasons that like I was a little bit anxious about this is that now the dataset is too clean. It's too perfect, right? I pasted in a lot of shit in my, in my prompts, right? So can it recognize my typos? Because if it's never seen a typo in its life, it's not going to be uh, understanding typos, right? Typos are just one simple form of mistake. There's grammatical mistakes, but then there's also mild conceptual mistakes, all of which gets cleaned up with, with uh, rephrasing. And sometimes that's bad.

• 46:39Speaker 02

  Yeah, I totally agree with you and especially since like all the benchmark we have, uh, we don't have like typos, uh, benchmark, I think. There is no, no such things, uh, I mean, they, they probably is to, to be honest, but I don't like, people don't usually benchmark against that. Obviously, the dataset innovation and the model innovation like always are biased towards maximizing performance on some specific set of benchmark. And you kind of miss those use case, as you said, which is our real case. We are like user, like the thing that you now care about, to be honest.

• 47:20Speaker 02

  But I'm not that worried, because I think the solution is just to include some bad data as well. And the model is just learning, it's just learning it. And you can probably include, there is probably some scientific ablation to make on that. But you train on so much trillion of token data that, to be honest, I think the model have time to learn that sometime users are typing bad stuff into their prompts.

• 47:46Speaker 01

  Yeah, you know, I still see, I feel like there will be some trade off somewhere. Because like, you know, obviously you have much, much more, much, much fewer typos. And not just talking about typos, right? I'm talking about like, oh, copy and paste like website data and like just throw shit in there. Like, because the pre-training data is bad, I actually have confidence that you can handle bad data, because I'm going to paste in that bad data. Anyway, so separately, I also wanted to mention, do you cover the Tencent Trillion Persona paper? Because like Persona rephrasing, billion persona, trillion persona. Basically Persona rephrasing is a very interesting way of doing multiple epochs. And you'll never repeat yourself, but it's actually kind of the same content. It's just you're rephrasing.

• 48:28Speaker 02

  So I think that's kind of what they did in Kini K2 ablation.

• 48:33Speaker 01

  Yeah.

• 48:34Speaker 02

  Yeah, I'm not sure, I didn't read the paper, I will read it probably after this.

• 48:39Speaker 01

  Basically role play, right? Like you're a doctor, you're a physician, blah blah. And then like, it's slightly different?

• 48:46Speaker 02

  Yeah, I totally agree with you. I think that's pretty, that's pretty important. And you know, there is this paper from, actually from people at Hugging Face at the time, that is saying that you basically can repeat your data like up to three epoch before seeing diminishing gain.

• 49:07Speaker 02

  Before dimension return.

• 49:09Speaker 01

  Four is what I remember, but yeah.

• 49:11Speaker 02

  Four, yeah, I don't remember exactly the number. But I don't think they do a ablation on like the different data sets. So with, with Lumen, which is Lumen, Benal.

• 49:23Speaker 01

  Yeah, she spoke at my conference last year, yeah.

• 49:26Speaker 02

  Yeah, yeah, I was there, my boy.

• 49:28Speaker 01

  Oh, yeah, thanks.

• 49:29Speaker 02

  Was in the public.

• 49:30Speaker 01

  We're, we're gonna do that again in Europe.

• 49:32Speaker 02

  Oh, at NURIPS?

• 49:34Speaker 01

  Yeah, are you, are you coming, San Diego?

• 49:36Speaker 02

  Uh, probably yes. I, I'm not sure. I didn't book my flight, but, uh, but I think we'll be coming. And, and I think this is like heavily dependence on, uh, on the data sets. And this is very under explored. For example, uh, we have like the, the fine math data set, which is very high quality math data set. We also have like the, the, the stack ADU one. And I'm pretty sure that if we repeat more time the, the fine math data sets, this number of, uh, three X, we probably, we can probably repeat more of high quality data set because it contains like more information. And what you say is like with rephrasing, you can pretty much get

• 50:18Speaker 02

  probably, uh, like something, I don't know, I didn't do ablation, but you can probably do like this, this standings factor when you can repeat your data by rephrasing it a bit, uh, a lot. Uh, like you can repeat, uh, repeat it quite a lot. And, uh, I think this is quite under explored and I would love to, to see some papers and or maybe I would do it myself. But, uh, but this is pretty cool. And this is kind of good old, uh, data augmentation that we, people were doing like during the time of computer vision. Uh, with basically, uh, I don't know what they were doing, but they are like shifting the image, zooming out, zooming in, uh, adding some bit of noise, stuff like that.

• 51:01Speaker 02

  And they, they, they had better generalization, that the train and test loss were a bit better. So yeah, I think this is still an open question, at least in the open, uh, at least from my knowledge. And this is pretty, pretty important one.

• 51:16Speaker 01

  I think the, the last data set piece question I have, and I'm happy to change topics, is, uh, one of the lessons that we had from the dataology episode was that effectively what we're doing was removing the post-training data into the pre-trained data, right?

• 51:29Speaker 02

  Yeah.

• 51:29Speaker 01

  That's, you have, you talked about this conversational rephrasing. That's instruction tuning.

• 51:34Speaker 02

  Yeah, I mean. There is all this mid-training stuff that's. I mean, you you can inject the data in post-training, mid-training, pre-training, whatever. I think the the what matters is that you. I think the what matters is like the model see your data. And post-training should be very to to very like to to basically sharpen the distribution of your model to to very specific topic. And so it's for me, it's good. And that's what people do nowadays like G N M. Uh, I think they train on, I don't, I don't remember exactly, but I think it's like eight trillion uh token at the mid-training state. Can you can you call it mid-training? I'm not sure, to be honest.

• 52:18Speaker 01

  Uh. Whatever you want.

• 52:20Speaker 02

  It's pretty crazy. Yeah, eight trillion tokens is like, I mean, and they they train only on code and math and they do some refining as well. And uh, and yeah, I think uh, I think that's basically what a lot of people are doing now. And we are doing this as well. We are we are basically introducing higher quality data like um.

• 52:42Speaker 02

  I don't know why it is, but if you look at the. Uh, different phase, we are basically adding more code and math uh through the the train and less web data. Yeah, but we're only doing it like one point five trillion tokens.

• 53:00Speaker 01

  Yeah. Yeah, bu- by the way, oh yeah, this this chart, we haven't pulled it up yet, but like this chart I was. I was, it was what wowed me on the small island three. I was like, this might be the most open paper I've ever seen.

• 53:12Speaker 02

  That that's what we try at least to be to be as open as possible. We are so busy like all the checkpoints, the intermediate checkpoint we have. I mean, we didn't release it all, but every ten billion token, I think, or.

• 53:23Speaker 01

  Mm hmm.

• 53:23Speaker 02

  No, I think fifty billion. Uh, I don't remember exactly what I pushed to the. But if we like fifty billion token, uh, I think there is some checkpoints. And yesterday I was talking about uh some stuff about pre-training. And someone was saying, I hope like it would be cool if you release the the the intermediate checkpoint because I need to do some analysis about the. The pre-training uh din- dynamic. Into small M3. I mean, it's, yeah, you can use it. But this, this is pretty cool. And I think that's what we try to do. We also open source all the code base. So there is the Nanotron, which is doing the large scale training. There is the DataTrow, which is doing the data featuring. I think this one is very underrated. And it's, it's very good.

• 54:06Speaker 02

  That's, that's what we use to, to define web and find PDF, yeah.

• 54:10Speaker 01

  Uh, should we talk about that? Let's, let's, uh, let's open it up. I, I've never seen this, never heard of it.

• 54:14Speaker 02

  Yeah. DataTrow? I mean, I mean, it's just a library that's, that do data featuring and tokenization at scale. So you basically, it's very efficient to, to do this. And we do it on CPU mainly. And there is also now, I think we are doing some pipeline for doing rephrasing. Also inside the, the DataTrow library. There's also the Light eval one to do evaluation. And we are, we are doing a lot of stuff there.

• 54:38Speaker 01

  Yeah, Light eval. I mean, I think a lot of people are like trying to move from the Hugging Face Harness and, uh...

• 54:44Speaker 02

  Yeah, the, the Hugging Face Harness.

• 54:47Speaker 01

  Yeah, yeah, yeah. Um, is Light Eval the new standard now? Should I, should everyone be running that?

• 54:52Speaker 02

  I mean, the, yeah, I think it's pretty good. The data logic, the B and R paper, they use Light Eval.

• 54:58Speaker 01

  Yeah. I, I think the community should have one. Uh, it's just that everyone is like, even, even the Hugging, even the OpenAI evals framework was so heavy that they, they had to, they had to release like, okay, here, our main framework is too heavy. Here's a simple evals. And then, and then they don't even use that. So I'm like, I don't know what to use, man.

• 55:14Speaker 02

  There is the eval framework, uh, every day.

• 55:18Speaker 01

  Yeah.

• 55:18Speaker 02

  Yeah, I think there is a lot of, uh, of some. But, uh, but I'd like to think that this Light Eval one is, is great. I think one thing which is cool at working at Hugging Face is that we, we, we are this multimodal team, which is doing like the small VLM. We have the robotic team. To be honest, I'm not really involved in that, but the, the, they are doing the, the small VLM, uh, model. A small VLA, sorry. Which is like the, the vision language, uh, action stuff. But this is pretty cool to, to have, like, all those people working with the, we are like our small LLM model as a backbone and then building some, some stuff around it. Even inside, uh, uh, our symmetry in base. So this is pretty cool.

• 55:57Alessio Fanelli?

  Yeah, so many memes.

• 56:00Speaker 02

  Yeah, I mean, I didn't make all the meme yet, but I have so much like this one was like, so this was for small M3 and I was doing ablation on Optimizer. And basically, I I was like, I mean, when I when I fell asleep, it was like here, so there was no spike at all. And I woke up and my graph looked like this and I was like literally crying because all my training was diverging.

• 56:28Alessio Fanelli?

  This is the quantum observer effect on training runs when you're watching it, nothing happens.

• 56:33Speaker 02

  Yeah, and and when you you fell asleep, like you you look at the the like the day after and it's nightmare. But we solved the bug. I think it was a bug in our codebase. And this was the the the main reason for the spike. And there is some fun stuff, like for instance, when I was doing ablation on Neuron, I was looking at like the Guaednam. And the the Guaednam was doing this like M shape. And I was going a bit crazy because like this is like the the M of Neuron, I was saying to people. So it's always funny like on the Slack that people are, I think that's like one of the the great fingertips that people are are like adding fun by doing those ablation. Yeah.

• 57:17Alessio Fanelli?

  All right, I know we're getting to time and you gotta go back to updating this with all the news from the last one hour that you've been here, just to make sure it's up to date. Any parting thoughts on smaller M, especially? I think there's, you know, obviously, even when GPT OSS came out, a lot of excitement about open models, logo models, blah blah blah. And then I think, you know, maybe the reality is that there's not as much adoption as people would like. Is there anything you wish people would do more of, maybe like, you know, underrated use cases for which you think these smaller models are like just as good as the frontier ones that people are not taking advantage of?

• 57:54Speaker 02

  Yeah, I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. It's like the same interface when I'm like, for example, researching very deep on papers. And when I'm, for example, doing like translation stuff, I want to translate something from English to French. But I know some of the easier users are using this one for translation. There is also obviously all the on-device applications, which are super important. There was this app keynote two hours ago. Well, they have saved the world AI a thousand times probably. And yeah, so obviously on-device applications are super important.

• 58:36Speaker 02

  And yeah, for the future of small L M, I think our goal is basically to build the model that the community don't have and need. So we are super open to feedback. Don't hesitate to tag me on X or any face or whatever. If you have like, if you think that there is like some gap in the, like what the old open source model are today. And you wish there was more, for example, a very small model like under one billion parameters or something like that. Yeah, I think we are pretty open on what to do next. We want to train our M O E because it's fun and everyone is doing that. And also I think there is a lot of different direction.

• 59:19Speaker 02

  And it's always good in terms of science to, because basically the, the, the, the Q and three or even deep seek. They don't realize, you release all the ablation they do. And I know as a researcher that that's something super important. And so when we do model training, we try to be as open as possible. And obviously, it's often appreciated by the community. And yeah, and so that's, that's one thing we want to explore with, with this M O E. And yeah, also our project to do, to do some distributed training across like different data center. Like Prime and selected for example.

• 1:00:01Speaker 02

  And, and yeah, this is, this is a project I'm looking forward as well.

• 1:00:06Alessio Fanelli?

  Awesome. Yeah, I, I do hope that there will be an easier way to just use these model line models through A P I.

• 1:00:12Speaker 02

  Yeah.

• 1:00:13Alessio Fanelli?

  Um, I found that a little clunky right now, just to find something that is token based, because the hugging face A B I charges by the hour, which. Kind of when you run it, it ends up costing the same as some of the other models. Like I have this like a MCP client I built and we use Llama AB for like a conversation naming. You know, I feel like that's like a great use case for like a 500 million parameter model. But there's no simple API to use it. I feel like that's another, in a way, it's almost like it's not economical to serve these models because they're so small. And so I think that's where they get stuck, you know, it's like, who's really serving the 500 million to a billion parameter models, you know? I think that's why hopefully Agameface can do more there too.

• 1:00:56Speaker 02

  Yeah, that's, that's very true. I think we should do some stuff here. Yeah.

• 1:01:01Alessio Fanelli?

  But thanks so much for the time and the masterclass. I don't know, is this file online anywhere? I think this might be the most valuable.

• 1:01:09Speaker 02

  Yeah, I mean, it's not online yet, but I will probably share maybe a more polished version of it. This is literally my draft, so.

• 1:01:20Alessio Fanelli?

  This is probably like the most valuable TL Draw file ever. Like I think this file, this canvas is probably worth millions of dollars of insights from your talk. So excited to see the final product and we'll link it. We'll make sure to release this maybe once the talk is also out so that then people can see the full slides.

• 1:01:40Speaker 01

  Just a parting question because I see that you, you know, you care a lot about training, optimizing and all that. And multi data center and probably like multiple architecture, like, you know, if you've got some AMD, you want to use that. Have you listened to Modular at all? Like, is that interesting to you?

• 1:01:55Speaker 02

  So your podcast with the.

• 1:01:58Speaker 01

  I'm just wondering like, you know, what, what, what's the state of that? If you, if you took a serious look at it, if you haven't, if you haven't, then I'm just checking.

• 1:02:05Speaker 02

  You mean like Modular, the, the language or the just training on different.

• 1:02:10Speaker 01

  Language and runtime, right? Basically like they're, they're basically offering like a superset of Python that should be good for you. You specifically, you are the target audience for this. And if, you know, and I'm just kind of trying to hear about like feedback or like.

• 1:02:25Speaker 02

  Yeah, I'm not very knowledgeable about it, but like the. Like the pre-training codebase is something that you you basically build through training and you iterate on it. So like changing some some like heavy stuff in it are always like a big deal for researcher, I think. Especially that everyone is like their own preference and stuff like that.

• 1:02:50Speaker 01

  Sorry.

• 1:02:51Speaker 02

  But I I didn't really, I'm not really knowledgeable about all of that and I need to watch a podcast with them. It's actually on my on my to watch list.

• 1:03:01Speaker 01

  You know, I I see I see Rust in your codebase, you Python, I see, you know, you're experimenting with different optimizers like, you know, I I'm not like the perfect ICP for them, but like you are. And so I'd be curious to to hear like, you know, someone who doesn't work for them, what's what's your opinion, you know? Cool.

• 1:03:18Speaker 02

  And I never tried, but.

• 1:03:19Speaker 01

  It's hard to try things. Yeah, you have a thing that works. Yeah. Cool. Well, thanks so much for your time.

• 1:03:25Speaker 02

  Yeah, thanks for that for the invitation and see you in Paris.

• 1:03:29Speaker 01

  See you in Paris.

• 1:03:29Speaker 02

  You sure, sure, yeah?

• 1:03:30Speaker 01

  Yeah. Awesome.