[LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka

Swyx joined SAIL https://blog.readsail.com/! Thank you SAIL Media https://substack.com/profile/392441355-sail-media, Prof. Tom Yeh https://substack.com/profile/105138944-prof-tom-yeh, 8Lee https://substack.com/profile/54423-8lee, Hamid Bagheri https://substack.com/profile/85969421-hamid-bagheri, c9n https://substack.com/profile/2411562-c9n, and many others for tuning into SAIL Live #6 with Nathan Lambert https://substack.com/profile/10472909-nathan-lambert and Sebastian Raschka, PhD https://substack.com/profile/27393275-sebastian-raschka-phd. Sharing here for the LS paid subscribers. We covered: This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  Okay, we're live. We have one person. Um, people will start trickling in. Thanks for coming to Sale Live number six. This is a very exciting one. I think we have a, I mean, the topic's always fun with these is whatever is the topic of the day on our little rat raising minds trying to keep up with AI. But we're welcoming the latest writer that is joining the Sale Coalition. So I think this just means more content for Sale. I think I've been a fan of Swyx and a friend for a while at this point. So I'm very happy to have his content join this. And I think you've been doing great stuff recently and continuing to evolve this. So, welcome to the team. I just, this is my friends and colleagues in the AI media space. And it's just great to be able to

• 0:45Speaker 00

  support people and keep that network closer. So.

• 0:48Speaker 01

  Yeah. Thanks for, I just wanted to say, uh, uh, thanks for joining us. It's really a pleasure to have you on here, Sean, or Swyx. Um, so yeah, awesome. I just coincidentally listened to your podcast about the benchmark. Um, so yeah, awesome to, you know, small world, awesome to have you here.

• 1:09Speaker 02

  Yeah, thanks for having me. And yeah, just glad to be on and chat. Uh, I've never ever done one of these Substack Live things, so I'm curious how it works. Cause I always think about Substack because I can use a better platform, uh, but they want to go multimedia.

• 1:25Speaker 00

  I think the live thing before we get to technical content is actually good because it gives it a different edge. It's just like a little bit sharper when you know you're live. I think we've all done a lot of podcasts, even podcasts that are unedited and put those later. But I think the live thing is a different element that can be tapped into nicely. So I don't know, why don't we, why don't we just dive into it? We're going to start with distillation. I put, I put how models cheat in the top so we can talk about benchmarks. I think Anthropic posted this pretty spicy blog post this week. I think it was essentially detailing how they found distributed. Distillation quote unquote attacks on their services from prominent Chinese labs. And I'm very unsurprised with Anthropic calling in an attack.

• 2:08Speaker 00

  I think that that's fits with a lot of their branding. Okay, nice, green share. This is what we mean. Sean Swyx is such a pro. And it's like, and the screen share paper was only dropped a few days ago. But essentially, it's Anthropic is detailing how they found distributed accounts across multiple Chinese labs, building data to their LLMs and described what they were doing. And why Anthropic is concerned about this in their worldview of like AI geopolitics. And I think it's very interesting because I'm of the opinion that the Chinese labs like obviously should do this. They're in a massive GPU shortage and using API's is way easier than generating synthetic data on their own.

• 2:53Speaker 01

  Nathan, if I may interrupt you here, maybe we should just for the general audience, just define distillation before we maybe dive into the details. Yeah, so distillation, that's like a broader concept. It's not like a new concept that came up with LLMs. It's like an older concept in machine learning in general. And distillation essentially is, the idea is that you're taking a larger model and train it on the outputs. Sorry, you have a larger model, let it generate outputs and train a smaller model on these outputs of the larger model. And the idea is that you can train the smaller model more efficiently using that larger model. And originally, I think you just brought up the paper here.

• 3:37Speaker 01

  Originally, what you would do is you would train on the logits. So old school machine learning people might remember from deep neural networks, like the logits, the outputs of the last layer that you usually work with them to compute the loss function across entropy term. And you would train on this signal. And nowadays, in the context of LLMs, it's a bit more loose. So it does not have to be these logits that you train on. It could be just the output data, synthetic data like Nathan just said. So for example, it's It's actually a very common practice. For example, in DeepSeek, um, R1 in the paper, um, or other people do that to other companies.

• 4:17Speaker 01

  They would train the flagship model, the largest model, the R1 model with um, six hundred and seventy one billion parameters. And then they would uh, create smaller variants, like uh, I forgot the numbers, but one, three billion uh, in a smaller range. Like these small uh, models you can run locally and they are trained on the outputs of their own larger models. Um, I think now the thing is also, I mean, this is very common practice. Everyone does that when they are producing the smaller model variants. Um, now I think the, the question or the point Nathan brought up is what happens if you are a company and you generate this synthetic data from another company's L L M and then train your own model on it. So, sorry, that was just like a little interruption, but yeah, distillation in short

• 5:00Speaker 01

  is training a smaller model on the outputs of a larger model, basically, yeah.

• 5:04Speaker 00

  Yeah, and I think this is even possible at the frontier. So like people distill from something like Claude Opus to build Claude Sonnet. Like this is, they're generally doing very similar things internally. They have different access to different tools and like richer tools. And then the other context is that all of these large labs for years have had terms of service where they say that you effectively cannot use the outputs from these APIs to train something like a competitive AI model. It is vague terms. In terms of service or not a contract, essentially in terms of service is something that can be, you essentially are using a service. And then if a, the provider finds you violate it, they can cut off your access. That's just kind of like a basic thing. So

• 5:46Speaker 00

  these have not been enforced within the US much at all. I think there was one case, maybe like ByteDance a year or two ago, that OpenAI cut off their API. But this was discussed so much right after ChatGPT, when people were building the first open models on like Alpaca and things. So it's like, is OpenAI gonna come after us for doing these research models? And it totally died down. People worried about this for like over a year. It was kind of an insurparable discussion. So nothing really happened. And then this is like the first prominent.

• 6:16Speaker 00

  Reemergence of the discussion, kind of to make, I think it's because people are far more worried about AI competitiveness.

• 6:22Speaker 01

  Yeah.

• 6:23Speaker 00

  But I'm curious about you guys' thing.

• 6:25Speaker 01

  Yeah, can we talk a second about even how they would detect, because you said in the beginning something about a distillation attack? And you didn't say that specifically, but you kind of like implicitly put quotation marks on attack. So how would you even detect that? So I think, I mean, distillation in that context means really like literally just letting ChatGPT Claude generate synthetic data and then you collect that synthetic data and train your own model with supervised learning, supervised fine tuning on it. But then how would you even detect that this is a distillation attack versus just an evaluation? Because right now I'm actually running, I mean, I'm distilling myself for chapter eight of my book.

• 7:07Speaker 01

  But I'm doing it with open weight models, so no worry, Anthropic, please don't worry about it.

• 7:12Speaker 00

  I distill from API models for my job.

• 7:16Speaker 01

  Yeah, I use OpenRouter right now and just still from the DeepSeek version 3.2 model, which I think these folks are okay with that.

• 7:24Speaker 01

  But what I wanted to say is, so when I'm evaluating models, I use basically almost the same script. So when you're evaluating in a model, you have a question and you let the model generate the answer, right? So you generate the response to your benchmark question. And in my benchmarks, I have data sets from math 500, 500 examples. I have a bigger math data set of 12,000 examples. So you're basically just running an API in a loop to let it generate these questions and, and sorry, the answers. But then how would a company know, okay, this person is just evaluating versus this person is now

• 8:04Speaker 01

  saving that data and then later training their own model? Like, you see what I'm saying? Like, it's the same, the same process.

• 8:10Speaker 00

  Thing. So like, when you're evaluating, at least the basic evals, or you're gonna do it once and not do it. There's some amount where you are, I mean, like they say stuff here, but there's also more of it where you just start.

• 8:24Speaker 00

  Not like they're not gonna, I think most of it is quantity. And then they're gonna look at patterns across similar accounts is what they're saying.

• 8:30Speaker 01

  Yeah, exactly.

• 8:31Speaker 00

  I think they're gonna see like really repetitive stuff.

• 8:34Speaker 01

  Yes, so I think the interesting point this leads to is like, um, I mean, you can do evaluation at a large scale. If you are a big company, you wanna know whether your L M performs very well. You have a large suite of um benchmarks you are gonna run. And but then you said like maybe looking for patterns. Um, so you would maybe one way would be, okay, um, this is a familiar question. It comes up in the benchmarks. So this person is maybe not stealing our answers. It's just using it for benchmark purposes. But then it means kind of like that they are looking at what you're generating there, you know, which is, I mean, of course, nothing is private when you are using L L Ms on the internet. The data is somewhere, you know, intermittently stored.

• 9:16Speaker 01

  But then it kind of like almost implies that they are checking what you use the L L M for, what you generate, which is kind of like a sensitive topic, almost like privacy wise, right? So that's kind of like an interesting point because, I mean, of course, you mentioned the terms of service that you are not allowed to distill. But you're not distilling, so the point I'm trying to make is you're not distilling life when you are on the platform. You are doing it somewhere later. You're just letting the L L M generate answers. And I find it kind of interesting that a company would look at that, uh, even like at the scale and, you know, call you out like, hey, you're you're generating too many answers here. That's not cool or something, you know, that's kind of a weird thing.

• 9:56Speaker 02

  Yeah, I I wanted to respond a couple, uh, this is like a few sentences back, but, uh, actually, uh, Anthropic has blocked U S companies first before the Chinese companies. Um, he has blocked both OpenAI and X C I from uh, from from using the models. And I think maybe explicitly accused X C I of distilling stuff. I don't I don't know, but definitely not like in a in a full block course like this. So this one is like definitely the most high profile. Case. Um. And yeah, and like I, I, I do think like it is actually pretty hard to distinguish from like, hey, I'm just running my internal benchmark, man. And of course, it's going to be very high volume of like all of the same stuff. Uh, because you know, especially like some benchmarks,

• 10:38Speaker 02

  you have to run like three, four, uh, three to five times. Like this is the exact same questions, right? Like, um, I, I do think like, obviously, if you, if you get to the hun- to the, the tens of thousands, hundreds of thousands, then you're like, you're okay, you're not just running benchmarks. Like you, you are, um, distilling this, uh, this thing.

• 10:57Speaker 01

  There is a good point in the chat, um, like how would the distribution of questions look like? Um, if you're distilling, and I think, uh, related to your point, um, at a certain point, when you have a certain magnitude of answers generated, it might look suspicious, but I mean, there are a lot of legit use cases. If a company uses your, let's say, open AI Claude API as their own chatbot and they have a lot of customers, it's naturally a lot of answers that are generated. And so, um, they would probably look at distributions like, um, maybe, uh, you would expect a very broad distribution when you are on distilling because you want to cover pretty much everything. And when you are running benchmarks, it's maybe more

• 11:38Speaker 01

  specific. Uh, you're running a math benchmark, it's just math, or if you have a customer chatbot, it's more like customer answers. But yeah, I think they would maybe analyze your distribution. I feel like this is, um, kind of a weird thing to do. I don't know if you're a company and you're looking into your customer privacy, like data generated, you know, like, of course, it's, well, you have to expect that it's not private, but still kind of like a weird thing that they, that they do that essentially, yeah.

• 12:07Speaker 02

  Um, okay. What else do we have to talk about?

• 12:12Speaker 02

  I, I think, is it, is it interesting? Okay. Um, I did, I did, uh, okay. So one thing, uh, this is a little bit of Substack, like, you know, authors back and forth. Uh, one thing I did was I threw it into Nano Banana, which is like, uh, like it's kind of like a decent visual, right?

• 12:26Speaker 00

  Throw it into Nano Banana two. It's a Nano Banana two live pod. Like it just released ten minute, five minutes ago.

• 12:33Speaker 02

  I, I had, uh, this is actually Nano Banana two. I, I, uh, so because I'm in an early access program, they cut you over to Nano, to, to new Nano Banana, and I couldn't access the old one. So I was like, I was trying to like, do like a diff. I couldn't do it because.

• 12:47Speaker 00

  Classic. That is classic early tester program shit. Look at the pain we have to deal with here.

• 12:53Speaker 02

  Is it interesting that you seek use so much less than Minimax? I think Nathan, in your write-up, you had a little bit of a comment about like.

• 13:01Speaker 00

  This is a political blog post in a way, maybe not political, but they're trying to make a point that is more about making a point than the details. Like, the DeepSeek thing is definitely way smaller scale. So, okay. Most of the labs will experiment with all the APIs they can get access to. Like, data is just so important. And you're gonna have a pipeline where you could sub in any API and then run an ablation to see if it gives you performance. The API is kind of free, like, just do it.

• 13:31Speaker 00

  And the millions of exchanges is a bit more of a bet. You can, you can measure that a bit longer and it takes a lot longer to get.

• 13:39Speaker 00

  The millions of exchanges is like tens of billions or hundred billion tokens. And it takes a lot longer to actually get that out of the API, especially when they have to spread it across a ton of accounts. These accounts are all rate limited and have other problems like. That takes longer, but this tiny one is so fast, so. I don't, like, I, that was generally my point, that it made it clear that Anthropic's trying to use the DeepSeek name as the only Chinese AI name that people in the US know.

• 14:04Speaker 01

  Like, marketing-wise, like, to make it, you know, stick or to, yeah. Actually, you mentioned also like the, the different APIs and everything. I'm not like sponsored by them, or I have no affiliation, I've never talked to anyone from that company, but OpenRouter, for example, is a good example where I, I've been using it a lot for the open weight models because for the bigger ones, they're too big to run them locally. And what's nice is they do also offer, so it's basically just routing you through other companies' APIs and they select automatically at that point, what is the cheapest one at that point? I sometimes get some failures, I think when it switches, sometimes it crashes, but like in my script, maybe it's like something I have to fix there, but. So even then, if you're distilling, you can do that from multiple providers.

• 14:47Speaker 01

  But yeah, of course, if you are wanting something from ChatGPT or Claude, it's always gonna go through the official one and then it gets, I guess, suspicious. But you could also technically distill a bit from open, through OpenRouter, through, I mean, through their account, your direct account, you can make multiple accounts. And it's kind of interesting that they, that they track all that. And then like, yeah, different topic now that you called out, that they call out DeepSeek, which is quite interesting, yeah.

• 15:18Speaker 02

  For what it's open router seems to not be using deep seek in most of these.

• 15:24Speaker 00

  These are free models. Deep seek is not.

• 15:25Speaker 01

  Yeah, I see. Yeah, I mean, I'm using the paid API. I should also say I uh it was also it's also nice. They show you how much it costs and the tokens per second for different providers. So if you go to the search in the top, you can go to the different deep sequence. I just like it because I do a lot of um model comparisons. And then uh this one is an older model, so maybe it only has one provider. Um but if you go to, I think, deep seek R1 or something, or even the normal 3.2, there should be multiple providers that uh if you scroll down, yeah, you can see see there are different um providers and uh different tokens per seconds, different costs.

• 16:04Speaker 01

  Um so it's kind of like a I just like that website because it's just quick to use the API and they have a uh open AI um like API. So it's almost like it's not sponsored or something. I just find it generally useful. So but uh yeah. Just a side note.

• 16:21Speaker 00

  Do you want to go back to the comparison? Did you have a high level point to make there?

• 16:25Speaker 02

  Oh, um, okay. Uh, just a just a couple. Um, one, I think I think the timing post moonshot releasing their stuff, post Minimax releasing their stuff, but pre DCV4, I think that was strategic. Um, I think that may also have factored into why Minimax was more detected, like had a higher number. Uh, so like, you know, like when you collect data is actually very important, right? And so they interrupted or they found Minimax during the training of Minimax 2.5, right? Which I mean, we we will confirm this later on if if we if we do end up doing a call with with them. Um, and so obviously, like, the number is going to be very high because they they're like actively looking for it. And then they

• 17:10Speaker 02

  they banned uh the Minimax accounts and Minimax changed uh their their things. Actually, I don't think that's exactly what happened. Sorry. Uh, let me correct myself. Um, while Minimax is distilling, they release OPUS 4.6. Uh, and they said. Uh, they said that they redirected nearly half the traffic. So I'm like, this is like, okay, very, very clearly like this is them, right? It's the same exact traffic, uh, switched to a new model. The moment a new model releases. Okay, cool. Uh, DeepSeek maybe wasn't doing that because they, they hadn't been, uh, working on, on their stuff actively. I don't know, right? Like it could, it could be a different thing. Or DeepSeek is just way more efficient. Like I, I got all I need for a hundred fifty K. You guys, you know, are so inefficient.

• 17:47Speaker 00

  It'd be so interesting if we knew the timeframe of this. Like are all of these API requests within the last four weeks? Are they within the last six months? Like, that's such a different nature of what is going on.

• 17:58Speaker 02

  Exactly, right? That's what I'm saying. Like DeepSeek was training two point one, two point two, like, you know, a year ago, like.

• 18:04Speaker 00

  Yeah, or like, I don't know, DeepSeek OCR, they were like, I guess they said what it is, but it's not that.

• 18:09Speaker 02

  Yeah, yeah.

• 18:10Speaker 01

  Uh, but like also scale wise, I do think, uh, yeah, Minimax is, uh, three times smaller. It's, it's just like a faster model. They don't use, uh, MLA and they don't use the DeepSeek sparse attention, but it is, I mean, just, I think it's just group query attention, but it is still a pretty snappy model. So it's, I, I think just attractive maybe to, um, to use it. And, uh, the other one, top of my head, I don't know. Maybe they had like some free, free tier or something like that. Um, where I think when the models come out, they sometimes offer a free usage. And that was a more recent model than, uh, I think DeepSeek, the last one was from December, the V three point two.

• 18:46Speaker 02

  Yeah, yeah. So, uh, you know, may, maybe this is an irrelevant point because they, they were training before and, uh, before would have the same amount of traffic, or they're just way more efficient, right? Uh, it does bring to mind like.

• 18:57Speaker 00

  Efficiency thing is not it. I can guarantee it. Like that is not. Yeah, it's a small chance that they, it's like there's a chance that they got the right research idea early and like found the right data to use. But it's not that they're like gonna be ten, three X more efficient.

• 19:13Speaker 02

  Okay. So it's like, you know, it's a timing thing, or they just actually don't use it that much. Um, I, I mean, like you, you play this out like, uh, I was like, okay, why, why don't they share? Right? Like they're all buddies. All right. Like what? And like, you know, it, it, it, it does come to a point where like, okay, let's have all of China just like distribute it to like every.

• 19:34Speaker 00

  I can talk about this a little bit that we're doing. There's a lot of, not a lot of research, but there's a few research projects trying to understand like, how do you use? Distillation data. I think SFT is the cleanest example where you're doing like the, you're doing this auto aggressive loss on Q and A pairs. But the strongest model is not necessarily the best teacher and most of us in this area think it's due to like some. You have to match the probabilities of the tokens to the base model. So like. What's happening is that Quen dense models are the best teachers for a lot of open weight models. And I think that's because a lot of open weight models are either Quen or. Have been like Quen like for a while. So like OMO learned really well from Quen and obviously like other Quen models did.

• 20:16Speaker 00

  But like scaling these pipelines up to use say GLM four point seven or a bigger deep seek model or. A more recent big Quen MOE, like all of these, it's a lot harder to just generate the data from the same prompts. With like the right sampling settings and then do SFT on them and actually make the numbers go up. Interestingly, GPT OSS is a pretty good teacher, but there's like a huge gap there where it's like, just because you have this data. Does not mean it's actually going to make your model better. So you have to do the research to be like, oh, we learned that. We get signal out of Claude, we need to get a hundred billion tokens ASAP because it's going to just immediately make our model better. Like, that's not a commonplace to be in, in modeling because this like weird teacher student dynamic going on.

• 21:01Speaker 00

  So I can see that being different across labs.

• 21:04Speaker 01

  I can, uh, yeah, I, I think also it has something to do. I mean, I noticed also if you are distilling, um. The smaller model from the same model family performs better. And I think it's to your point. That, um, if you have a very, very strong model, um, it might be also too different. Or like if the style is too different and then it's too much of a leap for your model to adapt. Like it's too, too different from the Q and A answers during the pre-training or.

• 21:30Speaker 01

  And so you make a bigger leap. And another thing I wanted to say about, you mentioned OMO and I, it's been a while since I read the paper, but you might know way better than I do, but I think you did also train on the logits. And, um.

• 21:43Speaker 02

  We didn't do technical distillation. We just took the tokens.

• 21:48Speaker 01

  Oh, I see, I see, I see. Okay, then it was probably a different paper. I think Google does that for the Gemma models.

• 21:52Speaker 02

  Yeah.

• 21:53Speaker 01

  Because here, there's also then the distinction, because you mentioned Quen and other models. You can only do that for open weight models, because if you do that for Claude or OpenAI, that would not work with the Logits, because they don't provide them. They only provide them for some tokens, like a hundred or a thousand top tokens. And so it is, in a sense, if you want to do the real, in quotation, market distillation, it is kind of like even easier to do that from open weight models, because you can control it. But then also, like you said, well, we need a hundred billion tokens ASAP. That is not an easy thing to do, because even like, I mean, it's like forty tokens per second or something for these large end models when you generate answers. And getting that million, billions tokens, it takes time, right?

• 22:34Speaker 01

  So it's almost like easier to start distilling from a medium model. You just get, so it's like the question, more data versus more high-quality data, right? So it's also like a sweet spot to, like an experiment itself in an evaluation study, right?

• 22:52Speaker 02

  Yeah, I like that Nathan had to call it technical distillation, because it is no longer the default, even though it was the first.

• 22:57Speaker 01

  Yeah.

• 22:59Speaker 02

  Also, I'll note a fun fact. I did my Jeff Dean interview recently, and I tried to get out of him, but he like sort of dodged it a little bit, that, you know, you remember like, there were actually three sizes of Gemini models. There was Nano, Pro, and Ultra. And I was like, where is Ultra? They keep it in a basement and they distill from it, right?

• 23:23Speaker 01

  Interesting, yeah, yeah. Maybe also, is it like to safeguard yourself, so no one can make a call?

• 23:29Speaker 02

  Yeah.

• 23:30Speaker 01

  Or the price also, but probably both.

• 23:32Speaker 02

  Yeah, I mean, and I think like, this is how like, I always think of like, the model you deploy is never the model you train. Because you train the dense, and then you deploy the M O E, right? Like, you basically always do it, like at every lab.

• 23:47Speaker 00

  Say more. Like, do you think they're really distilling from dense models?

• 23:50Speaker 02

  I mean, like, I think that, like, that is like the full, like, when, like, just unlimited resources, don't care about inference, just care about maxing intelligence? Why not?

• 24:02Speaker 00

  I'm not a hundred percent sure. I think the MOEs just give you a flop. Like, I don't know if that's actually how I think of gains of MOE when you have really good MOE architecture. But I do think that they have bigger models that they distill from. And they train internal models different than external because the external models have been getting a lot smaller. Which is the kind of weird thing. We don't have a good way to measure it. Maybe Dylan will backwards figure it out and inference max whatever the heck. They'll deal with a blue model set.

• 24:32Speaker 01

  But I'm always suspicious with these things. Also, it's really like a capacity thing too. Uh, how many people use the model at the same time? Uh, hardware, how much is allocated? And it's always, it's like, yeah, maybe a rule of thumb, but yeah, it's really tricky. I think it's really hard to say anything from these numbers.

• 24:49Speaker 00

  I do think that they might start restricting models to only be in products and not being in API. I think the whole API business is brutally competitive and I don't have a good sense for what the defensibility of it is. I think like it makes sense for something like Google and Azure and any existing cloud businesses to have APIs. And that's kind of a more natural transition. But like the Anthropic and OpenAI API, like the transition from their products, which are their big differentiation, whether it's ChatGPT and Claude Code and Codex. The different, like you don't get people to go use the API from that. And I think you get a lot of people that are already spending on clouds that then go to use the APIs, which is why like Lambda and Nebius are gonna have these API products.

• 25:32Speaker 00

  But like, isn't it, if Claude's really worried about distillation, like they should put the model release in Claude Code ASAP and then just not bother with the API. I don't know when that'll happen, but it could.

• 25:44Speaker 01

  I do think though, it's a big customer base, the API customer base. Any, any type of product that is built on, uh, I mean, um, with LLMs like customer chatbot types of things, but also more generally, I do think the problem like with, I don't know exactly how the plants work in Claude, but you would reach a token max where you can only get so much with your subscription. You can, I think, buy more tokens, but I think it's just easier with the API at a certain scale. Also like the whole um open claw customer base, right? Because they don't allow the plan anymore in the open claw context. So you have to use the API. And I do think, given how many tokens open claw generates, it's actually not a bad business if you don't lose money on these, you know, on these tokens.

• 26:28Speaker 01

  If you sell it at a not subsidized price, I do think the API is actually not a bad business model.

• 26:37Speaker 02

  Yeah.

• 26:37Speaker 00

  Uh, I take a side. Do you want to try a tie break? I'm obviously being provocative. Like, I don't really know, but I can see it. Like Anthropic gives Apple vibes to me.

• 26:45Speaker 02

  Hmm. Uh, I mean, like Anthropic has a higher chance of doing this, yes. Uh, OpenAI, just because I like have talked to the people so much. Like, I I just don't super believe that they will have locked models um to to to products. Um, only only out of, I guess, idealism and some principles rather than economic incentive. Economic incentive would agree with you that they should have uh private models to to products. And like recently, they've done this, right? The last three GPT fives all had codex variants that were two to four weeks ahead uh inside of, released only inside of codex rather than as an API. So they're starting to get there.

• 27:30Speaker 02

  But uh just like constitutionally, I don't think the the people that run these things believe in like locking things behind APIs because they have such a huge market anyway. So they they like kind of don't care. And then they also like, if you're genuinely like sort of zealot, like if you're not trying to maximize the value of your company and genuinely just trying to spread AGI everywhere, uh then then you release the API because you just don't know what people are going to build with it.

• 27:56Speaker 01

  Hmm. Uh, one more thing, though, with the codex uh thing. I we will have to see, I think, next time because I think this time it might also be a bit biased towards releasing it in a codex because they almost released it simultaneously with their app that they want to promote at the moment. So it could have been like um more like they did that so that anyone checks out the app. Um, and but we'll.

• 28:17Speaker 02

  Yeah, yeah, but there's always a two to four week exclusive window. Yeah, yeah, and uh, you know, that's their right.

• 28:25Speaker 01

  Yeah, sure.

• 28:26Speaker 02

  You want to put it, want to promote codex, that's pretty effective.

• 28:28Speaker 01

  Yep.

• 28:29Speaker 02

  Um, we have a bunch of questions in the chat for like other.

• 28:32Speaker 01

  Yeah, that's.

• 28:32Speaker 02

  Do we want to cover benchmarks and then this thing go?

• 28:36Speaker 01

  Go right ahead.

• 28:38Speaker 02

  What do you want to do? It's your sub stack, I don't know.

• 28:41Speaker 00

  Oh man, it's a collective. You should just dive into what you're interested in.

• 28:46Speaker 02

  I mean, Sebastian was interested in the Sweet Bench stuff. So like, you know, this past week Sweet Bench verified died. Or like officially.

• 28:55Speaker 00

  What do you mean by this?

• 28:57Speaker 01

  Yeah, let's define Sweet Bench first, maybe.

• 29:00Speaker 02

  Okay, you know, I happen to have the post on this. So let me just remember.

• 29:06Speaker 01

  So the broader topic, the umbrella topic here is how do we compare which LLM is currently the best LLM? Like, one of the ways would be Sweet Bench, basically. But then, yeah, I will maybe let you explain because you had this brilliant podcast or article.

• 29:23Speaker 02

  I mean, okay, where do you want me to start? You want me to just define Sweet Bench, I guess?

• 29:28Speaker 01

  I guess, yeah, so maybe going from, so basically that it is a coding benchmark. And then that Sweet Bench is like a popular way to compare capabilities of LLMs. And then there is Sweet Bench verified. But maybe, yeah, we should talk more about Sweet Bench first.

• 29:47Speaker 02

  So Sweet Bench was a paper out of Princeton from Ophiopressus Group. And they do a lot of good like code benchmarking work. And so, and it happened to be that they just kind of drew thousands of example sort of open source issues and PRs that closed those issues from open source. There's a bit of selection bias here because they only focus on popular open source and only a small number of popular open source, but a large number of issues from those open source. And then they just kind of dredged up some passing tests and then some failing tests that you need to make pass in order to pass the score. When it, when it launched, it was kind of obscure.

• 30:30Speaker 02

  Devin actually was the first one to pick, choose it as a benchmark to report. And then it went from like, I think at launch it was like 13% and now everyone's at 80%, something like that.

• 30:41Speaker 02

  Sweet Bench, because it was, it was done on like a student budget, was very kind of, let's call it sloppy or whatever.

• 30:48Speaker 00

  Terminal Bench is like this now too, like they're, they're just aggregated. It's like hard to do a benchmark that is well calibrated across topics and different.

• 30:56Speaker 02

  Yeah, yeah, it is hard, it is hard. So, you know, for the, for the small group that is watching, I'm actually working on it with Cognition to launch a new benchmark here. But yeah, so OpenAI was like, okay, guys, we're like, Sweetbench is taking off. We're gonna adopt this, but we refuse to abide by like the full Sweetbench. We were just gonna like actually go and go and curate like 500 subsets of the original Sweetbench. And they actually hired humans to like go and vet through. Like, I think the, it's somewhere inside of this blog post, but basically they hired like three humans for every task to just vet whether the task was like high quality or not, because there's a lot of slop in there. Anyway, like, okay, this is the 500 that we, that we're gonna endorse.

• 31:41Speaker 01

  It's like a curated subset of Sweetbench where 500, let's say, challenging problems that are supposedly well-defined.

• 31:49Speaker 02

  Yeah, yeah. And, and what's, what's really funny is that at launch, so this was launched in 2024. At launch, OpenAI could not run all of its own 500. So for a while, like, there was like a few releases on OpenAI that reported on a subset of the subset, because they could run it on their eval infrastructure.

• 32:10Speaker 02

  So like their numbers were higher because their denominator was lower, which is, which is very funny. Anyway.

• 32:15Speaker 01

  Maybe in that context, we should say what Sweetbench kind of looks like. I think it's like basically like a code that has bugs in it. And usually the task for the LLM is to fix the bug in the code, right?

• 32:28Speaker 02

  It's right here. The whole thing's, the whole thing's open, which becomes a problem in the future. But right now, you can see the whole thing, right? You can see the, the repos from the issue ID and the problem statements. And then you have also the, the test that you're supposed to pass and fail. So it's all here on Hugging Face. And you can see that's a 500. Anyway, I think we don't have, we don't want to get too lost in the details on sort of.

• 32:53Speaker 01

  You just wanted to say, or like define the context like that this is a coding benchmark essentially 500 examples that are available on the internet.

• 33:02Speaker 02

  Okay, and then if you want more, a bit more historical context, this is like a step up from human eval, which is more on completions, right? This is, this was in my mind, the first proper agentic benchmark, I guess, apart from Talbench, where they give you the problem and the end result and they don't really specify how you're supposed to get there. Whereas I think a lot of like previous benchmarks like MMLUs of the world and human evals, which is in the coding domain, also released by OpenAI, was very much, here's like the problem statement and then give me the right answer immediately after without that much sort of extra files or anything that you're supposed to run. So it, like, the other ones are more autocomplete, this one is more agentic.

• 33:46Speaker 02

  It's all a spectrum, obviously, because you can use agents to solve autocomplete, but yeah, that's not what human eval was testing. Anyway, I, I just, I wanted to make sure, like, people understand that OpenAI actually put, invested a lot of money and effort into making Sebenj verified from.

• 34:01Speaker 00

  Question, how much money do you think this costs?

• 34:04Speaker 02

  Oh my god, don't, don't do this.

• 34:06Speaker 00

  Millions? I would guess order of a couple, like, it could be a few million, but probably.

• 34:09Speaker 02

  I'd say, yeah, I'd say a couple million. I, I, you know, so basically you do like, okay, what's the first filter pass and then like, okay, it's 500 times three because they had three.

• 34:19Speaker 01

  People per, yeah.

• 34:21Speaker 02

  Yeah, three people per thing and then maybe like a couple more sort of verification passes or whatever, right? So like, yeah.

• 34:28Speaker 02

  So then they were like, oh, so this year, they're like, oh, well, not only is it saturated because like progress, everyone just takes turns to increment by 0.1 every time they release a new model. It's like, it's bullshit. It's obviously bullshit. Like the inherent noise in just running these models varies by like 0.5 to like 1 every time you run it, like, you just choose the highest.

• 34:50Speaker 01

  A little nitpick, I don't think it can be 0.1% because, like what you said before, because it's 500 examples. I think the smallest increment is. Point two percent if I.

• 35:01Speaker 02

  Okay.

• 35:01Speaker 01

  They might average. Because like, uh, but like little detail. Yeah, sorry.

• 35:05Speaker 02

  I, I, I think in so, uh, as we progress to the next, uh, era of benchmarking, um, the N, so this, the N here is five hundred, right? The N, uh, doesn't directly correlate to the percentage points because you get sub points as well.

• 35:19Speaker 01

  Ah, yeah, good point, good point.

• 35:21Speaker 02

  So, so it's like terminal bench, even though it has ninety something tasks, like you, you, you can get subdivisions, uh, less than one percent. Um, anyway, so, uh, not only do they, do they have this, they, they actually audited their own, like, they're like, okay, like, how come everyone is saturating at eighty percent? Like, what, what's, what's up with the remaining twenty percent? How come everyone's like failing at it? And they were like, oh, actually, we looked, we, we paid even more people, six people per task now, with an extra team if, if any sort of, uh, positive identification is found. Um, and we were like, fifty nine percent of them cannot even be solved at all because the original benchmark was, was like, still slop, like stuff, stuff got through that was not solvable. And I, I actually tried to like illustrate this in, in my post. Um, so here, this, this is an impossible test, right?

• 36:06Speaker 02

  Okay, so here's an example, and this is, this is the sort of value add I did on top of the, the original post. Here, here's an example of a Sweetbench verified task that passed the first round of human verification, right? So here's the task, like, and we want to implement Python type hints or something. We want to see, expect to be here, I want to see a string in the output, right? So, um, if you were given this, you would, you would never pass this because the test said, I am looking for something called get annotation. And if you don't give me this magic string get annotation, you will fail this task.

• 36:38Speaker 01

  Why? So, yeah, it's way too specific. Yeah, yeah, it's like, uh, yeah.

• 36:45Speaker 02

  Right? So, so, so this is just a bad, uh, ta- task that somehow escaped, uh, their validation.

• 36:51Speaker 01

  So the only way you could kind of solve it is if you're memorizing the.

• 36:55Speaker 02

  Yeah, exactly, exactly. Which is actually a nice, like, I think every benchmark should include stuff like this. Where if like

• 37:02Speaker 01

  Like a honey pot.

• 37:03Speaker 02

  If you solve this, you're like, oh shit, that is a canary, right? Like, it's like, oh, like, I mean, you're definitely cheating.

• 37:08Speaker 01

  Like a sanity check, yeah, yeah, yeah.

• 37:10Speaker 02

  Yeah, yeah.

• 37:10Speaker 01

  That makes a really nice point, yeah.

• 37:12Speaker 02

  Yeah, so, so like, I, I, I just think like, it, it, to me, it's a beautiful point of like, how hard it is to make evals. That there was these like multiple rounds, there was original C bench, which like, the, the, the Princeton kids did do initial first pass. Then there's a second pass of opening I doing C bench verified. And then, and then like every single person that ran for the next, C bench verified for the next one point five years did not call this out.

• 37:38Speaker 02

  Until opening I was like, hey, let's, let's like, look at the data. So I think it's like really interesting. The, uh, while they were looking at this, they had a second thing, uh, that they, they, they, they looked at the chain of thought. And inside the chain of thought, they found G P T five's own chain of thought to start including, uh, information from the future. Right, where, because it was trained on, uh, because the problem is open source and because it was trained on information from GitHub. It would, it would like use advanced knowledge of future versions of the Django version that they were using, uh, to solve the, to solve the problem. Like, they, they knew.

• 38:14Speaker 01

  Seen stuff like this in the real world, where the models will hallucinate the new version of the A P I, even if your script isn't on it. Like, I think a lot of the hugging face stuff is like the worst with this, where like the models just are totally googably glopped, like. They've seen all the versions and the A P I has changed too much over time, where they fucking throw something out there.

• 38:34Speaker 02

  Yeah, so, so, um, the most egregious, yeah, I mean, I, I, I think like, you know, there's, there's a lot of this, right? Like, the sort of ethical behavior, like, okay, so you can blame things like, oh, you should not have released, um, this, the full data set in public. Because obviously people can train a full data set. But like, it's not like the researchers are trying to do this, like, because these things are also open source, like, it is like any, any data set that touches GitHub. Any training corpus that touches GitHub is going to just eventually absorb this. And then.

• 39:02Speaker 01

  Yeah, yeah, and it's not even this website or the repository directly, it's a clone of this repository or someone else who has that. Develops their own open source library and has that in the unit tests or something where it's not even intentional or malicious or anything. It's like by accident, you already absorbed that, yeah.

• 39:18Speaker 02

  Yeah, or, or a new feature that releases this edit only feature, it gets written up in a blog post or a conference talk or something. And then it just makes it in, right? Like, it's, it's really funny. Um, okay, so to me, like, OpenAI could have stopped there and said, okay, we're done. They did one more extra thing, which is kind of funny. They also then ran Flash, Gemini and Opus. And this one, it was more, it was like even more egregious. Okay. They just gave the task ID and just said, repeat this revenge task to me. And so from task ID, they can just vomit out the whole statement and the solution.

• 39:55Speaker 00

  These are crazy. The stuff that's in these models, when you zoom in deep, is really, really incredible. Because like, these are models that are like really, really well done, but there's just so much complexity in all the pieces of the pudding that get put in the recipe. There's just so many weird.

• 40:12Speaker 01

  I also still find it fascinating that, I mean, of course, it's like kind of by design when you're training that you memorize things, because that's literally like next token prediction. But given that how big a model is and how much data it sees, and usually it sees only the data once, that it is still has enough capacity to memorize, you know, like it's kind of like, so usually I would think, okay, I would have to train multiple epochs to be able to memorize. But no, it is enough maybe to include it once or twice in the training corpus. And it can do a perfect rendition or perfect recap of what it is in there, which is kind of fascinating. Even if people don't want that, it's, you know, it's crazy.

• 40:57Speaker 00

  Yeah. Wow, he's got good at this. There's essentially like a duplication level that you need at each stage of training, and it's not easy to measure. So like, if you do too much at pre-training, your model forgets basic facts. And at post-training, it's probably closer to these abilities. And I think that that is a thing that is not well reflected in a, like, you could see it in a vials of your knowledge tank. And I think this is like an art that they have probably gotten good at.

• 41:24Speaker 01

  Yeah, like continued pre-training does also require some revisiting of old data. Otherwise, like you said, you have the forgetting. But it's still fascinating to me that with such a small fraction usually, because you usually use one or two, five percent for like continued pre-training, that it's enough to have the model memorize almost everything, which is fascinating. Yeah. I don't know, it's just like a, still after all these years, fascinating, yeah.

• 41:51Speaker 02

  I think there's so one of the pet topics that I pursue like two, three times a year on my stuff is the information theory of L L M's. And I I still think it's like super understudied, like how come you can memorize from one pass?

• 42:07Speaker 01

  Yeah, exactly, right.

• 42:09Speaker 02

  And then and then also like people forget like uh superposition, um which is like Anthropic's uh original mech interp work. Also basically stuffs information inside the smaller bits that then get forgotten. But like, how how does supervision actually work? I the people I don't think I've seen a convincing study on on that.

• 42:32Speaker 02

  Okay, anyway, anyway, I don't know, I I I'm done on my on my sweet bench right now. I don't know if you have thoughts or questions or whatever. But I I do think like uh this is an example of like, yeah, the models unintentionally cheated and and benchmarks are hard to make. Uh and we need new ones and uh you know, if this happens to sweet bench verified, like which I think is the most scrutinized benchmark in the world.

• 42:53Speaker 01

  I in my recent post, uh I had like a bar plot where I showed the sweet bench verified numbers for most models. And like you said, they were all eighty something percent, like but literally eighty point between one and nine, let's say. Where there's almost zero variation, even like something like minimax M uh two point five, which I do think is worse than um G P T five point two. Like, no offense, it's a smaller model, it's a cheaper model. I I don't, I for my usage, by using open runner, it's a little bit worse. But on this particular benchmark, it's the same. It's like this is like, I I don't think what I'm saying is that. M two five should get less score on sweet bench, but I think other models should get get more score.

• 43:37Speaker 01

  But like you said, the problems are just impossible to solve. But one point I think we didn't bring up is we said that sweet bench verified has issues. So what do we do about it? I think there is like a sweet bench pro now, which is kind of like a.

• 43:54Speaker 01

  I would say like verified, try to fix the regular Sweet Bench and Pro tries to fix or verify. But I haven't looked into this. Is it like another subset or is it a completely different set of problems?

• 44:06Speaker 02

  Yeah, it's a new set. Uh, so the, you know, Sweet Bench draws from like a twenty two, twenty twenty two ish, twenty twenty three ish era of problems. So all you do, you, you, there's a few things you do, right? One, you do private, private-public splits, right? That's super obvious. Two, you update the dates that which you draw from. And then three, you diversify the repos and the languages, right? So the, these are all just like very, very super basic fixes. Uh, and then obviously try and fix the testing. Um, super basic fixes to the original Sweet Bench, uh, which it doesn't take a genius to, to figure out, but they did the hard work and.

• 44:43Speaker 01

  But it is in a sense also what verified meant to do. So it's not, um, let's say people looked at this again, but it's no guarantee that it doesn't also still have issues that might be discovered later on, right? I mean, it's.

• 44:55Speaker 02

  No, so, so Sweet Bench Verified, uh, was, was an intentional subset, right? This, these guys were like, no, no, no, we, we need to, we need to have a superset. Uh, we, not even superset, we need to.

• 45:05Speaker 01

  Yeah, different. Yeah, yeah. But like what I was trying to say is when, uh, Sweet Bench Verified was developed, there were three, uh, people per task, making sure the task is well-defined and everything. But then two years later, it turns out, no, no, this was not the case for everything. And what I'm trying to say is it could be that Sweet Bench Pro is better, but it might still have issues that might not be obvious right now, but maybe in one to two years when we revisit this, uh, and you see some of the failure cases, maybe we'll discover, okay, this has still some issues. So it's not the guaranteed perfect set is what I'm saying. I don't know, but it's just like a suspicion here.

• 45:40Speaker 02

  Uh, totally, totally. Um, I, you know, I, I do think C, uh, Scale AI has a professional interest in making sure this is.

• 45:47Speaker 01

  Yeah, no, no. But what I was trying to say is, uh, Sweet Bench Verified also had a professional interest to make sure. Oh, very different incentive. I guess they all have very different incentives.

• 45:56Speaker 02

  This one has limited budget. This one has basically unlimited budget because it's like literally existential to Scale AI that they have good data.

• 46:04Speaker 02

  But I also, I also think it's really nice that this, this team, the evals team at OpenAI keeps endorsing Opus.

• 46:13Speaker 02

  It's kind of funny.

• 46:15Speaker 02

  So, so yeah, they deprecate Cbench verified and then they were like, we're going to report Cbench Pro now. And GPT-5 is like, you know, number one.

• 46:24Speaker 01

  Maybe if, do you know if I would want to evaluate on the private dataset, how would I do that? Do I provide the API to, is like an API call I have to do against Scale AI or?

• 46:36Speaker 02

  I don't know.

• 46:37Speaker 00

  My API key and agree to not, you have to like agree because if you don't have an agreement, then you can just have to keep the data. You have to like do special hoops to make sure that you don't steal the private eval.

• 46:50Speaker 01

  Yeah, my question was basically, do they even let you download the data? Or is it more like you send the answer to them and they do the evaluation on their backend? So that you don't even get to download the data, you know, like. Otherwise, like you said, you could, yeah, yeah. So basically you only provide the answer. So you have your LLM generated answer and you submit the answers. And then they have like some process to evaluate on their thing so that their data, private data never leaves their servers, I guess. Because otherwise someone might upload it or something like, you know.

• 47:24Speaker 02

  Yeah, I don't know. I don't, I don't have, I haven't tried it, so I don't, I don't, I don't really know. I'm sure you can sort of reach out to them to figure it out.

• 47:33Speaker 01

  Yeah.

• 47:34Speaker 00

  Anyway. I think this is good unless people have more comments that they want to have.

• 47:39Speaker 02

  I, I think this, but this is only coding, right? But like there's like every other domain needs this.

• 47:45Speaker 00

  The domain coding is the hot thing right now. I think the frontier evals are even more expensive, which is like the apex eval from Merker. Like evals are going to cost, this is millions, they're going to cost tens of millions and hundreds of millions of dollars at the frontier, which is just a very strange dynamic. Whereas like there's just so much about the ecosystem is forking between frontier models and then like research and other things. And trying to follow that dynamic and explain it to people, it's going to take a lot of work.

• 48:14Speaker 01

  But yeah, coding is, I do think, really interesting because that's what most people use L L Ms for these days. But also it is easier to evaluate. I think, once you leave like coding math, it becomes a bit obscure. How do you measure the quality of the answer? You get back to, let's say, preferences, I guess, which is more like a subjective thing where coding is more objective. So it is not a bad thing to do. I think the other day, though, um, uh, Anthropic acquired another company that does like a UI type of stuff on the computer. And I think that is uh.

• 48:47Speaker 02

  Minor doesn't really matter.

• 48:49Speaker 00

  Normal talent, normal talent flows.

• 48:52Speaker 01

  Total. No, I mean, I'm not trying to say this is like a big thing to talk about. Uh, what I'm trying to say is like, this is another interesting um point for evaluating L L Ms on those tasks. Because I think a lot of people want that to like, like they want an L L M to control the computer and do various things. But they are harder to measure. So that will be um maybe two years, we will have something more like benchmarks that can, it's harder to specify. It's kind of like, what is it called? Um, in programming, there's a unit testing and then the system testing, basically like the UI testing and stuff like that.

• 49:28Speaker 02

  Yes.

• 49:28Speaker 01

  Was okay, yeah. And so I think that is the next, uh, maybe gonna be the next.

• 49:32Speaker 02

  Basically, and that thing, yeah.

• 49:34Speaker 01

  Yeah, yeah.

• 49:35Speaker 02

  Uh, G D P Val is usually the thing that gets brought up here. Um, so if, I'll just leave it there. I, I think we've, we've sort of beaten the, the.

• 49:42Speaker 01

  Yeah, the benchmarks, yeah.

• 49:44Speaker 02

  Benchmarking, but uh, definitely G D P Val is sorta here. Um, I'll put it that way. Okay.

• 49:49Speaker 01

  Yeah. Yeah, but like the big topics are essentially the distillation and the benchmarks this week, yeah.

• 49:56Speaker 00

  And welcome to our coalition of.

• 49:59Speaker 01

  Yeah.

• 49:59Speaker 00

  Vals, whatever that means formally.

• 50:02Speaker 02

  Uh, I, I just, it just means I guess hanging with you guys, which is.

• 50:05Speaker 00

  Yeah. I describe, I describe, I mean, it's ultimately a media vehicle. And I think brands and vehicles for media are actually very influential today. I think you see many companies investing in it. And I think it's important to have people that you respect or are aligned with able to amplify each other.

• 50:24Speaker 01

  Yeah, it's also nice to talk to humans, because I noticed the last couple of weeks, if you go to social media, well, I think it's 50% lobsters, like open claw clients nowadays. I get a lot of emails, but also notifications or responses that are, they look AI-generated. So it's nice to also, you know, this human connection and actually talk to like an expert about things.

• 50:50Speaker 02

  Yeah. Cool. There are a bunch of like comments. I don't know if you want to do like quick hits or are you like?

• 50:58Speaker 00

  I have to go to a meeting. That's why I'm trying to wrap this up.

• 51:02Speaker 02

  I see, I see, I see. Well, you know, time is yours. What do you want to do?

• 51:07Speaker 00

  Anyway, thanks, everybody. We'll see you next week.

• 51:10Speaker 01

  Yeah, thanks, everyone, for joining. It was like a nice spontaneous, I guess, you know, discussion. I mean, it always feels nice to talk about things. And too bad, we didn't get too many, or we didn't get to discuss these chat questions, because also my screen, I probably need glasses at some point. My screen is pretty far away. I can just barely read them. But yeah, thanks, everyone, for commenting. It is just nice to see also so many people excited about these topics.

• 51:41Speaker 00

  Yeah. Hopefully, see you later.

• 51:43Speaker 01

  Have a good rest of the day. Bye.