Benchmarks 201: Why Leaderboards > Arenas >> LLM-as-Judge

The first AI Engineer World’s Fair talks from OpenAI https://x.com/aiDotEngineer/status/1811093507463074018 and Cognition https://x.com/aiDotEngineer/status/1811458198920151536 are up! In our Benchmarks 101 https://www.latent.space/p/benchmarks-101 episode back in April 2023 we covered the history of AI benchmarks, their shortcomings, and our hopes for better ones. Fast forward 1.5 years, the pace of model development has far exceeded the speed at which benchmarks are updated. Frontier labs are still using MMLU and HumanEval for model marketing, even though most models are reaching their natural plateau at a ~90% success rate (any higher and they’re probably just memorizing/overfitting). From Benchmarks to Leaderboards Outside of being stale, lab-reported benchmarks also suffer from non-reproducibility. The models served through the API also change over time, so at different points in time it might return different scores. Today’s guest, Clémentine Fourrier https://twitter.com/clefourrier, is the lead maintainer of HuggingFace’s OpenLLM Leaderboard https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard. Their goal is standardizing how models are evaluated by curating a set of high quality benchmarks, and then publishing the results in a reproducible way with tools like EleutherAI’s Harness https://github.com/EleutherAI/lm-evaluation-harness. The leaderboard was first launched summer 2023 and quickly became the de facto standard for open source LLM performance. To give you a sense for the scale: * Over 2 million unique visitors * 300,000 active community members * Over 7,500 models evaluated Last week they announced the second version of the leaderboard. Why? Because models were getting too good! The new version of the leaderboard is based on 6 benchmarks: * 📚 MMLU-Pro (Massive Multitask Language Understanding - Pro version, paper https://arxiv.org/abs/2406.01574) * 📚 GPQA (Google-Proof Q&A Benchmark, paper https://arxiv.org/abs/2311.12022) * 💭MuSR (Multistep Soft Reasoning, paper https://arxiv.org/abs/2310.16049) * 🧮 MATH (Mathematics Aptitude Test of Heuristics, Level 5 subset, paper https://arxiv.org/abs/2103.03874) * 🤝 IFEval (Instruction Following Evaluation, paper https://arxiv.org/abs/2311.07911) * 🧮 🤝 BBH (Big Bench Hard, paper https://arxiv.org/abs/2210.09261) You can read the reasoning behind each of them on their announcement blog post https://huggingface.co/spaces/open-llm-leaderboard/blog. These updates had some clear winners and losers, with models jumping up or down up to 50 spots at once; the most likely reason for this is that the models were overfit to the benchmarks, or had some contamination in their training dataset. But the most important change is in the absolute scores. All models score much lower on v2 than they do on v1, which now creates a lot more room for models to show improved performance. On Arenas Another high-signal platform for AI Engineers is the LMSys Arena https://chat.lmsys.org/, which asks users to rank the output of two different models on the same prompt, and then give them an ELO score based on the outcomes. Clémentine called arenas “sociological experiments”: it tells you a lot about the users preference, but not always much about the model capabilities. She pointed to Anthropic’s sycophancy paper https://www.anthropic.com/research/towards-understanding-sycophancy-in-language-models as early research in this space: We find that when a response matches a user’s views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. The other issue is that Arena rankings aren’t reproducible, as you don’t know who ranked what and what exactly the outcome was at the time of ranking. They are still quite helpful as tools, but they aren’t a rigorous way to rank capabilities of the models. Her advice for both arena and leaderboard is to use these tools as ranges; find 3-4 models that fit your needs (speed, cost, capabilities, etc) and then do vibe checks to figure out which one is best for your specific task. LLMs aren’t good judges In the last ~6 months, there has been an increased interest in using LLMs as Judges: rather than asking a person to evaluate the outcome of a model, you can ask a more powerful LLM to score it. We covered this a bit in our Brightwave episode https://www.latent.space/p/brightwave last month as well. HuggingFace also has a cookbook https://huggingface.co/learn/cookbook/en/llm_judge on it, but Clémentine was actually not a fan of this approach: * Mode collapse: if you are asking a model to choose which output is better, it will just self-reinforce its own preferences. It will also prefer models from its own family (i.e. GPT models will prefer other GPT models over Claude outputs). If these outputs are then used to fine-tune the model, you will further mode collapse the model. Cohere for example has said they do not train on any model-generated data to avoid this. * Positional bias: LLMs usually prefer the first answer, so you can’t naively give them options and ask them to rank them, but you also have to mix up the order in which they appear. * Don’t score, rank: rather than asking a model to assign a score to each output, you should have it stack-rank them. The models aren’t trained to score things, so even though they might understand what response is better, assigning a score to it is hard. If you do have to use LLMs as Judges (we aren’t all ScaleAI-rich!), she suggested using an open LLM like Prometheus https://prometheus-eval.github.io/prometheus/ or JudgeLM https://arxiv.org/abs/2310.17631 to make sure you can reproduce those rankings in the future. Show Notes * Clémentine Fourrier https://twitter.com/clefourrier * Hugging Face https://huggingface.co/ * OpenLLM v2 Leaderboard https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard * Let’s talk about LLM Evaluation https://huggingface.co/blog/clefourrier/llm-evaluation * Leaderboard V2 Blog Post https://huggingface.co/spaces/open-llm-leaderboard/blog * Latent Space Benchmarks 101 https://www.latent.space/p/benchmarks-101 * Gradient AI epsiode on Long Context Evals https://www.latent.space/p/gradient * Allen AI long context novel evals https://arxiv.org/abs/2406.16264 Companies and Organizations * Anthropic https://www.anthropic.com/ * Cohere https://cohere.com/ * EleutherAI https://www.eleuther.ai/ * INRIA https://www.inria.fr/en * ICLR https://iclr.cc/ (International Conference on Learning Representations) People * Aidan Gomez https://x.com/aidangomez * Dan Hendrycks https://people.eecs.berkeley.edu/~hendrycks/ * Edward Beeching https://huggingface.co/edbeeching * Haley Sholkoff https://x.com/haileysch__ * Lewis Tunstall https://huggingface.co/lewtun * Nathan Habib https://huggingface.co/SaylorTwift * Thomas Scialom https://x.com/thomasscialom Projects, Models, and Benchmarks * LMSys Arena https://chat.lmsys.org/ * ARC AGI Challenge https://arcprize.org/arc * Allen Institute ARC Challenge https://allenai.org/data/arc * BigBench https://github.com/google/BIG-bench * GAIA benchmark https://arxiv.org/abs/2311.12983 * GPQA https://arxiv.org/abs/2311.12022 * GSM 8K https://github.com/openai/grade-school-math * IFEval https://arxiv.org/abs/2311.07911 * LightEval https://github.com/huggingface/lighteval * ML perf https://mlcommons.org/en/ * MMLU https://github.com/hendrycks/test * JudgeLM https://arxiv.org/abs/2310.17631 * Prometheus https://prometheus-eval.github.io/prometheus/ * RavenWolf https://x.com/WolframRvnwlf * SWE-Bench https://www.swebench.com/ * Vantage https://handbook.vantage.sh/aws/reference/aws-gpu-instances/ Timestamps * [00:00:00] Introductions * [00:02:32] How Clémentine went from geology to AI * [00:05:52] Origin of the OpenLLM Leaderboard * [00:09:06] How v1 Benchmarks Were Selected * [00:10:49] The Problem with Current Benchmarks * [00:13:45] Saturating benchmarks and the future of evaluation * [00:16:14] Issues with human evaluations * [00:24:07] AI girlfriends as the multi-turn benchmark * [00:25:35] What's New in OpenLLM leaderboard V2 * [00:28:12] Benchmark Answers Black Market * [00:30:21] The impact of prompt formatting on model evaluation scores * [00:33:30] Difficulty and Computational Constraints of Evals * [00:36:28] The Responsibility of Setting Standards * [00:40:35] The Economics of OpenLLM * [00:44:15] Long context reasoning benchmarks * [00:46:34] Agent benchmarks, GAIA, and the ARC AGI challenge * [00:50:43] Vibe check for benchmarks * [00:53:16] Request for benchmarks * [00:56:48] v3 predictions? Transcript Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO-in-Residence at Decibel Partners https://decibel.vc/, and I'm joined by my co-host Swyx, founder of Smol AI https://smol.ai/. Swyx [00:00:13]: Hey, and today we have a super special guest that we've been trying to book on the schedule for a while. It's Clémentine Fourrier. I'm trying my best to do the French, but maybe you can do a better job of it than me. Clémentine [00:00:26]: This was perfect. It's Clémentine Fourrier, but your pronunciation was really on point. Swyx [00:00:31]: There was a Fourrier, which is very sort of French intonation, which I don't really understand. So I'll introduce you off of your LinkedIn and I would love for you to fill in the blanks. You are currently a research scientist at Hugging Face and the maintainer of the OpenLLM leaderboard, which we'll talk about very shortly. Previously, you were at INRIA as well, but then it looks like you also concurrently got your PhD at the same time. How does that work? Is that a very common thing? Clémentine [00:01:01]: So I basically did my PhD at INRIA, technically. So INRIA funded my PhD and PhDs in France are three years, but I also worked as an engineer at INRIA before my PhD, hence maybe the confusion. Swyx [00:01:14]: I think there's a rise in universities having sort of industrial attachments to these things. And I think it actually makes for a much more grounded study, especially if you're doing your sort of graduate studies and all these things. I think it's rising in North America as well with Berkeley and with Waterloo in Toronto. Cool. Like, you know, there's, there's a lot of other things we can, we can introduce. I can't really pronounce the name of the, the university you went to, but what else should people know? Clémentine [00:01:44]: So I actually, technically I'm an engineer in geology. So I studied rocks and I graduated in 2015 after having done like extensive studies about rocks. And I discovered I was very bad at it, but I was very good at computer science. So I went to computer science. What stuck with me though is that geology is very much an experimental science. And I think that machine learning is very much an experimental science too, even though people want to claim that it's pure math. And I worked on several machine learning projects throughout the years, a bit of the prediction of illnesses in the brain at Brain and Spine Institute in Paris. I worked as an engineer in a research team in NLP where I did my thesis and then I joined Hugging Face. Swyx [00:02:32]: Do you have a favorite rock fact or sort of rock story before we get into the NLP stuff? Clémentine [00:02:38]: Okay. I was not expecting this question. Swyx [00:02:43]: I did my geography A-levels and I always loved learning about like isostasy and stuff like that where you have different plates kind of up and down in the mantle. And I don't think people think about vertical dimensions to geographical plates, but it's real. Clémentine [00:03:02]: Yeah, definitely. And like when you do geology, the time scale is just not the same. There is like one specific place in France where you can see rocks that are 1 billion years old and like the sheer scale of this is huge. Yeah, that's what I loved about geology, that the scale is completely different and it makes us see the rest of the world in perspective, I guess. We are like a blink in the length of time of the earth. Swyx [00:03:31]: But a very significant blink. So you went from large monoliths to large language models. I don't know how to make that transition there. And yeah, so like maybe could you describe your journey into Hugging Face? Obviously, I think you're like our second or third person from Hugging Face on the podcast and it's like the definitional sort of open AI company, maybe the real open AI. Clémentine [00:03:56]: Yeah, I did. So at the end of my PhD, I realized that I did not want to stay in academia. And I actually got contacted by Meta because they wanted to offer me an internship. And I was like, wow, I can do an internship during my PhD. Where do I want to do an internship? And so I applied to Hugging Face. Thank you Meta for like opening this door for me. And I actually was hired to work on pre-trained graph transformers. And we train foundational graph transformer models. And it was a very interesting project. But it was a bit hard to accomplish with the resources we had at the time. We tried it for three months, gave it three months more. So the first three months were my internship, three months more were my first three months at Hugging Face. And then we dropped it. We still left a lot of artifacts about graph machine learning that people can use. But we stopped trying to basically compete with Google on this specific topic. And then we had a team which was doing model, like literally LLM training at the time. And we made a list of the different topics. And one topic which people were not interested in was actually evaluation. So I spent a month just reading all the papers I could about evaluation. And I discovered that it was very interesting. And so we started setting up our own internal evaluation suite, which later became LightEval. And once Tom saw that we were interested in evaluation, he sent us the leaderboard, which was a completely different initiative at the time. And then we basically became a small team doing evaluation and leaderboards at Hugging Face. I'm saying we because I'm including Nathan Habib, who is the engineer working with me on evaluation and leaderboards at Hugging Face. Alessio [00:05:52]: Just to set the stage, maybe back in April 2023, we did our Benchmarks 101 episode. And I think everybody was trying to figure out how do you actually evaluate these models. And the models were not very good. Well, first of all, there were not that many models. And the models were not very good at a lot of things back then. Can you maybe give people a bit of a background on how many models you're testing on the leaderboard? I know it's thousands and thousands of models now. And then how you're thinking about what benchmarks matter. And we can go into some of the details. But I think just explain the scale, how many models there are, how many people contribute to this community outside of the actual Hugging Face maintaining team. Okay. Clémentine [00:06:33]: So very beginning, it was really just an internal research project, because our reinforcement learning team wanted to compare some results that they had with published papers, and they did not manage to. And so they opened a small leaderboard where they had manually evaluated a bunch of models. And the community really took over. People were super motivated. And after one month and a half, that's when it was given over to Nathan and I, so that we could make it into an actual engineering product, which could run an actual production rather than a research project. So at the moment, we have evaluated on the version one of the OpenLLM leaderboard, 7,400 models, which have been community submitted for most of them. I think around 800 discussion threads of users interacting with us, either for support or for suggestions. We have had several million visitors since the creation of the leaderboard. And yeah, the scale of it is quite huge. We actually have, from time to time, startups sending us thank you messages saying, oh, our model ranked so high on the leaderboard that we actually got a funding round thanks to you. And so they are very happy about this, and we get thank you messages. So it's quite used, especially by the community. A lot of the community is using it to test their methods and test how well their ideas perform against the SOTA models. Swyx [00:08:11]: It's instructive to rewind back maybe one, maybe two years ago when this kind of leaderboard practice wasn't normal. It's not normal to have an independently validated leaderboard. Everyone just kind of runs their own evals and publishes their evals against their models on their own paper, and it's not reproducible. So I think it's really about reproducible science. I think the only other, before this, it was kind of like ML perf, that's the other big leaderboard that I can think about where, obviously, maybe AlexNet, specific competitions, specific benchmarks, but not something that aggregates across all the other benchmarks. Maybe HuggingFace was involved in BigBench in the earlier days. Clémentine [00:09:03]: Maybe some of the other people in the team. Swyx [00:09:06]: Anyway, so this is the first time getting everything together. And so what was your thinking around inclusion, right? Because I think that's another element that we'll talk about V2 later on, but V1 was your selection of here are the top benchmarks. I don't know if there was any story to tell behind that apart from, was it obvious? Were there any controversial choices? Clémentine [00:09:29]: So for V1, Edward Beeching and Lewis Tunstall, who were our reinforcement learning team at the time, basically wanted to look at the scores which were there in every paper. So they took all the big RL papers of the time and they looked, and you had GSM 8K, you had MMLU, you had ArcChallenge, systematically. So they took those benchmarks because, yeah, they were kind of obvious which ones were the standards of the time. And when we added evaluation, I think we actually added GSM 8K later on. We also tried to add Drop, which we dropped because there was an implementation problem. When we did our round two, we based ourselves, like not the V2, but I'm going to say V1.5. We interacted a lot with the community to see what was missing in terms of evaluations, what were the capabilities that people wanted to see. And we added those datasets at the time. We keep interacting a lot with the RLHF team, like Lewis Tunstall specifically has been very helpful in helping us choose the last set of evaluations for the V2 also. Alessio [00:10:49]: In the V2 announcement blog posts, you mentioned some of the issues that all their benchmarks had. I feel like it's funny to me that now everybody's bringing up these issues, but before they were just using the numbers for marketing and promotion saying, look how good we do. But now it's like, no, actually the benchmark is wrong, like it should be harder. Are people just finding out recently about these problems because now the scores are getting so high that you're actually inspecting the benchmarks and maybe in the past you were scoring so badly that maybe you weren't as worried about the overall quality? Or what do you think now, like, you know, the last maybe like two, three months have been really like where the leaderboard and like has been kind of like taken off as far as popularity and then why it's now the right time to do V2. And then we'll talk about what V2 actually is. Clémentine [00:11:36]: For the first question, when you read evaluation papers, actually a lot of the datasets from, I'm going to say it's a pre-LLM period, are datasets which have been turked basically. So datasets which were made by people which are underpaid, where English is usually not the native language. And so a lot of them have a lot of mistakes and it's kind of obvious just from reading the paper that there are going to be issues because when you generate 10,000 samples, it's quite hard to manually verify each and every one of them. The datasets we've been using, such as MMLU, ArcChallenge and stuff, were of higher quality from the start, but the attention which was given to them forced people to at one point really explore the datasets to see where the scores were coming from. And yes, at the moment, like when a benchmark reaches saturation, so when models basically get the same performance as humans on a benchmark or go above human performance, which the press really likes to say, what it actually says is usually that the models are completely contaminated on said benchmarks and they are now doing errors which humans would not be doing. For example, for MMLU, human performance is at 80-something, also because a number of the questions that humans fail are actually wrong. So the correct answer to those questions is actually a wrong answer. And so the humans who got a wrong answer on this actually had the correct answer. And so if you're getting above humans on this, you have actually learned to predict very wrong stuff, which I find absolutely fascinating. So evaluation has good enough signal-to-noise ratio, if the quality is high enough, it's going to be useful enough for a period of time, and once you reach saturation, you want to inspect it more. Swyx [00:13:45]: There's a three-way race condition as we all figure out who's going to go first. Yeah, so I really like this concept of evaluation. So actually, yeah, I think there's typically what I always say is like sort of 25 is random chance, 50 is average human, 75 is expert human, 90 is you're cheating. And the question now is that most models are high 80s in MMLU, and so it's not challenging anymore or we've sort of saturated it. It looks like some people have put out MMLU Pro. I've seen a few variations of what comes after MMLU. Dan Hendrycks, who came out with MMLU, has promised to make his own MMLU. To me, what I worry about MMLU Pro or any other MMLU variant is that this will last one year. Yeah. Clémentine [00:14:39]: And then what? And then we do Leaderboard v3. Oh, okay. That's it? Swyx [00:14:45]: I mean, yes. Clémentine [00:14:46]: Sorry to disappoint, but like, we basically expect the scale of AI progress to go so fast that anyway, we will have to renew them. Of course, some of it will be for contamination issues, people trying to game the leaderboard, cheating and stuff. But a lot of it will just be because the benchmarks will have become just too easy, like the scale of the progress we did in just one year on those benchmarks is already huge. And I think that's also why the leaderboard was so important, because everybody wanted to climb the scores. And so those evaluations really saw a jump in the performance, like we've got curves on version one, which is archived, but still accessible, of model performance through time. And you can really see the steps which you had for each evaluation. Alessio [00:15:47]: I think the other thing to talk about here is whether or not humans are good at judging and evaluating these models. We're kind of slacking about it over today, but I would love to get your thoughts. It's like, at what point are we not the best people to test these models anymore? And how do you kind of balance the machine benchmarks, the MMLUs of the world, the LMSIS kind of human-driven rating, and then the AI judges, so to speak? Clémentine [00:16:14]: I have many opinions about human evaluation. But I think that... Alessio [00:16:20]: We've got time, so... Clémentine [00:16:23]: Basically, to go back to the initial separation, just to make it clear, so automated benchmarks, like the one we're using on the OpenLLM leaderboard, are usually fair and reproducible. Every model gets evaluated in exactly the same way, and you can really reproduce the scores you get. They tend to be also limited in the scope of what they allow to evaluate, because if you're looking at a multi-choice question, well, it's not telling you how good the model is at generating poetry, for example. So people have been using human evaluations to kind of go further in terms of the capabilities we can evaluate. We've got three types of human evaluations, in my opinion. We've got vibe-check evaluations. We've got Arena-type evaluations, like the LMsys Chatbot Arena. And then you've got human experts, so paid human annotators who will evaluate stuff, which is the approach that Scale has, for example. I think that paid human experts is a really good way to evaluate models, because you can actually give a proper grid of things you want people to check. And because they are actually paid to do so, you can hope for quite a high quality. But since human experts are expensive, people have tried to use model-as-judges, which is our third approach. Model-as-judges, I won't delve too much into this at the moment, but I think they are a problem for the field, actually. I think people should stop using LLMS judges, because they have a lot of subtle biases that they introduce in evaluation. They tend to prefer outputs from the same families. They tend to prefer first answers, which is called a position bias. They tend to prefer long and verbose answers. They struggle with evaluating models in a continuous range. So if you absolutely want to use a model-as-a-judge for your specific use case, do not use GPT-4 also because it's closed source and it will not be reproducible at all. Use a small model such as Prometheus or JudgeLM, and just use it to give you rankings, such as this option is better than this other option. Don't ask it to give you scores, because at the moment those models are not able to do this in a proper fashion. And I saw on Twitter a couple of days ago, Aidan from Cohere, who was saying that their models have a very distinct style, because they don't train with other models' outputs. They actually took the time to gather super high quality data. And other models kind of sound the same because of this. And I think for evaluation, it's going to be the exact same problem. If you choose your model based on model evaluation, you're going to make it kind of the same as all the other models. To go back on human evaluation, if we go on this distinction of VibeCheck versus R&R versus human experts, I think that VibeChecks are actually quite necessary. If you are an engineer and you want to know which model is best for your specific use case, please do a VibeCheck. You can look at a general leaderboard, like the OpenLLM leaderboard. It will tell you which model is best in a range of tasks. And for your use case, you need to test it yourself. For the R&R or R&R-like systems in general, they are trying to rely on wisdom of the crowd approaches. But the wisdom of the crowd tends to work for quantifiable things, right? So it was initially done to try to see if a crowd could average the weight of a pig at a farmer's market in the 18th or 19th century. And it's been reproduced by asking people to estimate a number of a marble in a jar. And for anything which is like super quantifiable, it works very well. But when you're just telling people what is a good output, it's much harder to get something reproducible and experimental science is based on reproducibility, like rigorous protocols. And when using an R&R, you're not getting that. I think that an R&R is a very good sociological experiment, however. I think it's telling you a lot about the users. It's telling you a lot about what are the prompts, how people interact with models. And I also think that you can crowdsource evaluations if you have clear metrics. For example, for red teaming. You can definitely crowdsource red teaming because whether the model gave you private information or whether the model was toxic is something you can have a strict like yes or no answer, in a sense. But for anything else, it's very limited. There were a bunch of papers which were very interesting about this at ICLR this year. There was the psychophancy paper of Anthropic, where basically they showed that humans tend to prefer models which go their way and which agree with them because we want people to like us and apparently we want models to like us too and to agree with us too. Swyx [00:21:49]: Arguably, that's alignment, you know, we want models to like humans. Sometimes it's good. Clémentine [00:21:56]: Yeah. But you also want humans to actually say the truth. Disagree. Yes. Exactly. To challenge you. Definitely. If what you're thinking is not factual. There was also this cool paper by Cohere and the University of Edinburgh, which was human feedback is not gold standard, I think. And where they actually established super interesting things such as humans prefer models which are over assertive. And if you have the choice between an answer which is false, but given super assertively and an answer which is right, but not as assertive, humans naturally will say the assertive but false answer is a better one. So basically, Arenas are not giving you factuality, which should be a super important aspect of LLMs, I think. Alessio [00:22:52]: That's the same with everyday life. You know, people just trust the person saying the thing assertively, even though it's false, and then actually try and figure out what the truth is. So yeah, I think you mentioned that, you know, it's like a more social experiment. I think it's a good point. Like the same biases that people have interacting with humans, they kind of put in the models themselves. Clémentine [00:23:15]: Yeah, definitely. In these things. But there's also the fact that some like the judgments and the likings that we have in real life do not necessarily have the same impacts as LLMs, which are used in production, right? So you don't want the best LLM, according to everybody, to be the one which is going to be the most psychophantic and then get propaganda chatbots or something. On anything like an Arenas, there can be also the problem of the lack of diversity of the annotators, because most of the users of the chatbot arenas, for example, tend to be, from what I gathered, men from the US. I'm sorry, but this is not a diverse demographic. So those are reasons for which human evaluations, in my opinion, are quite limited. Swyx [00:24:07]: I'll throw in one more, which is, I think the sample of the chatbot arena data is actually out there. And most of them are single turn tests as well. Clémentine [00:24:19]: Definitely. Swyx [00:24:20]: So multi-turn is not tested at all. Clémentine [00:24:22]: At the same time, I won't complain too much about this because we also tend to not evaluate multi-turn for automatic benchmark. So I cannot really say anything about this. Swyx [00:24:32]: The AI girlfriend community has got you there. They're very good at the multi-turn and you just need to go to OpenRouter to see which the top trending bots are. For those who don't know, a lot of this is covered in your blog posts, which I think you wrote after ICLR, which is, let's talk about LLM evaluation. You cover sort of a top-down, what you think about evals, and you even point to RavenWolf for the vibe check, who apparently blogs a lot on HuggingFace, because HuggingFace is now a blogging platform and does really good vibe checks, apparently. Clémentine [00:25:08]: He does. I actually found out about the guy on Reddit because he does extremely long threads about the different models he evaluates and the kind of questions that they get right or wrong. He does his evaluations in German, if I remember properly. So it's usually very interesting to see how he does it. He's super rigorous, but he does, I don't know, 15 prompts. So a rigorous vibe check. Swyx [00:25:35]: So I've read those things on the local LLM subreddit and it's a little bit excessive. I don't know if I need all that, but I'm glad somebody does it. To me, he's my automated vibe eval. I don't know who he is, but he shows up, so that's about it. So we wanted to cover specific choices around the new leaderboard. So congrats on launching it. You corrected a bunch of very fundamental data science things, like the variance between the benchmarks, as well as selecting for better benchmarks. I think obviously MMLU Pro is the top one, just because that's the top number that a lot of people report. The headline figures are, for example, it's 10 choices instead of four, and it's actually reviewed by experts instead of just not reviewed by experts. Any other sort of special notes that you would, basically, I want to do a quick tour around the ones that you picked, right? MMLU Pro, GPQA, I think these two are very well regarded. I have Eval as well. I noticed that Apple Intelligence is the only benchmark that Apple Intelligence used. Everything else was their own internal evals, but Apple Intelligence picked IFEval as their benchmark. Anyway, so do you want to comment quickly on some of the ones that you picked? Yeah. Clémentine [00:26:50]: So for IFEval, I think it's a very interesting one because it's like unit tests, but for language, right? When you evaluate coding LLMs, you give them a bunch of unit tests, and you see if the functions that the LLM has written is able to make all the unit tests work. And IFEval behaves in literally the same way. They are giving prompts with very strict instruction formatting, and they are only evaluating instruction following. And I find it very interesting because it's not a metric which is ambiguous at any step. A lot of evaluations which are looking at the content are going to be using bag of words or embeddings to try to get like semantic similarity. Here you don't care. You are literally evaluating on understanding instructions. And I think it's a very smart data set. I loved it. We also added GPQA, which I've wanted to add to the leaderboard since it came out. Basically MMLU, but PhD level. Super complex questions which have been written by PhD experts and which are easy kind of to answer. If you have a PhD in the field, but not if you don't. So I think those ones are super interesting. They are only in science. Alessio [00:28:12]: Yeah, I wanted to know if there's a black market for like the actual data sets that go in the benchmark. I know you have a gating mechanism to get the actual questions to make sure that models don't get contaminated. Do you ever get people reaching out to you? They want to buy the question answers to get better scores on the model. I wonder if marketing budgets are being spent on that. Clémentine [00:28:34]: So for GPQA specifically, anyone can have access to the answers. You just need to create an account and say yes to the gating system and you will have access. The gating system is mostly here for bots, basically to prevent bots passing the web from getting access. However, for the Gaia benchmark, which I was part of, which is a benchmark for agents, we actually got contacted by some institutions from some specific countries who were actually like, well, can you give us the answers to the test sets? We're going to keep it for our Intel benchmarks. And we were like, no, have you heard of what a test set is? But we actually got contacted and they were like, yeah, we think it would really help our safety for our use cases. It's funny. Alessio [00:29:25]: Yeah. Well, I asked thinking that you would say no, nobody would ever ask that, but humans are humans. Also, I know you work closely with Haley Sholkoff from Flutter AI on this, so I DMed her. Last night I asked her some questions I should ask you. So thank you, Haley, for your help. She told me to ask you about MMLU prompt format choices and whether or not there's a right choice when building prompts for the benchmarks. And this is kind of like the GPQA example, you know, maybe you two are experts, so you're kind of having these discussions. For me, it's like, I don't even know what all the options are. So I would love for you to maybe break that down too, you know, there's the benchmark, which is like the questions and the answers and like how you evaluate them. But then there's also how do you prompt the model to actually ask them? So any insights you have, I'm sure would be fun to share. Clémentine [00:30:21]: Okay, so for MMLU specifically, it's a multi-choice evaluation. So you have a prompt and you've got many ways to prompt a model. MMLU that we chose is the one which was used in the harness. So it's question, column, the actual content of the question, return to the line, choices, column, go to the next line, A dot first choice, B dot second choice, and then we return to the line, answer, column. And we did a bunch of experiments at some point by trying different methods, just removing question, removing choices, removing answer. And we got a variation of 30 points on 100, depending on the prompt choices, and 30 points is insane in terms of the variation of evaluations. So the smallest prompt we have was just asking the question, and then we look at the log probabilities of all the choices. So we select the good choice as the one with the best log probability. The more complex one that we had was questions, the question, choices, and enumeration of the choices, but prefixed with letters between parentheses and not letter and then a dot. And this one got the best scores across most models. And in terms of contents, both source prompts have the same, because if you look at the log probabilities, if the model actually has the knowledge, the best log probability should be the best choice, and giving it explicitly the choice should not change anything in terms of contents you're looking at. But yeah, we got 30 points of difference on this. And we actually partnered with Outlines to do a blog post about it on how structured generation can improve evaluations by a lot. And in terms of MMLU, you can also evaluate it in another way, which is what Helm does. And in this case, you do not look at the log probabilities of the choices, you actually ask the model to generate a letter. And you take the generated letter, even if it's not in the option spaces, let's say. So if you say I've got choices A, B, C, D, and the model answers cat, well, cat is wrong. And so, shame for the model, it is wrong. We chose to run multi-choice evaluations in a log-likelihood way because it's way less expensive than running evaluations in a generative way for most tasks. And it's also kind of easy to parallelize, usually, because if you're only looking at one token of generation, then you can batch it very easily. Alessio [00:33:30]: Are the multiple choice benchmarks much easier for the models? Do you have any intuition on how would you stack rank? Because you have the MMLU, then you add GPUA, then you have the math benchmark, you have BBH. Then you run multi-choice, then there's open generation without formatting, then there's formatting-driven ones like IFEval. Which ones are hardest, most impressive? Which ones are easiest? And how did you pick this exact mix? Clémentine [00:34:01]: The two hardest evaluations on our benchmark are math because we only selected the hardest questions. We selected the level five questions. This is a choice that we made because we wanted an evaluation which was discriminative to allow us to see which models were actually good or not. And also because it's very costly to run the full data set. We realized that it would take several hours for a 7b just for this specific data set. And we were like, no, we've got to cut stuff. What do we do? And so one of the reasons behind us using so many multi-choice evaluations is the fact that we are compute constrained. We are using nodes with H100 on them. So every evaluation we'd run on one node with 80 gigs of RAM. And if you look at, for example, Vantage, which shares prices of those kinds of instances, in terms of public price, we are at about $100 an hour. So if we evaluate a 7b model at the moment, it takes approximately two hours. If we evaluate a 70b at the moment, it takes around 20 hours. So there's a limit to how much compute and how much money we can spend on this, right? And this is also a reminder, which is important for the community, because sometimes we get some messages like, I submitted a 70b model yesterday, why was it not evaluated? And I'm like, first of all, do you think compute grows on trees? If you have an NVIDIA GPU tree, give it to me, right? I want more GPUs. And also, it takes a lot of time to evaluate models. And yeah, to go back to your initial question about the benchmarks, the two hardest are so math, and yes, it's a generative evaluation, and generative evaluations in general are harder than multi-choice, but they are also harder to get right because of the metrics. I can go back to this afterwards. And the second hardest evaluation we have is MUSR, multi-step soft reasoning. And it's hard because it's super long context. Basically, it's murder mysteries, and then the model needs to find who is the culprit. The murder mysteries are like rule-based generated, and few models do better than random on this one at the moment. Alessio [00:36:28]: Yeah, great. Great to see benchmarks that models don't do well on. If you just look at the results, it's like, these models are amazing. And then you use them, and you can clearly see there's a lot of room for improvement. So that's great. How do you kind of take this, in a way, responsibility, right? For whether you want it or not, this is one of the lighthouse things that people look at when evaluating models, like your leaderboard. What are maybe some of the hard decisions that you have to make internally? Because you kind of have to balance how you can face the company, but also the scientific objectivity of these things. What are discussions that you had internally on how to pick this, and balancing the commercial side versus the more research side? And yeah, whether or not you had people reach out to you and say, hey, you got this completely wrong. This is actually what the leaderboard should look like. How do you deal with those disagreements from the community? Clémentine [00:37:26]: We know that we have, as you mentioned, a huge responsibility towards the community, because this is a place where people can evaluate their models, and they can also compare and cut through all the marketing b******t, right? If you release tomorrow a model, and you're like, my model is the best model ever, we will actually evaluate it, and we will give you a number. We need to be very fair about our evaluations. This means that for the choice of evaluation, we discussed a lot internally with different people, so Louis, Tensel, Tom Wolfe, Nathan, and I, basically, and so we made short lists of which evaluations are relevant at the moment, both in terms of their contents, in terms of their stability, how well they are seen in the community. And then we spent, I'd say, about a month just running the evaluations on a wide variety of models to make sure that the implementations were absolutely correct and fair for all models. For example, when we were evaluating the version 1 of the leaderboards, we observed that Drop was using a dot as an end of sentence token, and so a lot of floating point answers would be cut off, and so would be incorrect. This was for the v1, and so we actually had scratched entirely this evaluation, because the implementation was incorrect. For v2, we spent much longer just looking at every nook and cranny, making sure the few short samples were fixed, making sure everything was properly formatted, that there were no backslash n running around or whatever. We also know that some models have issues with their tokenizers, so we made sure that they were still being evaluated properly on generative evaluations, because we know it's going to be used, and so numbers need to be as right as possible. And there isn't really a commercial aspect to the leaderboard, however, because basically we are just spending money on the thing, because we think it's a very useful resource for the community to have, but people are not paying for their evaluations to be there. It's a gift to the community, I guess. Swyx [00:39:56]: I wonder about the compute, right? You have basically a standing H100 cluster, but the number of models grows every day. I think you cache them, you also remove models that are maybe contaminated. I think that this happens a lot, that some new model will suddenly show up at the top of the leaderboard, and then people will discuss, and they're like, oh yeah, it's contaminated, and you have to withdraw them. I just wonder about the economics of this thing. How much are you spending? You just have one standing cluster, and you just have a queue. Is that as simple as that? Clémentine [00:40:35]: It's actually more complex. HuggingFace has one research cluster, and so the research cluster is used for every research experiment we have. If the FindWeb team is creating a new super cool dataset for you to train your model on, it's going to be on the cluster. If the IDFX team is creating a new multi-model model, it's going to train on the cluster. The OpenLLM leaderboard team is running on the spare cycles of that. We actually changed the way that our jobs are queued and launched. Basically, the leaderboard jobs are launched with the lowest priority of the cluster. Anything which is launched will kill our jobs if the cluster is too full. So that's why we can give it to the community, in a sense, because it's not costing us that much. It would be lost compute anyway. However, it means that sometimes the queue holds because the cluster is full, and users are not always super happy about it. But they get cool machine learning artifacts, so I think they should be happy. Swyx [00:41:41]: Is there a way for the community to donate compute to you? Is there an interface that you can easily transfer your jobs to a different cluster? Clémentine [00:41:51]: It's actually been discussed a lot, and we are thinking about adding the option to run evaluation on inference endpoint, where people would be able to pay for the compute of their evaluation. The thing is, at the moment, we really wanted to use a EleutherAI harness because it's a big stable library that everybody uses, and we think that Elusive is doing a great job at evaluations in general. But we have the functionality to run evaluations on inference endpoint in our own evaluation library, which is called Lightval. So we will have to port this functionality to the harness before being able to give it to the users. It's not been high on our priority list because then we will have to set up possibly another space where evaluation will run, or maybe people will have to duplicate some stuff. It's more engineering, and we've been a bit swamped with things to do. Swyx [00:42:47]: I can imagine. Yeah, so hopefully when that opens up for inference endpoints, the only thing I'll caution is that all the inference providers write their own CUDA kernels and implementations of stuff. So sometimes you won't get one-for-one the same model, even though it's the same weights, but it's not exactly the same performance of the model because they quantize or do whatever they want to do with the shortcuts for attention. Clémentine [00:43:17]: So regarding quantization, we usually indicate precisely what the precision of the model is. So you can find some models in several precisions. I guess this should be fixed by SART, but yeah, if evaluations run on different hardware with different batch sizes, results are going to be slightly different. Swyx [00:43:38]: We're going to ask maybe three dimensions of benchmarks, and then we'll ask about missing benchmarks that you really want from the community. So the first one is something that the community is discussing a lot, which is long context. You already talked about Muser, but the other one that's popular is the very famous needle in a haystack. There are a lot of variations of needle in a haystack. We talked about this in a previous podcast with advanced needle in a needle stack and variable tracking and all that. Do you think there should be a long context version of the leaderboard, or how are you going to cut it such that you accommodate those things? Clémentine [00:44:15]: For the leaderboard specifically, that's why we added Muser, because it's long context reasoning. In terms of high quality long context reasoning benchmarks, I can think of two which I really like. One is called a benchmark for learning to translate a new language from one grammar book, and it's actually a very fun data set where they basically provide the LLM with a grammar book written by a linguist on a small language, which is super low resource called Kalamang. Since it's so low resource, you're sure that there is no data about it anywhere on the web. Then they ask questions about the grammar, what would be the correct form, etc. This is reasoning, this is language skills, this is super long context because it's a book. I think this data set is very interesting in terms of long context. There was also LNAI, which made a benchmark which they called a novel challenge for long context model, where basically they took full-on novels published last year. They asked people who had read it to do summaries and to do adversarial descriptions of events happening in the book, which require you to have understood the full book to answer. They prompted models with that, so also a very long context evaluation because you've got a full book and then you've got those questions that you need to answer correctly and also not contaminated because hopefully the books are not in training data yet. Yeah, it's new novels. Yeah, definitely. So I think those kind of data sets are more interesting. Yeah, go ahead. Swyx [00:46:03]: You just gave me an idea that Goodreads should be a data set because these are all novels that are commentaries about the contents of the novel. Clémentine [00:46:12]: Definitely. There is definitely something to do about this. Swyx [00:46:18]: Okay, that's long context. Sorry, go ahead, Alessio. Alessio [00:46:21]: You mentioned the GAIA benchmark before that you worked on. What about agents, all of that part? Do you think we have good agent benchmarks? Do you think agent benchmarks are worth it? Yeah, curious for your thoughts. Clémentine [00:46:34]: So for agent benchmarks, I haven't followed the literature so closely for this year, but when we did the GAIA benchmark, the main problem that we observed was that almost all agentic benchmarks would take LLMs, put them in a black box environment, which was absolutely not the real world, and then ask them to do things using very specific APIs. And that's kind of what started the GAIA project, actually, because we had this mental model of what agents could do, especially like AI assistants. We had this list of tasks of we expect them to be able to browse the web, we expect them to be able to extract information from structured places, from having access to modality, tools, etc. And from this, we built the GAIA benchmark. So really not from a capability standpoint, but more through, I'm going to call it proxy tasks, right? We expect agents to be able to do stuff and stuff. So reasoning on so many items, using so many tools. And that's how we built it. Instead of creating those boxed environments, which do not generalize well to the real world, GAIA basically tests your model on the real world. So I hope we get more datasets like GAIA. We basically provided the full recipe, and I really think that anyone could contribute or create similar datasets. So that would be one of the directions I would be excited about to see GAIA 2, GAIA 3, people thinking about creating tools also. Depending on which tools are created, some tasks are going to become way easier. So how do you add complexity to that, etc? Swyx [00:48:28]: I interviewed Thomas Scialom at the ICLR poster session on GAIA, and for people who want to know more about GAIA, they can refer to our ICLR episode. The other big agent benchmark of this year has been SweetBench, much more coding oriented. I'm just curious if you have any thoughts or if you've looked at SweetBench at all. Clémentine [00:48:49]: I remember going to the poster actually, but no, out of the blue, I wouldn't be able to give you feedback on it right now. Swyx [00:48:55]: Just poking. Okay, then we have a question about ARC. Alessio [00:49:00]: Yeah, just curious to get your thoughts. You know, obviously the ARC challenge got a new million dollar boost to get it solved a couple of weeks ago, so a lot of eyes on it. I think maybe some people are saying... Clémentine [00:49:14]: Because we've got two ARC challenges. Like we've got challenge, which is a subset of the LNAI ARC dataset, and then you've got the Cholet ARC AGI challenge. Which one are you talking about? Alessio [00:49:25]: Yes, the AGI challenge. Well, first of all, I'm curious if you think that actually is AGI, if you solve it, and just overall thoughts on the more challenge-driven things rather than evaluation, benchmarks driven. Clémentine [00:49:41]: I don't think if you solve it, it's AGI. I think that focusing at the moment on trying to reach AGI is also a very bad objective, to be fair. But I'm very excited about this specific dataset. I'm looking forward to see what happens because I took a stab at some questions and basically they are great. They are pure logic. One of the things that we are missing at the moment in terms of LLM evaluations is complex logic, I think. Models are very bad at this. If they manage to learn the patterns and generalize on something which is logic-based, then we will have reached a step in reasoning, which will be very interesting. Alessio [00:50:26]: Just overall, more meta question. How do you figure out whether or not a benchmark is actually useful? Everybody wants to build benchmarks, kind of like test sets and things like that. Do you have any quick ways, kind of like you have a vibe check for model? Do you have a vibe check for benchmarks? Clémentine [00:50:43]: Actually, I do, but... Okay, so first thing is, and like the low investment version is, you first look at the paper and you want to see who made the dataset. And by this, I mean, was it model generated? Was it human generated? Were the annotators paid properly? Are they actually native English speakers if your dataset is in English? Etc. You want to know what is the quality of the dataset from the metadata, basically. And then you want to know what were the assumptions behind the dataset. What do they think their dataset is a proxy for? And does that sound logical? And then you want to look at the questions. You want to actually go through the dataset. You want to look at the prompts. Are you able to solve them? Do you see obvious mistakes? Are the prompts cohesive in terms of format? Like, is the formatting consistent? And you want to ideally take a small look at the codes. And if you have more time to invest on this, you can basically just use it for yourself on a bunch of models that you know are good. You want to use it on a small good model, like maybe, I think, 5.3. Like, it's very debated, but it's not that bad for its size. You've got a bunch of around 2 billion parameter models, which are good enough for this. So it wouldn't be too expensive. And then you want to test it on a very big model. That everybody knows is good. Like, when to or command R plus, for example. And if it's a generative model, you look at the generations. Are they, like, well made? Are they truncated? Do they look realistic? And then it gives you more of an idea of the quality. Because the quality of a lot of benchmarks will rely on the quality of their metrics. And if you are using, for example, an exact match metric, you want to make sure that you can actually extract something from the answer. GSM 8K is very good at this, because the output format is very constrained. But some evaluations are very bad at this. Drop, for example, is using a combination of bag of words to estimate whether the correct answer was given. And this is not a good metric, for example. Swyx [00:52:58]: There's an old school NLP thing to use bag of words. Yes. It's kind of like a blue score. Okay, just in case you have one. Is there something that you wish somebody built a benchmark for that you really wanted to include, but you couldn't find it? Clémentine [00:53:16]: Yes. I think that there are a bunch of things that we would need. But one thing is model calibration. Nobody's evaluating model calibration at the moment. And I think it's a problem. Model calibration is... Swyx [00:53:29]: What is model calibration? Clémentine [00:53:32]: You have a very confused face. This is very fun. Basically, a model is said to be well calibrated if the log probability score of an answer correlates well with how correct the answer is. So you want a model which... You can basically see it as the self-confidence of the model, right? So you want a model which tells you, yes, this is true, to have high probabilities of this, if it is actually true. And this thing specifically is called calibration. And it's not that hard to measure. You could use any multi-choice evaluation set to test this. I think there are more interesting datasets to build to test that. But if we have well calibrated models, it will open the door to basically being able to have models with confidence intervals about their answers. And you would be able to say, the model is highly confident about what it's saying. Or the model is in doubt, and you could give small confidence scores. I think this would be very interesting. Swyx [00:54:42]: Yeah, there's some papers at ICLR on uncertainty as well. The quick response I'll give to that is, I think it's well known that base models are better calibrated than instruct-tuned models, right? So just the instruct-tuning just screws it up, makes it overconfident, makes it too much like a human. Clémentine [00:55:00]: Therefore, it's... Yeah, it's tricky. Swyx [00:55:02]: But yeah, I agree with this thing. We should have a benchmark for it and it'll get better. Clémentine [00:55:07]: Yeah, I hope so. And I guess a bunch of other things would be interesting to evaluate. I think that robustness to prompting, nobody does it because it's too expensive. But if I prompt a model with 10 variations of the exact same prompt in terms of content, I don't want to get 10 different answers, right? And it's kind of linked to calibration. It's something that should be taken for granted in LLMs, but it's actually not working that well. And if I had to take a third choice, because I'm very greedy, you asked me for one, but you're getting three. I would love to see more things about psychofancy and basically all the ways into which models can be problematic in their interactions and put people in basically thought bubbles. You don't want people to be on social network too when they talk to a chat model, right? You want the chat model to be assertively saying what is factually true or not. Some things are factually true, right? The earth is round, gravity exists. A lot of things should not be debated and models should be assertively telling users that they are wrong if they are saying that those do not exist. Awesome. Alessio [00:56:27]: This was a great kind of run through the leaderboard and a lot of the questions we already took a lot of your time. Before we wrap, maybe just one last thing. Any predictions for like leaderboard v3? Like if you go one year from now, do you think most models will have kind of top this new v2 too? Or how long do you think it's going to last before you need a new one? Clémentine [00:56:48]: I'm actually working on the next version. Clémentine [00:56:53]: I'm actually working on the next version, which now I'm not going to talk too much about it. But I think that we still have a lot of range for reasoning and mass evaluations at the moment. I think that we still have a lot of the evaluation space to explore. Long context, we're just getting started. I assume that some things like instruction, for like EF eval, for example, I assume that models are going to become very good at it very soon. And sadly, probably GPQA, because I think that it's going to be contaminated at some point. But yeah, basically the next version of the leaderboard would be depending on how fast models changed. It would be a similar version with reasoning, mass, maybe code if we can add it, because now all models should be able to code a bit. And I would really like to add a psychofancy evaluation for the next version. Yeah, well, but it's in the far future. So that's the end of my predictions. Alessio [00:57:58]: Awesome. Yeah, thanks so much for coming on. We're going to link all of your previous work in the show notes so that people can read through it. And people can follow you on Twitter or X to stay up to date. Sorry, Yvonne, don't unfollow us. Swyx [00:58:10]: Follow her on Hugging Face. Hugging Face is a social network. Clémentine [00:58:13]: Yeah, that's true. Alessio [00:58:16]: Yeah, that's it really. Thank you so much. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, Partner and CTO in Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI.

• 0:13Speaker 01

  Hey, and today we have a super special guest that we've been trying to book on this schedule for a while, it's Clementine Fourier. I'm trying my best to do the French, but maybe you can do a better job of it than me.

• 0:26Speaker 02

  This was perfect, it's Clementine Fourier, but your pronunciation was really on point.

• 0:31Speaker 01

  There was a Fourrier, which is very French intonation, which I don't really understand. So I'll introduce you off of your LinkedIn, and I'd love for you to fill in the blanks. You are currently a research scientist at Hugging Face, and the maintainer of the Open LLM leaderboard, which we'll talk about very shortly. Previously, you were at INRIA as well, but then you also, it looks like you also concurrently got your PhD at the same time. How does that work? Is that a very common thing?

• 1:00Speaker 02

  So I basically did my PhD at INRIA, technically. So INRIA funded my PhD, and PhDs in France are three years. But I also worked as an engineer at INRIA before my PhD, hence maybe the confusion.

• 1:14Speaker 01

  I think there's a rise in universities having industrial attachments to these things, and I think it actually makes for a much more grounded study, especially if you're doing your graduate studies and all these things. I think it's rising in North America as well with Berkeley and with Waterloo and Toronto. Cool, like, you know, there's a lot of other things we can introduce. I can't really pronounce the name of the university you went to. But what else should people know?

• 1:44Speaker 02

  So I actually, technically, I'm an engineer in geology. So I studied rocks, and I graduated in 2015, after having done extensive studies about rocks. And I discovered I was very bad at it, but I was very good at computer science, so I went to computer science. What stuck with me, though, is that geology is very much an experimental science. And I think that machine learning is very much an experimental science too, even though people want to claim that it's pure maths. And I worked on several machine learning projects throughout the years, a bit of the prediction of illnesses in the brain. At the Brain and Bite Institute in Paris, I worked as an engineer.

• 2:27Speaker 02

  In a research team in NLP where I did my thesis and then I joined Hugging Face.

• 2:32Speaker 01

  Do you have a favorite rock fact or sort of rock story before we get into the NLP stuff?

• 2:38Speaker 02

  Okay, I was not expecting this.

• 2:43Speaker 01

  I I used I did I did my geography A levels and you know, I always loved learning about like isostasy and stuff like that. Where you have different plates kind of up and down in in the in the mantle. And I don't think people think about vertical dimensions to geographical plates, but it's real.

• 3:02Speaker 02

  Yeah, definitely. And like when you do geology, the the time scale is just not the same. There is like one specific place in France where you can see rocks that are one billion years old. And like the the sheer scale of this is huge. Yeah, that's what I loved about geology, that the scale is completely different and it makes us see the rest of the world in perspective, I guess. Uh, we are like a blink into like lengths of time of years.

• 3:31Speaker 01

  But a very significant blink. So you went from large monoliths to large language models.

• 3:39Speaker 01

  Make that transition there. And yeah, so like, maybe could you describe your your journey into Hugging Face? Obviously, I think you're like our second or third person from Hugging Face on on the podcast. And it's and like the definitional sort of open AI company, maybe the real open AI.

• 3:57Speaker 02

  Yeah, I did. So at the end of my PhD, I realized that I did not want to stay in academia. And I actually got contacted by Meta because they wanted to offer me an internship. And I was like, wow, I can do an internship during my PhD. Where do I want to do an internship? And so I applied to Hugging Face. Thank you, Meta, for like. Opening this door for me. And I actually was hired to work on pre-trained graph transformers. Can we train foundational graph transformer models? And it was a very interesting project. But it was a bit hard to accomplish with the resources we had at the time. We tried it for three months, gave it three months more.

• 4:41Speaker 02

  So the first three months were my internship, three months more were my first three months at Hugging Face, and then we dropped it. Uh, we still left a lot of artifacts about graph machine learning that people can use. But we stopped trying to basically compete with Google uh on this specific topic. And then we had a team which was doing model uh like literally L L M training at the time. And um we made a list of the different topics. And one topic which people were not that interested in was actually evaluation. So I spent a month just reading all the papers I could about evaluation and I discovered that it was very interesting.

• 5:22Speaker 02

  And so we started setting up our own like internal evaluation suite, which later became Lighteval. And once Tom, so that we were interested in evaluation, he sent us uh the leaderboard, which was a completely different initiative at the time. And then we basically became a small team doing evaluation and leaderboard at Hugging Face. I'm saying we because I'm including Nathan Habib, who is uh the engineer, working with me on evaluation and leaderboards at Hugging Face.

• 5:52Alessio Fanelli?

  Just to say the the stage maybe back in April twenty twenty three, we did our benchmarks one on one episode. Um, and I think everybody was trying to figure out how do you actually evaluate these models? And the models were not very good. Well, first of all, there were not that many models. And the models were not very good at a lot of things back then. Um, can you maybe give people a bit of a background on like how many models you're testing on the leaderboard? I I know it's thousands and thousands of models now. And then how you're thinking about what benchmarks matter? And we we can go into some of the details, but I I think like just explain the scale, how many models there are, how many people contribute to like this community outside of the actual Hugging Face maintaining team.

• 6:32Speaker 02

  Okay, so very beginning, it was really just an internal research project because our reinforcement learning team wanted to compare some results that they had with published papers and they did not manage to. And so they opened a small leaderboard where they had manually evaluated a bunch of models. And the community really like took over, people were super motivated and after like one month and a half, that's when it was um given over to Nathan and I. So that we could make it into an actual like engineering product which could run an actual production rather than a research project.

• 7:13Speaker 02

  So at the moment, we have evaluated on the version one of the Open LLM leaderboard, seven thousand and four hundred models, which have been community submitted for most of them. I think around eight hundred uh discussion threads of users interacting with us either for support or for suggestions. We have had several million visitors uh since the creation of the leaderboard. And yeah, the scale of it is quite huge. We actually have from time to time startups sending up like thank you messages saying, oh, our model ranked so high on the leaderboard that we actually got a funding round thanks to you. And so they are very happy about this and we get thank you messages.

• 7:56Speaker 02

  So it's like it's quite used, um especially by the the community. A lot of the community is using it to test their methods and test how well their ideas perform against like the SOTA models.

• 8:11Speaker 01

  I think it's instructive to rewind back like maybe, you know, one, maybe two years ago, when like this this kind of leaderboard practice wasn't uh normal. It's not it's not normal to have like an independently validated leaderboard. Everyone just kind of runs their own evals and publishes their evals against their models on their own paper and it's not reproducible, right? So I think it's really about reproducible science. And um I I think the only other before this, it was kind of like ML perf. Like that's the other like big leaderboard that I can think about where like uh uh obviously maybe AlexNet, like specific competitions, specific benchmarks, but not something that aggregates across all the other benchmarks.

• 8:56Speaker 01

  And maybe Hugging Face was involved in Big Bench.

• 9:00Alessio Fanelli?

  Uh, in the, in the sort of earlier, earlier days?

• 9:02Speaker 02

  Maybe some, maybe some of the other people in the team.

• 9:06Alessio Fanelli?

  And anyway, so this is like the first time getting, getting everything together and like, uh, so like, you know, what was your thinking around like inclusion, right? Because I think that's another element that we'll, we'll talk about V two later on, but V one was your selection of like, here are the top benchmarks. Uh, I don't know if there was any like story to tell be behind that apart from like, was it obvious? Was, was there any controversial choices?

• 9:29Speaker 02

  So for V one, Edward Beaching and Louis Tensel, who were our like reinforcement learning team at the time, basically wanted to look at the, the scores which were there in every paper. So they took all the big RL papers, uh, of the time and they, they looked and you had GSM eight K, you had MMLU, you had arc challenge systematically. So they took those benchmarks because yeah, they were kind of obvious which ones were the standards of the time. And when we added evaluation, uh, I think we actually added GSM eight K later on. We also tried to add drop which we, which we dropped because there was an implementation problem.

• 10:14Speaker 02

  When we did our round two, we based ourselves like not the V two, but the, I'm going to say V one point five. We interacted a lot with the community to see what was missing in terms of evaluations. What were the capabilities that people wanted to see? And we added those data sets at the time. We keep interacting a lot with the RLHF team, like Lewis Tensel specifically has been uh, very helpful in helping us like choose the, the last, uh, set of evaluations for the V two also.

• 10:49Alessio Fanelli?

  And in the V two announcement blog post, you mentioned some of the issues that older benchmarks had. I felt like it's funny to me that now everybody's bringing up these issues, but before that we're just using the numbers uh, for marketing and promotion, saying, look how good we do. But now it's like, no, actually the benchmark is wrong, like it should be harder. Are people just finding out recently about these problems? Because now the scores are getting so high that you're actually inspecting the benchmarks. And maybe in the past you were scoring so badly that maybe you weren't as worried about the overall quality. Or what do you think now? Like, you know, the last maybe like two, three months have been really like uh were the leaderboard and like has been kind of like taken off eh as far as popularity and then

• 11:31Alessio Fanelli?

  why it's now the right time to do V two and then we'll talk about what V two actually is.

• 11:36Speaker 02

  For the first question, when you read evaluation papers, actually, a lot of the data sets from, I'm gonna say the pre L L M periods are data sets which have been turked, basically. So data sets which were made by people which are underpaid, where English is usually not their native language. And so a lot of them have a lot of mistakes and it's kind of obvious just from reading the paper that there are gonna be issues. Because when you generate ten thousand samples, it's quite hard to manually verify each and every one of them. So the data sets we've been using, such as M M L U, Up Challenge and stuff, were of higher quality from the start.

• 12:22Speaker 02

  But the attention which was given to them forced people to, at one point, really explore the data sets to see where the scores were coming from. And yes, at the moment, like when a benchmark reaches saturation, so when models basically get the same performance as humans on a benchmark or go above human performance, which press like really likes to say, uh what it actually says is usually that uh the models are completely contaminated on said benchmarks and they are now doing errors which humans would not be doing. For example, for M M L U, human performance is at eighty something.

• 13:06Speaker 02

  Also because a number of the questions that humans fail are actually wrong. Um so the correct answer to those questions is actually a wrong answer. And so the humans who got a wrong answer on this actually had the correct answer. And so like if you're getting above humans on this, you have actually learned to predict very wrong stuff, which I find absolutely fascinating. So.

• 13:33Speaker 02

  Evaluation has good enough signal to noise ratio. If the quality is high enough, it's going to be useful enough for a period of time. And once you reach saturation, you want to inspect it more.

• 13:45Speaker 01

  There's a three-way race condition as we all figure out who's going to go first.

• 13:51Speaker 01

  Yeah, so I really like this concept of evaluation. So actually, yeah, I think there's typically what I always say is like sort of 25 is random chance. 50 is average human, 75 is expert human, 90 is you're cheating. And like the question now is that most models are high 80s in MMLU. And so like there's it's not challenging anymore or like we've we've sort of saturated it. It looks like some people have put out MMLU Pro. I've seen a few variations of like what comes after MMLU. Dan Hendrycks, who came out with MMLU, has promised to to make his own MMLU.

• 14:29Speaker 01

  It's to me, what I worry about MMLU Pro or any any other MMLU variant is that this will last one year.

• 14:37Speaker 02

  Yeah.

• 14:38Speaker 01

  And then, and then what? So like.

• 14:41Speaker 02

  And then then we do leaderboard V3.

• 14:44Speaker 01

  Oh, OK. That's that's the question. That's it?

• 14:48Speaker 02

  I mean, yes, sorry to the point, but like we basically expect the scale of AI progress to go so fast that anyway, we will have to renew them. Of course, some of it would be for contamination issues. People trying to game the leaderboard cheating and stuff. But a lot of it will just be because the benchmarks will have become just too easy. Like the the scale of the progress we did in just one year on those benchmarks is already huge. And I think that's also why the leaderboard was so important because everybody wanted to climb the scores. And so those evaluations really saw a jump.

• 15:32Speaker 02

  In the in the performance, like we've got curves on the version one, which is archived but still accessible of a model performance through time. And you can really see like the the steps which you had for each evaluation.

• 15:46Alessio Fanelli?

  I think the other thing to talk about here is like whether or not humans are good at judging and evaluating these models. We're kind of slacking about it over today, but I would love to get your thoughts. It's like, at what point are we not the best people to test these models anymore? And how do you kind of balance the machine benchmarks, you know, the M M L use of the world, the L M S is kind of human driven rating? And then the, you know, A I judges, um, so to speak.

• 16:14Speaker 02

  I have many opinions about human evaluation, um, but I think that's.

• 16:20Alessio Fanelli?

  We got time, so.

• 16:23Speaker 02

  Um, basically to go back to the initial separation, just to make it clear. So automated benchmarks, uh, like the one we're using on the open L L M leaderboard are usually like fair and reproducible. Every model gets evaluated in exactly the same way. And you can really reproduce the scores you get. They tend to be also limited in the scope of what they allow to evaluate. Because if you're looking at a multi-choice question, well, it's not telling you how good the model model is at generating poetry, for example. So people have been using human evaluations to kind of go further in terms of the capabilities we can evaluate.

• 17:06Speaker 02

  And we've got like three types of human evaluations, in my opinion, you've we've got like vibe check evaluations. We've got arena type evaluations, like the L M S uh child bot arena. And then you've got human experts. So paid human annotators, uh, who will evaluate stuff, which is like the approach that scale has, for example. I think that paid human experts is a really good way to evaluate models. Because you can actually give a proper grid of um things you want people to check. And because they are actually paid to do so, you can hope for quite a high quality.

• 17:46Speaker 02

  But since human experts are expensive, people have tried to use model as judges, which is our sort of approach. Model as judges, I won't like delve too much into this at the moment, but. I think they are a problem for uh the field, actually. I think people should stop using L L M as judges. Because they have a lot of subtle biases that they introduce in evaluation. They tend to prefer outputs from the same families. They tend to prefer first answers, which is called the position bias. They tend to prefer long and verbose answers. They struggle with evaluating models in a continuous range.

• 18:26Speaker 02

  So if you absolutely want to use a model as a judge for your specific use case, do not use like G P T four. Also because it's closed source and it will not be reproducible at all. Use a small model such as Prometheus or Judge L M and just use it to give you like. Rankings such as this option is better than this other option. Don't ask it to give you scores because at the moment those models are not able to do this in a proper fashion. And um I saw on Twitter a couple of days ago, Aidan from Cohere. Who was saying that their models have a very distinct style. Because they don't train with other models' outputs, they actually took the time to gather super high quality data.

• 19:11Speaker 02

  And other models kind of sound the same because of this. And I think for evaluation is going to be the the exact same problem. If you choose your model based on model evaluation, you're gonna make it kind of the same as all the other models. But yeah, to go back on human evaluation. If we go on this like distinction of vibe check versus R and R versus like human experts. I think that vibe checks are actually quite necessary, right? If you are an engineer and you want to know which model is best for your specific use case. Please do a vibe check. Uh you can look at. Uh a general leaderboard, like the Open L L M leaderboard, it will tell you which model is best in a range of tasks.

• 19:55Speaker 02

  But for your use case, you need to test it yourself. For the RNA or RNA-like systems in general, they are trying to rely on wisdom of the crowd approaches. But the wisdom of the crowd tends to work for quantifiable things, right? So it was initially done to try to see if a crowd could average the weight of a pig at a farmer's market in the like eighteenth or nineteenth century. And it's been reproduced by asking people to estimate a number of marble in a jar. And for anything which is like super quantifiable, it works very well. But when you're just telling people what is a good output, it's much harder to get something reproducible.

• 20:40Speaker 02

  And experimental science is based on reproducibility, like rigorous protocols. And when using an RNA, you're not getting that. I think that an RNA is a very good sociological experiment, however. I think it's telling you a lot about the users. It's telling you a lot about what are the prompts, how people interact with models. And I also think that you can crowd source evaluations if you have clear metrics. For example, for red teaming. You can definitely crowd source red teaming because whether the model gave you private information or whether the model was toxic is something you you can have a strict like yes or no answer, in a sense.

• 21:23Speaker 02

  But for anything else, it's very limited. There were a bunch of papers which were very interesting about this at ICLR this year. There was the psychofancy paper of Anthropic, where basically they showed that humans tend to prefer models which go their way and which agree with them because we want people to like us and apparently we want models to like us too and to agree with us too. There was a

• 21:50Speaker 01

  Arguably, that's alignment, you know, we want models to like humans, so.

• 21:54Speaker 02

  Okay.

• 21:54Speaker 01

  You know, sometimes it's good.

• 21:56Speaker 02

  Yeah, but you also want humans to uh actually say the truth.

• 22:00Speaker 01

  Disagree.

• 22:00Speaker 02

  Sometimes, yes, exactly.

• 22:03Speaker 01

  To challenge you.

• 22:04Speaker 02

  Definitely, uh if what you're thinking is not factual. There was also this uh cool paper by Kohir and the University of Edinburgh, which was uh human, human feedback is not gold standard, I think. And where they actually established super interesting things, such as humans prefer models which are over assertive. And if you have the choice between an answer which is false, but given super assertively, and an answer which is right, but not as assertive, humans naturally will say the assertive but false answer is the better one. So basically, R N As are not giving you factuality,

• 22:47Speaker 02

  which should be a super important aspect of L L Ms, I think.

• 22:51Alessio Fanelli?

  That's the same with everyday life, you know? People just trust the person saying the thing assertively, even though it's false, then then actually try and figure out what the truth is. So uh yeah, I I think you mentioned that, you know, it's like a more social experiment. Uh I think it's a, it's a good point, like the same biases that people have interacting with humans. They kind of put in the models themselves in these things.

• 23:16Speaker 02

  But um there's also the fact that some like the so judgments and the likings that we have in real life do not necessarily have the same impacts as L L Ms which are used in production, right? You do not want the best L L M according to everybody to be the one which is going to be the most psychophentic and then get a propaganda chatbots or something. On anything like an R N A, there, there can be also the problem of the lack of diversity of the annotators. Because most of the users of the chatbots are now, for example, tend to be, from what I gathered, men from the U S. And I'm sorry, but this is not a diverse demographic.

• 24:02Speaker 02

  So those are reasons for which human evaluations, in my opinion, are quite limited.

• 24:08Speaker 01

  I'll throw in one more, which is I think the sample of the Chatbot Arena data is actually out there, and most of them are single-turn

• 24:17Speaker 02

  Yes, also.

• 24:18Speaker 01

  test as well.

• 24:19Speaker 02

  Definitely.

• 24:20Speaker 01

  Right, so multi-turn is not tested at all.

• 24:22Speaker 02

  At the same time, I won't complain too much about this because we also tend to not evaluate multi-turn for automatic benchmarks. So I cannot really say anything about it.

• 24:31Speaker 01

  The AI girlfriend community has got you there.

• 24:37Speaker 01

  They are very good at the multi-turn, and you just need to go to OpenRouter to see which the top trending box. For those who don't know, a lot of this is covered in your blog post, which I think you wrote after I clear, which is let's talk about LLM evaluation. You cover, you know, you sort of top-down what you think about evals and, you know, you even point to Raven Wolf for the vibe check, who apparently blogs a lot on Hugging Face, because Hugging Face is now a blogging platform and does really good vibe checks, apparently.

• 25:08Speaker 02

  He does. I actually found out about this guy on Reddit because he does extremely long threads about the different models he evaluates and the kind of questions that they get, right or wrong. He does his evaluations in German, if I remember properly, so it's usually very interesting to see how he does it. He's super like rigorous, but he does, I don't know, 15 prompts, so a rigorous vibe check.

• 25:35Speaker 01

  So I've read those things on the local LLM subreddit, and like, it's a little bit excessive. I don't know if I need all of that, but I'm glad somebody does it, you know, he's, to me, he's my automated vibe eval. Like, I don't know who he is, but he shows up, so that's about it. So we wanted to cover specific choices around the new leaderboard, so congrats on launching it. You corrected a bunch of very, you know, fundamental data science things, like the variance between the benchmarks, as well as like selecting for better benchmarks. I think obviously MMLU Pro is like the top one, just because that's the top number that a lot of people report. The headline figures are, for example, it's 10 choices instead of four, and it's actually reviewed by experts instead of just not reviewed.

• 26:23Speaker 01

  Any other sort of special notes that you would, you would, basically, I want to do a quick tour around the ones that you picked, right? MMLU Pro, GPQA, I think these two are like very well regarded. IF eval as well, I noticed that Apple Intelligence is the only benchmark that Apple Intelligence used. Everything else was, was their own internal evals, but Apple Intelligence picked IF eval as like their, their, their benchmark. Anyway, so you want to comment quickly on, on some of the, the ones that you picked?

• 26:49Speaker 02

  Yeah, so for IF eval, I think it's a very interesting one because it's like unit tests, but for language, right? When you evaluate coding LLMs, you give them a bunch of unit tests and you see if the functions that the LLM has written is able to make all the unit tests work. And EFFV behaves in literally the same way. They are giving prompts with very strict instruction formatting and they are only evaluating instruction following. And I find it very interesting because it's not, it's not a metric which is ambiguous at any step. A lot of evaluations which are looking at the content are going to be using bag of words

• 27:34Speaker 02

  or embeddings to try to get like semantic similarity. Here you don't care. You are literally evaluating on understanding instructions and I think it's a very smart dataset. I loved it. We also added, so GPQA, which I've wanted to add to the leaderboard since, since it came out. Basically MMLU, but PhD level, super complex questions which have been written by PhD experts and which are easy kind of to answer if you have a PhD in the field, but not if you don't. So I think those ones are super interesting. They are only in science.

• 28:12Alessio Fanelli?

  Yeah, I wanted to know if there's a black market for like the actual data sets that go in the in the benchmark. I know you have a gating mechanism to get the the actual questions to make sure that models don't get contaminated. Do you ever get people reaching out to you? They want to buy the the question answers to get better scores on the model? I wonder if uh uh marketing budgets are being spent on that.

• 28:34Speaker 02

  So uh for GPQA specifically, anyone can have access to the answers. You just need to create an account and say yes to the gating system and you will have access. The gating system is mostly here for bots, basically to prevent bots passing the web from getting access. However, for the Gaia benchmark, which I was part of, which is a benchmark for agents, we actually got contacted by some institutions, institutions from some specific countries, who were actually like, "Well, can you give us the answers to the test sets? We're gonna keep it for our internal benchmarks." And we were like, "Uh, no, have you heard of like what a test set is?"

• 29:16Speaker 02

  But we actually got contacted and they were like, "Yeah, we think it would really help uh safety for our use cases." So.

• 29:25Alessio Fanelli?

  Funny, yeah, well, I I asked thinking that you would say no, nobody would ever ask that, but.

• 29:32Speaker 02

  People do.

• 29:33Alessio Fanelli?

  Humans, humans are humans. Uh, also, I know you work closely with Haley Sholkoff from Luda AI on this, so I DM'd her. Uh, last night I asked her some some question I should ask you, so thank you Haley for your help. She told me to ask you about MMLU prompt format choices uh and whether or not there's a right choice uh when building prompts for for the benchmarks. And you know, this is kind of like the GPQA example, you know, maybe you two are experts, so you're kind of having these discussions. For me, it's like, I don't even know what all the options are. So I would love for you to maybe break that down too, you know, there's there's the benchmark, which is like the questions and the answers and like how you evaluate them. But then there's also how do you prompt the model to actually ask them?

• 30:17Alessio Fanelli?

  So any insights you have, I'm sure will be will be fun to share.

• 30:21Speaker 02

  Okay, so for MMLU specifically, it's a multi-choice evaluation. So you have a prompt and you've got many ways to prompt a model. MMLU that we chose is the one which was used in the harness. So it's question, column, the actual content of the question, return to the line, answer, uh, choices, column, go to the next line, A, dot, first choice, B, dot, second choice, and then we return to the line, answer, column. And we did a bunch of experiments at some point by trying different methods, just removing question, removing choices,

• 31:04Speaker 02

  removing answer, and we got a variation of thirty points on a hundred, depending on the prompt choices, and thirty points is insane in terms of the variation of evaluations. So the smallest prompt we have was just asking the question and then we look at the log probabilities of all the choices. So we select the good choice as the one with the best log probability. The more complex ones that we had was questions, the question, choices, an enumeration of the choices,

• 31:46Speaker 02

  but prefixed with numbers between, with letters between parentheses and not letter and then a dot. And this one got the best scores across most models. And in terms of contents, both those prompts have the same. Because if you look at the log probabilities, if the model actually has the knowledge, the high, like the best log probability should be the best choice. And giving it explicitly the choice should not change anything in terms of contents you're looking at. But yeah, we got thirty points of difference on this. And we actually partnered with Outlines to do a blog post about it on how structured generation can improve evaluations by a lot.

• 32:34Speaker 02

  And in terms of M M L U, you can also evaluate it in another way, which is what Helm does. And in this case, you do not look at the log probabilities of the choices. You actually ask the model to generate a letter. And you take the generated letter, even if it's not in the option spaces, let's say. So if you say I've got choices A, B, C, D, and the model answers uh cat. Well, cat is wrong, and so shame for the model, he's wrong. We chose to run multi-choice evaluations in a log likelihood way because it's way less expensive than running evaluations in a generative way for most tasks.

• 33:18Speaker 02

  And it's also kind of easy to parallelize usually. Because if you're only looking at one token of generation, then you can you can batch it very easily.

• 33:29Alessio Fanelli?

  Are the multiple choice benchmarks much easier for the models? Like, do you have any intuition on like, how would you stack rank? You know, because you have the M M L U, then you add G P U A, then you have like the math benchmark, you have B B H. There's kind of like multi-choice, then there's like open generation without formatting. Then there's like formatting driven ones like I F E V L. Which ones are hardest, most impressive, which ones are easiest? And how did you pick this exact mix?

• 34:01Speaker 02

  The hardest, the two hardest evaluations on our benchmark are maths. Because we only selected the hardest questions, we selected the level five questions. This is a choice that we made because we wanted an evaluation which was discriminative to allow us to see which models were actually good or not. And also because it's very costly to run the full dataset. We we realized that it would take like several hours for a seven B just for this specific dataset, and we were like, no, we've got to cut stuff. What do we do? And so one of the reasons behind us using so many multi-choice evaluations is the fact that we are compute constrained. We are using nodes with eight hundred on them.

• 34:46Speaker 02

  So every evaluation would run on one node with eight hundred with eighty gigs of RAM. And if you look at, for example, Vantage, which shares like prices of those kind of instances, in terms of public price, we are at about a hundred dollars an hour. So if we evaluate a seven B model at the moment, it takes approximately two hours. If we evaluate a seven T B at the moment, it takes around twenty hours. So there's a limit to how much compute and how much money we can spend on this, right? And this is also a reminder, which is important for the community, because sometimes we

• 35:28Speaker 02

  get some messages like, I submitted a seven T B model yesterday, why was it not evaluated? And I'm like, first of all, do you think compute grows on trees? If you have an Nvidia GPU tree, give it to me, right? I want more GPUs. And also, it takes a lot of time to evaluate models. And yeah, to go back to your initial question about the benchmarks, the two hardest are so math. And yes, it's a generative evaluation and generative evaluations in general are harder than multi-choice, but they are also harder to get right because of the metrics. I can go back to this afterwards. And the second hardest evaluation we have is M U S R, multi-step soft reasoning.

• 36:13Speaker 02

  And it's hard because it's super long context. Basically, it's murder mysteries, and then the model needs to find who is the culprit. The murder mysteries are like rule-based generated. And few models do better than random on this one at the moment.

• 36:28Alessio Fanelli?

  Yeah, great. Great to see benchmarks that models don't do well on, you know? Like if you just looked at the results, it's like these models are like amazing. And then you use them and you can clearly see there's a lot of room for improvement. Um. So that's great. How do you kind of take this, you know, in a way, responsibility, right? Like for, you know, whether you want it or not, this is one of the lighthouse things that people look at when evaluating models like your leaderboard. What are maybe some of the hard decisions that you have to make internally? Because you kind of have to balance, you know, how you can face the company, but also like the scientific objectivity of these things. What are, you know, discussions that you had internally on like how to pick this and like balancing the commercial side versus like the more research side?

• 37:14Alessio Fanelli?

  And yeah, whether or not you had people reach out to you and say, hey, you got this completely wrong. This is actually what the leaderboard should look like. Uh. And how do you deal? How do you deal with those disagreements from the, from the community?

• 37:25Speaker 02

  We know that we have, as you mentioned, a huge responsibility towards the community. Because this is a place where people can evaluate their models and they can also compare and cut through all the like marketing bullshit, right? If you release tomorrow a model and you're like, my model is the bestest model ever, we will actually evaluate it and we will give you a number. So we need to be very fair about our evaluations. So this means that for the choice of evaluation, we discussed a lot internally with different people. So Lewis Tensall, Tom Wolf, uh, Nathan and I, basically. And so we made shortlists of which evaluations are relevant at the moment,

• 38:09Speaker 02

  both in terms of their contents, in terms of, um.

• 38:15Speaker 02

  Their their stability, how well they are seen in the community. And then we spent, I'd say about a month just running the evaluations on a wide variety of models to make sure that the implementations were absolutely correct and fair for all models. Um, for example, when we were evaluating the version one of the leaderboard, we observed that drop was using a dot as an end of sentence token. And so a lot of floating point answers would be cut off and so would be incorrect. This was for the V one.

• 38:57Speaker 02

  And so we actually had scratched entirely this evaluation because the implementation was incorrect. So for the V two, we spent much longer just looking at every like nook and cranny, making sure the few short samples were like fixed, making sure everything was properly formatted, that there were no like backslash N running around or whatever. We also know that some models have issues with their tokenizers. So we made sure that they were still being evaluated properly on generative evaluations. Because we we know it's going to be used and so numbers need to be as right as possible. And there isn't really a commercial aspect to the leaderboard, however.

• 39:41Speaker 02

  Um, because basically we are just spending money on this thing. Because we think it's a very useful resource for the community to have. But people are not paying for their evaluations to be there. It's a gift to the community, I guess.

• 39:56Speaker 01

  I wonder about the the the compute, right? Like, um, so you have, uh, you have a basically a standing H one hundred cluster. Um, and the but the number of models grows every day. Uh, I I I mean, I think you I think you cache them. You also remove models that are maybe contaminated. I think that this happens a lot, that some new model will suddenly show up at the top of the leaderboard. And then people will discuss and they're like, oh, yeah, yeah, it's contaminated and you have to withdraw them. But like, you know, I I just wonder about the economics of of this thing. Like, how much are you spending? Like, I you just have one standing cluster and you just have a queue. Is that is that is that as simple as that?

• 40:34Speaker 02

  Okay, so it's actually more complex. So Hugging Face has one research cluster. And so the research cluster is used for every research experiment we have. So if the fine web team is creating a new super cool dataset for you to train your model on, it's gonna be on the cluster. If the ID-FIX team is creating a new multimodal model, it's gonna train on the cluster. So the Open L L M Leadable team is running on the spare cycles of that. We actually changed the way that our jobs are queued and launched. And basically, the leadable jobs are launched with the lowest priority of the cluster. Anything which is launched will kill our jobs if the cluster is too full.

• 41:17Speaker 02

  So that's why we can give it to the community, in a sense, because it's not costing us that much. It would be lost compute anyway. However, it means that sometimes the queue holds because the cluster is full. Um, and users are not always super happy about it. But they get cool machine learning artifacts, so I think they should be happy.

• 41:41Speaker 01

  Is there a way for the community to donate compute to you? Is there an interface that, you know, you can easily transfer compute, uh, transfer your jobs to a different cluster?

• 41:51Speaker 02

  It's actually been discussed a lot and we are thinking about adding the option to run evaluation on inference endpoint, where people would be able to pay for the compute of their evaluation. The thing is, at the moment, we really wanted to use a L user AI harness because it's a big, stable library that everybody uses. And we think that L user is doing a great job at evaluations in general. But we have the functionality to run evaluations on inference endpoint in our own evaluation library, which is called Lightbulb. So we will have to port this functionality to the harness before being able to, uh, give it to the users. It's not being high on our priority list because then we will have to set up.

• 42:36Speaker 02

  Possibly another space where evaluation will run or maybe people will have to duplicate some stuff. It's more engineering and we've been a bit swamped with things to do.

• 42:48Speaker 01

  I can imagine. Yeah, so, okay, hopefully, you know, when that opens up for inference endpoints, I would only, the only thing I'll caution is that if all the inference providers write their own CUDA kernels and, you know, implementations of stuff. And so sometimes you won't get one for one, the same model, even though it's the same weights, but it's not exactly the same performance of the model because they quantize or do whatever they want to do with shortcuts for attention.

• 43:17Speaker 02

  So regarding quantization, we usually indicate precisely what the precision of the model is. So you can find some models in several precisions. I guess this should be fixed by SART. But yeah, if evaluations run on different hardware with different batch sizes, results are going to be slightly different.

• 43:38Speaker 01

  We're going to ask maybe like three dimensions of benchmarks and then we'll ask about missing benchmarks that you really want from the community. So the first one is something that the community is discussing a lot, which is long context. You already talked about Muser, but the other one that's popular is the very famous needle in a haystack. There are a lot of variations with needle in a haystack. We talked about this in a previous podcast with sort of advanced like needle in a needle stack and variable tracking and all that. Do you think there should be like a long context version of the leaderboard or like how are you going to cut it such that, you know, you accommodate those things?

• 44:15Speaker 02

  For the leaderboard specifically, that's why we added MUSA, because it's long context reasoning. In terms of high quality, long context reasoning benchmarks, I can think of two, which I really like. One is called a benchmark for learning to translate a new language from one grammar book. And it's actually a very fun uh dataset where they basically provide the L L M with a grammar book written by a linguist. On a small language which is super low resource called Kalamang. And so since it's so low resource, you're sure that there is no data about it anywhere on the web. And then they ask like questions about like the grammar, what would be the correct form, extra.

• 45:00Speaker 02

  So this is reasoning, this is language skill, this is super long context because it's a it's a book. So I think this dataset is very interesting in terms of long context. There was also Alan A I which made a benchmark uh which they called a novel. Challenge for long context model where basically they took like full on novels published last year. They asked people who had read it to do like summaries and uh to do adversarial. Descriptions of events happening in the book which require you to have understood the full book to answer.

• 45:41Speaker 02

  And they prompted models with that. So also a very long context evaluation because you've got a full book. And then you've got those questions that you need to answer correctly. And also not contaminated because hopefully the books are not in training data yet.

• 45:57Speaker 01

  Yeah, it's new new novels.

• 45:59Speaker 02

  Yeah, definitely. So I think those kind of datasets are more interesting. Yeah, go ahead.

• 46:04Speaker 01

  You just gave me an idea that uh Goodreads should be a dataset because these are all novels. That are commentaries about the contents of the novel.

• 46:12Speaker 02

  Definitely. There's definitely something to do about this.

• 46:17Speaker 01

  Um, okay, that's yeah, that's long context. Sorry, go ahead, Alessio.

• 46:21Alessio Fanelli?

  You mentioned the Gaia uh benchmark before that you worked on. What about um agents? All of that part. Do you think we have good agent benchmarks? Do you think agent benchmarks are worth it? Yeah, cheers for your thoughts.

• 46:34Speaker 02

  So for agent benchmarks, I haven't followed the literature so closely for uh this year. But when we did Gaia, a benchmark, the main problem that we observed was that almost all agentic benchmarks would take L L M's, put them in a black box environment, which was absolutely not the real world, and then ask them to do things using very specific APIs. And that's kind of what started the Gaia project, actually. Because we had this mental model of what agents could do, especially like A I assistants. We had these lists of tasks of we expect them to be able to browse the web.

• 47:17Speaker 02

  We expect them to be able to extract information from structured places, from having access to modality tools, et cetera. And from this, we built uh the Gaia benchmark. So really not from a capability standpoint, but more through, I'm gonna call it proxy tasks, right? We expect agents to be able to do stuff and stuff. So reasoning on so many items, using so many tools. And that's how we built it, instead of creating those boxed environments, which do not generalize well to the real world. Gaia basically tests your model on the real world. So I hope we get more data sets like Gaia.

• 48:01Speaker 02

  We basically provided the full recipe and I, I really think that anyone could contribute or create similar data sets. So that would be one of the directions I would be uh excited about. To see Gaia two, Gaia three, people thinking about creating tools also. Depending on which tools are created, some tasks are going to become way easier. So how do you add complexity to that, et cetera?

• 48:28Speaker 01

  I interviewed Thomas Gallum um at the I Clear uh poster session on Gaia. And you know, for, for people who want to know more about Gaia, they can refer to our I Clear episode. The other big agent benchmark of this year has been Sweet Bench, much more coding oriented. I'm just curious if you have any thoughts or if you've looked, looked at Sweet Bench at all.

• 48:49Speaker 02

  I remember going to the poster, actually. But no, like out of the blue, I wouldn't be able to give you feedback on it right now.

• 48:55Speaker 01

  Just poking. Uh, okay, then, then we have a question about Arc.

• 48:59Alessio Fanelli?

  Let's you. Yeah, just curious to to get your thoughts, you know, obviously, the ARC challenge got a a new million dollar boost to get solved a couple of weeks ago, so a lot of eyes on it. Um, I think maybe some people are seeing the.

• 49:14Speaker 02

  Because we've got two ARC challenges, like we've got challenge, which is a subset of the L N A I ARC data set, and then you've got the Cholé ARC A G I challenge. Which one you're talking about?

• 49:25Alessio Fanelli?

  The the A G I, the A G I challenge. Uh, well, first of all, I'm curious if you think that actually is A G I, if you solve it. Uh, and just overall thoughts on uh the more challenge driven uh things rather than evaluation benchmarks driven.

• 49:41Speaker 02

  I don't think if you solve it, it's A G I. I think that focusing at the moment on trying to reach A G I is also a very bad objective, to be fair. But I'm very excited about this specific data set. I'm looking forward to see what happens because I took a stab at some questions and uh like basically they are great, they are pure logic. One of the things that we are missing at the moment in terms of L N M evaluations is a complex logic, I think. Models are very bad at this, and if they manage to learn the patterns and generalize on something which is um logic based, then we will have reached a step in reasoning, which will be very interesting.

• 50:26Alessio Fanelli?

  Just overall more meta question. How do you figure out whether or not a benchmark is actually useful, you know? Everybody wants to build benchmarks, uh kind of like test sets and and things like that. Um, do you have any quick ways, kind of like you have a vibe check for model, do you have a vibe check for benchmarks?

• 50:43Speaker 02

  Actually, I I do, but okay, so first thing is, and it like the low investment version is you first look at the paper and you want to see who made the data set. And by this, I mean, was it model generated? Was it human generated? Were the annotators paid properly? Are they actually native English speakers if your data set is in English? Et cetera, you want to know like what is the the quality of the data set from the metadata, basically. And then you want to know what were the assumptions behind the dataset. What do they think their dataset is a proxy for? And does that sound logical? And then you want to look at the questions. You want to actually go through the dataset. You want to look at the prompts. Are you able to solve them?

• 51:27Speaker 02

  Do you see obvious mistakes? Um, are the prompts cohesive in terms of format? Like, is the formatting consistent? And you want to ideally take a small look at the code. And if you have more time to invest on this, you can basically just use it for yourself on a bunch of models that you know are good. Uh, you want to use it on a small good model, like maybe, um, I think five, three, like, it's very debated, but uh, it's not that bad for its size. Um, you've got a bunch of around two billion parameter models, which are like good enough for this, so it wouldn't be too expensive. And then you want to test it on a very big model, uh, that everybody knows is good,

• 52:12Speaker 02

  like when to or command R plus, for example. And if it's a generative model, you look at the generations. Are there like well made? Are they truncated? Do they look realistic? And then it gives you more of an idea of the quality. Because the quality of a lot of benchmark will rely on the quality of their metrics. And if you are using, for example, an exact match metric, you want to make sure that you can actually extract something from the answer. G S M eight K is very good at this because the output um, format is very constrained. But some evaluations are very bad at this. Drop, for example, is using a combination of bag of words to estimate whether the correct answer was given.

• 52:57Speaker 02

  And this is not a good metric, for example.

• 52:58Speaker 01

  It's a, there's a old school N L P thing to use bag of words and

• 53:03Speaker 02

  Yes.

• 53:04Speaker 01

  Like, it's kind of like a blue score. Okay, just in case we, you have one, um, is there something that you wish somebody build a benchmark for that you really wanted to include, but you, you couldn't find it?

• 53:16Speaker 02

  Yes. I think that there are a bunch of things that we would need, but one thing is model calibration. Nobody's evaluating model calibration at the moment, and I think it's, it's a problem. Model calibration is

• 53:29Speaker 01

  What is model calibration?

• 53:32Speaker 02

  You have a very confused face. This is very fun. Um, so basically, a model is said to be well calibrated if the log probability score of an answer.

• 53:43Speaker 01

  Oh.

• 53:44Speaker 02

  Correlates well with how correct the answer is. So you want a model which, you can basically see it as the self-confidence of the model, right? So you want a model which tells you, yes, this is true, to have high probabilities of this if it is actually true. And this thing specifically is called calibration. And it's not that hard to measure. You could use any multi-choice evaluation set to test this. I think there are more interesting data sets to build to test that. But if we have well calibrated models, it will open the door to basically being able to have models with confidence intervals about their answers.

• 54:29Speaker 02

  And you would be able to say the model is highly confident about what it's saying, or the model is like in doubt. And you could give small confidence scores. I think this would be very interesting.

• 54:42Speaker 01

  Yeah, there there are some papers at ICLER on uncertainty as well. Uh, the the quick response I'll give to that is, I think, uh, it's well known that base models are better calibrated than instruct tune models. Right? So the just the the the instruct tuning just screws it up, you know, makes it overconfident, makes it too much like a human. Therefore, it's, um, yeah, it's tricky. But yeah, I I I agree with with this thing. We should have a benchmark for it and it'll get better.

• 55:07Speaker 02

  Yeah, I hope so. And I guess a bunch of other things would be interesting to evaluate. I think that robustness to prompting, nobody does it because it's too expensive. But if I prompt the model with ten variations of the exact same prompt in terms of content, I don't want to get ten different answers, right? And it's kind of linked to calibration. It's something that should be taken for granted in L L M's, but it's actually not working that well. And if I had to take a a third choice, because I'm very greedy, you asked me for one, but you're getting three. Um. I would love to see more uh things about psychofancy.

• 55:45Speaker 02

  And basically, all the ways into which models can be problematic in their interactions and put people in basically salt bubbles. You don't want people to be on social network too when they talk to a chat model, right? You want the chat model to be assertively saying what is factually true or not. Some things are factually true, right? The earth is round, gravity exists. Like, a lot of things should not be debated. And models should be assertively telling users that they are wrong if they are saying that those do not exist.

• 56:27Alessio Fanelli?

  Awesome. This was a a great kind of run through the leaderboard and a lot of the questions. We already took a lot of your time. Before we wrap, maybe just one last thing. Any predictions for like leaderboard B three? Like, if you go one year from now, do you think most models will have kind of topped this new B two two? Or how long do you think it's gonna last before you need a new one?

• 56:48Speaker 02

  I'm actually working on the next version.

• 56:53Speaker 02

  I'm actually working on the next version, um, which, no, I'm not going to talk too much about it. But um, I think that we still have a lot of range for reasoning and mass evaluations at the moment. I think that we still have a lot of the evaluation space to explore. Uh, long context, we're just getting started. I assume that some things like instruction for like EF eval, for example, I assume that models are going to become very good at it very soon. And uh sadly, probably GPQA, because I think that it's gonna be contaminated at some point. But yeah, basically, the next version of the leaderboard would be depending on the how fast models changed.

• 57:39Speaker 02

  It would be a similar version with reasoning, maths, maybe code if we can add it, because now all models should be able to code a bit. And I would really like to add a psychofancy evaluation for the next version. Yeah, well, but it's in the in the far future, so that's the end of my predictions.

• 57:58Alessio Fanelli?

  Awesome. Yeah, thanks so much for coming on. We're gonna link all of your previous work in the show notes, so that people can read through it. And people can follow you on Twitter or X to stay up to date. Sorry Elon, don't unfollow us.

• 58:10Speaker 01

  Follow her on Hugging Face. Hugging Face is a social network, yeah.

• 58:13Alessio Fanelli?

  Yeah, that's true.

• 58:16Alessio Fanelli?

  Yeah, that's it really. Thank you so much.