Evaluating LLMs with Chatbot Arena and Joseph E. Gonzalez

In this episode of Gradient Dissent, Joseph E. Gonzalez, EECS Professor at UC Berkeley and Co-Founder at RunLLM, joins host Lukas Biewald to explore innovative approaches to evaluating LLMs. They discuss the concept of vibes-based evaluation, which examines not just accuracy but also the style and tone of model responses, and how Chatbot Arena has become a community-driven benchmark for open-source and commercial LLMs. Joseph shares insights on democratizing model evaluation, refining AI-human interactions, and leveraging human preferences to improve model performance. This episode provides a deep dive into the evolving landscape of LLM evaluation and its impact on AI development. 🎙 Get our podcasts on these platforms: Apple Podcasts: http://wandb.me/apple-podcasts Spotify: http://wandb.me/spotify Google: http://wandb.me/gd_google YouTube: http://wandb.me/youtube Follow Weights & Biases: https://twitter.com/weights_biases https://www.linkedin.com/company/wandb Join the Weights & Biases Discord Server: https://discord.gg/CkZKRNnaf3

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Lukas Biewald?

  You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald.

• 0:11Lukas Biewald?

  Today, I'm talking with Joey Gonzalez, a leading AI researcher at Berkeley with a real focus on LLMs. He's the founder of Chatbot Arena, which is the main place that people in the industry go to see the differences between the top LLM models. He's also the founder of Run LLM, a product that many people use for technical documentation. He's a world expert on building production LLM systems and an expert on how to evaluate models live in production. And so this is a really practical, interesting conversation that I hope you enjoy.

• 0:44Lukas Biewald?

  Maybe we start by the question of what's the research that's most exciting to you right now? And then we'll go into the stuff I've prepared.

• 0:51Speaker 01

  Cool. So, hi, I'm Joey Gonzalez. I'm happy to talk about some of the research we're doing in my group. I am fortunate enough to have a really cool group of students. So we're doing a lot of fun things. The Chatbot Arena is one of the big, big projects in my group, thinking about how we evaluate LLMs in the wild. I can tell you the story behind it, which is kind of funny, and you know, how it came to be where it is today, which is like, pretty influential, actually, for a project that started almost as an accident. I can talk about some stuff. We've been looking at tool use with Gorilla, with thinking about how to evaluate agents. We've been, in my group, looking at memory and how LLMs manage the memory, their context over time. We've been doing some really exciting recent

• 1:32Speaker 01

  work around working with large tables and be able to ask questions. So I'd like to talk to my data, which sounds like text to SQL, but you know, I think there's a lot more to it. Often, being able to make sense of what's in my table requires AI to look at the rows themselves. And how we do that, how we make that cost effective, has been a lot of fun. One of my students, a big weights and bias user, actually, has been looking at vibes, which is like crazy to be doing research on vibes, but in 2025, that's what we do. And so the idea is, can I look at the output of an LLM and get a sense of the vibe of that LLM quantitatively with confidence intervals? And then use that to figure out, you know, is this LLM going to, this particular vibe going to evoke a

• 2:13Speaker 01

  positive response from users and negative response, which we find to be situation-dependent, which is interesting. So different vibes for different settings. Doing a little bit with vision. So a lot of fun stuff.

• 2:24Lukas Biewald?

  I was excited to talk about vibes, actually. I love that you have a sub stack. I've always kind of felt like, you know, the, the research paper format doesn't quite work for a lot of the interesting, you know, stuff that's going on. So I, I was enjoying your, um, your articles and I thought one of the most compelling ones, which I totally agree with, um, the conclusion. So maybe that's dangerous that maybe, maybe there's confirmation vibes here. But, um, I, I like, I, I, what's the title? It's a great title. It's like vibes. Yeah. In defense of vibes based evaluation.

• 2:58Speaker 01

  Yes.

• 2:58Lukas Biewald?

  Which I love. I love it. I love it. Cause I actually think, um, you know, I was thinking about it. Like I remember, you know, when I was working in ML, I had a, I had a boss that was like, I like to hire ex-biologists, not ex-physicists. And I was like, that's really interesting. Why? And he's like, well, like physicists are trained to sort of just like look at the data and the graph. And then biologists like look at the individual examples, which is sort of like, you know, he liked like vibes based evaluation. He's like, you know, in ML, you know, we spend all this time like kind of looking at the aggregate data and we don't like dig into the individual examples and see what's going on. And I always felt that like a lot of the early success I had in my career was from actually looking at individual examples and just getting a feel for it.

• 3:38Lukas Biewald?

  And then, you know, people really started to relate to this, you know, testing by vibes is kind of like stupid because yeah, I think a lot of new people came into the field. And your first thought is like, testing by vibes. You're just like looking at the data. How does it feel?

• 3:50Speaker 01

  Right.

• 3:50Lukas Biewald?

  And I actually think that that is like an incredibly good perspective of like, let's like, not forget to like, not just look at like, you know, F1 score or something and actually like, look at, okay, hey, what's going on? But then you did like a detailed, um, you know, like analysis of where it works and doesn't. So maybe you could just summarize the results and we could talk about it.

• 4:08Speaker 01

  Yeah. So we've had some tension here in my group and I will say coming to this, I had a very machine learning perspective. And again, my students that were very used to, you know, coming to my group meetings and like, let's look at the actual data, which I will again say, they're quite some biases users, like they're used to zooming in and like, let's look at the pictures where it failed. Like, oh, interesting. I see. So those students kind of were pushing me pretty hard. It's like, Joe, we gotta get beyond thinking about just the score or the, you know, the. In our ranking, it really matters the style that it answers. And my startup, it's funny, also came to the same conclusion. We had a startup run LM, we're building AI agents to help people solve problems. And solving a problem is part of the process, but how it does it really matters. Like, people don't like it when it's over preachy, people don't like it when it's like broadly speculative.

• 4:53Speaker 01

  They want short, clean answers, and if I don't get it, I don't know it, just say, I don't know. And, you know, point me at a reference. And so that that sense that it's more than just being correct, that it is about the behavior, the style, that the interaction. It's neat, it's not where I want it to be. So like, coming into this, I'm like, what is a vibe? And my student Lisa Dunlap, who's kind of pushed this this idea very effectively. It worked out a definition, which I like, you know, a vibe is something that helps me understand the style of how a model communicates. It should be something that others could understand. This is a very formal vibe, or very friendly vibe. Where the vibe is, it likes to give examples, give narratives, along with whatever it says.

• 5:36Speaker 01

  It should be something that that helps you differentiate models, like this model Claude is better at giving these examples. And something that also aligns with preference. So, you know, sometimes I want lots of examples, sometimes I just want the answer. And, you know, being able to sense in different settings when that vibe is important. And what my students have found is like, you can use LLMs to compare and contrast various conversations to start to extract likely vibe candidates. Given vibe candidates, you can start to use metrics and the LM as a judge approach to sort of model how that LM aligns on a particular vibe. I mean, then we can use this across many different LLMs, different samples to get an estimate and even confidence

• 6:18Speaker 01

  on, you know, how much a model favors a friendly explanation with lots of formatting or examples versus a more terse and concrete, short explanation, concise explanation. And so, yeah, this idea that correctness is only a small piece of the story is pretty neat, and it gets back to, I guess, human preference, that human preference is about correctness, but it's about a whole lot more. And vibes give us a tool to understand that a lot more.

• 6:46Lukas Biewald?

  And any like practical results on can you describe the vibes of the various models?

• 6:51Speaker 01

  So Lama is a funny model. I think that like the best result I've gotten so far is like Lama, the Lama three latest generation of models, they're much more friendly, they're entertaining, they're less formal. And we've compared like ChatGPT, GPT-4O, it's more formal. It likes longer explanations. We've seen formatting is pretty critical. People care a lot about formatting in the answers. So using Markdown, using LaTeX when needed seems to be important. So formatting is predictive of human preference. Formality is predictive of human preference, but it's very situation dependent. Some people want a friendly answer and that's preferred. In some situations, I want to, again, a concise, formal answer. I don't want to joke around. So yeah, it's pretty neat. There's a paper vibe check with more on this.

• 7:34Lukas Biewald?

  What about concise this? I feel like, you know, my experience like OpenAI is a little less concise than the other models. And I can kind of feel it when people, you know, write to me and they've like obviously used it. I feel like the tell to me is like being more verbose than necessary. Is that, is my vibe accurate?

• 7:50Speaker 01

  Your vibe is accurate. And we found that, in fact, one of the early kind of inclinations in the vibe work was Lisa would go, you know, Joey, I can guess which model I'm talking to the arena. It's got a vibe. It's got a sense to it. And when it's giving me the most verbose answer to a simple question, that's probably GPT-4. And it's true, there is this implicit signal. It differentiates models very effectively. And when in like in our startup, we find that like giving a concise answer is preferred. When I may be trying to explain something to a student, if I'm using AI to come up with an explanation, having a long, thought out process can be helpful. I think it connects to another trend that we're seeing in AI today, which is kind of interesting, this movement towards test time compute, which I think is

• 8:33Speaker 01

  a silly term. It's, you know, prediction time compute, but you know, in the AI world, we only have training and testing. So test time computed is, but the idea that if I think more out loud through my process, this chain of thought, but pushed to the extreme, that I will get ultimately to a better answer. And I think the folks at OpenAI are pushing that that perspective hard. And it is leading to, in many cases, better Final answers, but it also leads to a very drawn out explanation. So, yeah.

• 9:00Lukas Biewald?

  Oh, I see. So the like the explanation is like helping the accuracy by that is actually, right?

• 9:05Speaker 01

  It's a behavioral trick and it's a good one. Like teach the model to, you know, think through what was the question you asked me? Restate your question for you. Like, I don't want to hear it, but yeah, let me restate your question and you know, how should I think about it? Here's how I'll think about it. And then here's your answer. And that process is important for accuracy, but it comes at the cost of the vibe of being concise.

• 9:24Lukas Biewald?

  All right. Well, I was hoping to hold off on chat Bot Arena to save something for the end, but I feel like that ties so directly into chat Bot Arena. So maybe it's start for people that don't know what it is, who've been living in Iraq, like what it is and tell me the story of it. And then I want to get it to have the vibe.

• 9:39Speaker 01

  Tell me the story. I love it. Yes. So let me some brief context. Chat Bot Arena, very, very widely successful and lucky to have, you know, a project like this in my group.

• 9:48Lukas Biewald?

  Totally.

• 9:49Speaker 01

  Service that we've created where people can, anyone can go and chat with the state of the art models today. Open source and commercial and some of the models that even aren't available on the commercial website. So, you know, the next generation of OpenAI and Google's models. And you get to talk to two models at once and, you know, ask your question. You get two answers for free. And you can choose the better of the two and please do vote. And when you vote which one's better, we use that information to build a leaderboard that ranks models. And we do really cool stuff now where we can take those conversations and say, you know, you were asking about homework. It looks like math homework. We can now segment the feedback you give us into different categories. So we can say, look, these models are better math homework. We're not so good at, you know, writing a funny story.

• 10:31Speaker 01

  And so that kind of segmentation allows the AI community to better understand where models are headed. It allows the user community to go, you know what? If I want a funny story, I think I need to use Lama three. To get a better sense of, you know, where models rank on different kinds of tasks. And it's helpful for a lot of the, you know, the big model developers, because it's hard for them to run these studies across all the commercial models. And same for the open source communities. We developed models at Berkeley and at other places. We don't know how those models compare and getting this kind of broad community feedback, if you will, kind of the Wikipedia of MACH evaluation. Is important and it helps kind of level the. In fact, I'm getting ahead of myself. One of the things that we need to do better with the arena and be great to get more help on is getting it out to more people. I think the technology world has picked up on the arena, which is great.

• 11:16Speaker 01

  We need the rest of the world to give feedback to because getting a perspective that broadly represents, you know, human perspective is important to us.

• 11:23Lukas Biewald?

  Totally.

• 11:24Speaker 01

  But now the arena story is kind of funny. It didn't start out with such a grand ambition. Long ago in 2023, we had two models coming from Berkeley and one model from Stanford. And, you know, as good grad students do, we wanted to know is our model better than their model? And so we had started this project to evaluate the Vicuña model from Berkeley, the Koala model, also from Berkeley, and the Stanford Alpaca model. And we had actually started using GPT-4 to evaluate the models. We found out that GPT, that thanks to GPT-4, the Vicuña model is slightly better than the other two. And so we launched a website with our really cool model and a way to chat with it. It was a little single chat box and it was neat.

• 12:05Speaker 01

  And some user had a great question, hey, guys, can you put two side by side so we can put Stanford and Berkeley side by side and chat with both of them? And so we did that. And I think in a last minute, you know, like at the end of the meeting, like, hey, guys, what if we had it? So it was just random too. Like you could also just chat with two random models. It wouldn't tell you which ones. You could pick which one was better. Call it like battle mode or something.

• 12:23Lukas Biewald?

  Nice.

• 12:23Speaker 01

  Yeah, that'd be funny. Let's put that on the website. So we put it on the website and people used it. And like two weeks later, we had enough data to actually build a ranking. And we, you know, argued, how should we do it? Elo, Bradley, Terry, all these techniques. Yeah, we just picked one. We put it out and people were like, wow, this is a great way to just look at human ordering of models. We started adding commercial models and, you know, they like, oh, wait a second. I can now ring GPT-3, GPT-4. You know, Vicuña is moving up or down. And it became a place that people came to see where models are moving. Again, also to chat with models for free. So like we're giving some value back to the, to the people who work with the service. And we also say when you use this, you're giving data to the, to the world. So that data you, you contribute, your conversations will be opened potentially someday.

• 13:06Speaker 01

  And we have open sourced about a million already. We do a lot of work to try to remove anything you accidentally put in that conversation to, to identify you or other people. But this is, you know, an opportunity for us to, to build a data set that will help the open community build the next generation of, of chat AI. So it's not just, you know, owned by, by big tech. So it's become a bigger mission, but it started out with a pretty small, like, wouldn't it be fun, uh, story?

• 13:29Lukas Biewald?

  Totally. Okay. And so, I mean, what have you learned? I actually didn't know that you were able to categorize the content. And where do I find that? I somehow have

• 13:39Speaker 01

  Yeah, so if you go to the chatbot arena, which is now lmarena.org. I think we moved the domain to make it easier.

• 13:45Lukas Biewald?

  It's that AI, it looks like.

• 13:47Speaker 01

  AI, yeah. AI makes more sense. Yeah, so if you go to that domain, you can chat. If you have a leaderboard, there's a little pull down on the leaderboard that lets you pick. Maybe I'll bring it up so I can take a look at the latest results.

• 13:58Lukas Biewald?

  Oh, categories, sorry. I don't know, I missed this.

• 14:00Speaker 01

  What's kind of neat is it changes a lot and pretty quickly. So like, we're kind of surprised at how the things move over time. But if you pick a category like math, you can see how rankings move up or down. O1 preview does better. If you go to like, let's say, coding, all of a sudden, I think O1 mini does better. So in any one of these categories, we're seeing differences in models. We build these bootstrap conference rules, so we can see that they're really statistically different based on the votes that we've collected. If you look at different languages, models, some models are better in other languages than others. So we see a lot of variability. The other thing that's been kind of exciting is the open source models have come up really quickly. Like, you know, while I was like, man, these will never succeed. It's so hard to compete. And they have, and they've moved up.

• 14:43Speaker 01

  So yeah, the open world is also really competing on the leaderboard. We started adding a vision arena, so we'll have a vision leaderboard soon. And you'll see how like, when you start to introduce images, you get different rankings of models. Pretty interesting. Yeah, so a lot of cool stuff is happening now.

• 15:00Lukas Biewald?

  And so, okay, going back to like vibes and style, I would imagine that, you know, the style of response, you know, for just a human side-by-side evaluation is incredibly important to which one feels better. So how do you control for style?

• 15:16Speaker 01

  Ah, so we've started to do this on the leaderboard itself. And it's kind of neat. If we go back to this Bradley Terry model, how we build the rankings. So it's kind of a neat mathematical question. You have an A and a B. Which one's better? You know, we say A is better, B is better, A is better, and so on. So we get just these binary votes. And we use a model that's kind of like logistic regression. Right now, there's just The simplest sense, one feature for model A, one feature for model B. And then, you know, if model A wins, it's a plus one. And if model A loses, it's a minus one. That's a logistic regression problem. The weights attached to each of those, those features are the rankings on the leaderboard. We scale them so they look like Elo numbers for the chess enthusiasts out there. But it's essentially a logistic regression coefficient.

• 16:00Speaker 01

  You can add another term, which is like the presence or absence of formatting for model A or model B. And then we can start to see how those weights explain away some of the value of model A. And so we, we, we extend the logistic regression model, the Bradley Terry modeling framework to account for these extra factors.

• 16:18Lukas Biewald?

  Oh, interesting.

• 16:18Speaker 01

  And that lets us account for it. Yep.

• 16:20Lukas Biewald?

  It's not like the style of writing. It's like, is there, it's like the style of formatting.

• 16:24Speaker 01

  Yeah. So right now, when we look at style, we're looking at formatting, use of markdown, length. We started to control for length. So some, some people like longer, shorter stories. Is the length what's really determining the ranking or not? And so we started to add control for that. In fact, I think you can choose style control right now, which I believe is just kind of markdown. I guess we don't have length control in here yet. But yeah, we're adding support for, for breaking down things like length.

• 16:50Lukas Biewald?

  Interesting.

• 16:51Speaker 01

  Longer queries. Yeah, it's not up yet. Yeah. So this, this, yeah.

• 16:55Lukas Biewald?

  From your modeling, do you have like advice to like if OpenAI wanted to like win this side by side comparison? Should they go longer? Should they go?

• 17:03Speaker 01

  It's a good question. So the interesting point is it depends on the context, right? And so, you know, if I'm looking for an essay or a story, I might want longer. If I'm looking for, you know, a code snippet, I might want shorter. It's very context dependent, the, the style or vibe factors.

• 17:21Lukas Biewald?

  But okay, but in your logistic regression, do you have like, do you let the variable, do you, do you have like second order variables?

• 17:28Speaker 01

  Great question. So right now, the way we approach logistic regression is we do one calculation for all things. That is the arena overall score, which in some sense is silly in retrospect, since there's a lot of variability within a score. But people want to know what's the ranking just on average.

• 17:42Lukas Biewald?

  Yeah.

• 17:42Speaker 01

  All conversations. But we also now, if you say coding, for example, we just throw away all the conversations that didn't mention coding or didn't involve some coding component to it, which we use another model to, to assess. Is this a coding conversation or not? And then we just look at those conversations and recompute the, the leaderboard score with style control on and off. And you can see, you know, if I add style control on, how does that affect the rankings?

• 18:07Speaker 01

  And I think actually it came out when Lama came out, the Lama 3 model, people like, oh, Lama's winning, but it's because it's so good at markdown. It must be it. It's like, oh, maybe we need to control for that. And, and you know, that this is why this emerged.

• 18:20Lukas Biewald?

  Yeah, it's funny. You know, I mean, I feel like on one hand, it's like, maybe that's cheating. On the other hand, it's like, kind of giving people what they want, at least in a context where they're choosing between two things. Like, it almost feels like useful product feedback for a model.

• 18:32Speaker 01

  Yeah. Certainly, if you were designing a model today, you should incorporate style. If you're doing that, do think about how you present your answer across the board. Style seems to help and models that take advantage of it will do better. But we want to understand if that's, you know, the only factor. And there'll be more that, you know, as time progresses, we do more analysis, other factors that we could try to control for on and off and see how they affect the rankings.

• 18:55Lukas Biewald?

  And so you've kind of also put together these kind of benchmark sets, right? Like, it's not just the arena evaluation. Can you talk about those?

• 19:03Speaker 01

  Yeah. So part of running the arena is collecting this data and we want to give this data back in a way that will help people make better decisions. Putting them on the arena is a great way to get a ranking, but it's a hard way to develop a model. So you want more immediate feedback. And so having benchmarks on the kinds of conversations that we see is important. What we've been doing is we use models to try to identify which conversations are challenging. We collect statistics and we break it down by different categories, then try to turn those into new tasks. We calibrate an LM judge to evaluate that task so that you can run your model on that specific task, that conversation or that initial prompt, and then assess your model's performance using the judging frameworks.

• 19:45Lukas Biewald?

  What's your sense, I guess, for models these days? I mean, you're spending all this time looking at like model evaluations. I feel like when I ask people how models are doing, I get wildly different takes on it. Like, what's your feeling? Like, what is the boundary of what models do today?

• 20:02Speaker 01

  So I guess I already touched on the open source models are doing a lot better. Model quality is just is getting better quickly. In some sense, you know, the the middle is converging to something that's very usable. So both open source and commercial models are at a point where I think a lot of innovation now will happen on how we use them, what we build on top of them, the the compound AI, composition, how we transform data as we feed to them. At my startup, we're definitely seeing that, like it used to be, we'd run one model, I think we run many and we change. And one of the things that we've learned is you got to be agile because tomorrow there's another model out there, you probably need to try that one too. And for us at the company, like having good benchmarking, having good processes for, hey, there's another model, let's plug it in and see what happens.

• 20:47Speaker 01

  Does it improve our analysis of our rubrics at the downstream tasks? And we look at things like, is the explanation concise? Is it friendly? So being good at how we evaluate things became important because models are changing so quickly. And a best model of one task can change overnight. And that it's exciting. It's also challenging when you're building products out it, so.

• 21:10Lukas Biewald?

  Totally. What about like, like, are there are there benchmark tasks that are still hard for for models to get superhuman performance? Like, when I look around at, you know, the benchmarks and if they're like, if they're old enough, it almost always seems like, you know, they look impossible, models are struggling. And then, you know, quickly, the models like finds a way like, like, what's like left that we can actually automatically judge that doesn't work?

• 21:39Speaker 01

  It's funny, I've been collaborating with the psychology colleagues at Berkeley. And there's things like theory of mind tasks that are a little challenging. And it's funny, you could think of them as a psychological, like a neat thing to study. But I actually think they're a real problem. And the reason why is is we build agentic systems and we have agents working together. We will make better agents by not having every agent know everything every other agent has said. The idea that, you know, your team, your company runs in one giant Slack channel called General is not a good if you're running a company.

• 22:11Lukas Biewald?

  Right.

• 22:12Speaker 01

  So you want to be able to silo or restrain constrain the context to to the models that you're working with. So they can be more informed and more focused about the things that are important and not the things that aren't. But that means that they need to be able to work together and know, you know what, you don't know what I know. And I need to tell you that so that you can make progress on the task that I asked you to do. And you also, I had a great conversation with another person here and I know that. But again, you don't know what I said to that person or what they said to me. And so this idea of being able to work through an understanding of what other actors in the system know is important to build complex multi-agent systems. And so it's something we need to to work on. I think, yeah, there's a lot of opportunity in, again, how we build on top of these things

• 22:58Speaker 01

  to address some of these problems like theory of mind, but also how we just manage context, memory

• 23:04Speaker 01

  to make them better.

• 23:05Lukas Biewald?

  Totally, cool.

• 23:09Lukas Biewald?

  Okay, another another area of research that that you started to loot, but you have some interesting work on is L M as a judge. I feel like you were talking about this, you know, really early on and and sort of looking at, you know, how how well that works. I feel like L M as a judge went from research topic to like best practice that literally every organization I talked to under the sun does. So what's the what's your current take on the state of the art of L M as a judge? And I guess maybe you should describe what it is in case someone doesn't know.

• 23:39Speaker 01

  And again, I can tell the funny story of how it came to be. You have two models, a Berkeley model and Stanford model. They're trained on the base, the same base models. They kind of know the same things. So you ask them to do a standard benchmark, they get the same score. So how do you know which one's better? Well, I mean, you talk to one and it seems a whole lot better. Berkeley, it's got a, you know, a great story behind it. It's, you know, entertaining, engaging. We need a way to tell that. And, you know, we have a few weeks. We're in a race to get this out. So we could just pay a bunch of people, which is hard and figure out a benchmark, or we could ask G P T four to look at the responses of two models on open-ended tasks with follow-up questions. And so the students, not me, they're brilliant, proposed this idea of let's just ask G P T four to to rank

• 24:21Speaker 01

  pairs of models. We give them a rubric, you know, score them because we can, you know, get humans to do this quickly enough. And it kind of worked and we started to see pretty interesting differentiation. We flipped things around a little bit and saw that it was kind of consistent. And so That's how the first, like, Vicunia, the first result came out. It wasn't later until we tried to formalize this process. And at that point, we realized there's some bugs. So when you look at LMS as a judge and you say, hey, what's better, A or B? GPT-4 goes A. A, probably A. It has a preference for the first thing it sees. This is a bias that humans have as well. It's kind of interesting. So the order in which you present things matters. You have to try both directions to get a good measure.

• 25:03Lukas Biewald?

  Was it GPT-4 the one that didn't have the bias?

• 25:06Speaker 01

  It was just GPT-5. I forget. We looked at a bunch. It was, I think it might have been GPT-4. Anyways, this was a common bias that we saw in the models we were evaluating, which was frustrating because it turns out we had always put our model second. So we actually did even better. We just did.

• 25:19Lukas Biewald?

  Oh, wow.

• 25:20Speaker 01

  And our initial experiments did it in a way that wasn't to our advantage.

• 25:24Lukas Biewald?

  Nice.

• 25:24Speaker 01

  Yeah, so there's a position bias that was interesting. A length bias at that point, and maybe it was GPT-4 again. It preferred longer explanations. Big surprise. And so if you had a longer explanation, you would get a better result. So you need to try to top for length bias. We also looked at self-preference bias. So the models prefer their outputs generally, which again, not super surprising. So we had some issues initially. It's expensive doing lots of binary comparisons. So we started looking at Rubik's as a mechanism to collect more signal when you read a document. And well-defined Rubik's turned out to be a pretty useful strategy, and we use that in other areas as well. I think the binary comparison is still maybe a more consistent way to get signal.

• 26:06Speaker 01

  And like the latest MT bench, latest hard benchmarks use these binary comparisons as well. Yeah, it started out as a neat solution to a problem that we had that I think a lot have adopted. And it's a good, good strategy to get quick signal on how something's performing with good Rubik's.

• 26:23Lukas Biewald?

  Totally. And I mean, now you have like an incredible amount of data to look at, right? I mean, so have you gone back and like checked how well it's working?

• 26:34Speaker 01

  You know, it's a good question. So in the latest benchmarks released, we started using or continued to use the judging mechanism. We've gone back and looked.

• 26:44Speaker 01

  If we can run some of those experiments with newer models, I don't know. It's an interesting question, like if we had today's judges, would they do better? We started looking at panels of judges to see if that would improve stability and get some lift. One of the other weird things I had noticed that models seem pretty consistent in how they behave when given open ended things. Which like, what's your favorite color? Blue. And models share some of those biases too. And so it's possible that even though we're putting piles of judges together, we're not getting as much diversity as we think. And that's something we're actively looking at now. It's like, are models really that different? And you know, this idea that if you generate a bunch of answers that they'll be different. You look closely, they're not actually that different. And so, you know, understanding diversity is an important part of maybe where these things are headed.

• 27:28Lukas Biewald?

  You mean asking the same model, the same question?

• 27:30Speaker 01

  Yeah, so both, asking the same model to answer things differently. Like, we looked at this question of like, name a state in the United States. California. Okay, do it again. California. Do it again. Texas. Do it again. California. Okay. Get enough. It's hard. Don't use these and start with one of those. So they're not as diverse within a model. And then also we see across models, some similarity as well in, you know, their preferences. Which we sort of expect if trained on the, on the, you know, the, the web as a whole, there's probably some distribution on, let's say, states. And those models sort of converge to that distribution. But that might affect how we, you know, use sampling techniques to get, you know, a broad range of answers in our confidence. But if models only have one opinion, you're not going to see that, so.

• 28:13Lukas Biewald?

  Yeah, that makes sense. And I mean, is there a sense that like, using an L M as a judge on itself could? I mean, the obvious thought is that it could prefer, you know, what it's doing. I could think of lots of different reasons that could end up being the case.

• 28:26Speaker 01

  It doesn't work that well. They do for themselves. In our benchmark, it was hard to use the judge as a participant in the benchmark itself. So the panel was one way to try to get around that, but I don't think that actually works, so.

• 28:38Lukas Biewald?

  Tell me about tables and talking to your data. I actually hadn't, hadn't seen that when I was, when I was researching you. So I'm coming in here, Cole, to catch me up on what you're doing.

• 28:48Speaker 01

  Absolutely. So it's actually a fun collaboration with my former thesis advisor, Carlos at Stanford. And Matei Saharia, who's formerly at Stanford, now at Berkeley.

• 28:57Lukas Biewald?

  Is this Carlos's question?

• 28:58Speaker 01

  Question, yeah, exactly.

• 28:59Lukas Biewald?

  No way. Yeah, he was in my lab when I was. Baby at Stanford.

• 29:04Speaker 01

  Oh wow, that's very cool. Yes, this is a really fun project. And maybe the simplest way to describe it is I'd like to be able to ask questions of my table. But here's the thing, I don't want to just ask questions about the data in my table. I want to be able to ask questions that spans data in my table and human knowledge. And I want to be able to ask questions that dig into unstructured parts of my table, like the text. I think the example we've been kind of playing around with is like, what's the best cult classic romance film? And what were the general reviews around that film? And anyone who knows films will be like, oh Titanic, right?

• 29:40Lukas Biewald?

  What's your table?

• 29:41Speaker 01

  So my table doesn't have cult classic, for example. It just has romance films.

• 29:46Lukas Biewald?

  Oh, a table of romance films.

• 29:47Speaker 01

  Yeah, sorry, sorry, I need some more context. So suppose I take a standard table of movies, I have like all the films, they're gross. So I like, what was the most popular? I can get that information. I have genres, like it's romance or not romance, but I don't have cult classic. Like that's just not in a table. And I have the reviews as text, but I want to know, like, what did people think of the best cult classic romance film? And that's a question that spans human knowledge, tabular information. That's a question that requires calculation. The best means I want to find the max over gross, like best selling, most profitable. LLMs are terrible at that, by the way. Things you know, like if you want to, don't use an LM to do math over large amounts of data. It's expensive, it's not accurate, and it's slow.

• 30:31Speaker 01

  So like, yeah, we should use database for that. But, you know, ask a database to summarize the text in a column is not a great strategy either. Like you can group it, but you know, make sense of it, that's tough. So they don't have kind of natural language reasoning. And then, you know, being able to incorporate knowledge that's not in the database, but is general knowledge. That should be something we should be able to do. So we've been thinking about how to bring LLMs and databases together where it's not, I just suck all the data out and stick it in an LM. And it's not where I just run the LM, you know, on all the rows or just on a final tuple. It's like, how do they work more synergistically? And so this is table augmented generation. If you're familiar. Rag, Retrieval Log Management Generation.

• 31:15Speaker 01

  This is TAG, the analogy for tables. And it sort of has two, three main parts. Is one is like, I ask you a question. You should be able to take that question in natural language and generate a program that will run SQL program or sequence SQL programs that will generate the data to answer that question. And those programs themselves will also potentially call to an L M to be able to reason about the data that, you know, the table, that the L M can't, sorry, that the database cannot reason about. And then the final answer might be a table, but I want to, you know, what's the gist of Titanic, right? So it should be able to take that final table and then extract a natural language response.

• 31:52Lukas Biewald?

  You really think Titanic? Before we go for the Titanic is the best?

• 31:57Speaker 01

  Yeah.

• 31:57Lukas Biewald?

  Cult classic romance?

• 31:59Speaker 01

  It's the most grossing according to IMDb. We actually ran this experiment. Yeah, so the, I think is the largest grossing film. It is a cult classic and it is romance. And the reviews are mixed. Reviews are like, we liked it, but the plot was kind of predictable. Yeah, so it's, it's, it's a great example of like, a thing that, you know, a database can't do and L M can't either. But together they can. And I think, I think this is a pretty exciting future because it means that people, humans without sophisticated tooling or knowledge of, you know, data systems can ask really tough questions against large collections of data and hopefully get answers that make sense. In fact, you build a demo right for the election. They'll let you look into, you know, investment in various donors and you can ask natural

• 32:41Speaker 01

  language questions and we'll mix, you know, analysis of their political affiliations, interests and stuff with the text about their bills and stuff. So, yeah, be able to demonstrate these ideas.

• 32:52Lukas Biewald?

  Well, interesting. What an interesting project. And it came from this, the movie thing was really the motivating example here?

• 32:59Speaker 01

  No, unfortunately Titanic wasn't. The motivating example for me was like, I wanted to know how students were doing in my class. I would ask questions and other data and how to do it. And I think I need to be able to read the reviews and like, yeah. But it's an important question, I think.

• 33:14Lukas Biewald?

  Yeah, totally. It's funny, we have questions like that a lot of like, you know, I have a kind of long list of, you know, customer responses and I kind of want to like, you know, interrogate that. And that certainly is tricky to do, like you're saying, because you kind of want to different cuts of the table and then, you know, different kinds of summarization. And I've even wondered about, you know, kind of summarizing where models fail, right? Like, you know, weights and bias obviously tries to surface that data, but it seems like we could do better to be like, summarize like what, you know, what are these examples where the model's struggling? Which is kind of what a human, the first thing a human, you know, would be doing. So yeah, be fun to plug that in and see what we can do.

• 33:55Speaker 01

  Should.

• 33:57Lukas Biewald?

  Cool.

• 34:00Lukas Biewald?

  Okay, tell me about, about tool use.

• 34:03Speaker 01

  Tool use. Yeah, so tool use is a trendy topic and has gotten way more exciting in the past, you know, a few months for the AI world. I mean, abstractly, why should an LM do math? Humans, we don't add square roots of numbers, we use a calculator. So an LM should too. You use the web to figure things out. You don't just think it up on your own. And an LM should do that too. So there's a lot of excitement, has been even before large niggas really took off around tool use. My group was particularly interested in tools as they are APIs, like services that exist in the wild, rather than like a small set of, you know, very specific

• 34:44Speaker 01

  tools. Imagine, you know, being able to interact with the world as, you know, thousands, millions of services, be able to discover the right tool for a task and then be able to invoke that tool. And that was sort of the vision of Gorilla. But if we're like really honest, it's bringing kind of rag ideas to tool use. We just did it a while ago before it was trendy. And part of that was like being able to find a common way to describe a tool. Like what's a JSON spec for a service that I could digest into my list of things you could use model? And then let the model decide which one and how to call it, or, you know, what arguments to give. And that's become standard now, most modeling. Frameworks, most services, in fact, the latest Llama models have pre-training tokens for how to express these kinds of lists of tools.

• 35:28Speaker 01

  And then the model say, I'd like to invoke a tool with a very specific response that allows the the software system running that model to then select the actual API, make a Python call out to a web search engine, run some calculation on a calculator, which really, really augments what these models can do. And so that's been really exciting. When we first did this kind of RAG stuff with with Gorilla, we had a Llama 1 based model that could do better tool calling than, the Llama and was competitive with the GPT-4 models, which very soon after added very similar tool calling specifications. Recently, we've been thinking more about how to evaluate them. So there's lots of models in the wild. Can I compare them in their ability to call tools, reason about multiple tool calls in parallel,

• 36:13Speaker 01

  being able to chain tool calls to accomplish more complex tasks. And so my students have gotten pretty excited about how to evaluate the use of tools in my group.

• 36:22Lukas Biewald?

  Do you have any suggestions? I mean, there's something that I get asked a lot.

• 36:26Speaker 01

  Yeah, so great, great question.

• 36:31Speaker 01

  I mean, we have a leaderboard. It's pretty clear which ones are going to be in the lead right now. One of the things that I think is exciting for me is like, what makes a good spec? Like, how do I make it clear that like what this tool should do so that my models can call it effectively? I think having examples we found early on was helpful in our specs, giving no detailed descriptions of what's going on is pretty helpful. To, you know, putting effort into how we specify our functions makes a big difference in tool use, as one expects. So it's not super surprising.

• 37:01Lukas Biewald?

  Although we had to discover that the hard way. I was actually just talking with our

• 37:04Speaker 01

  Yeah.

• 37:05Lukas Biewald?

  designer who is building some agents using tools. And yeah, he was he was showing me how he improved the spec and that improved the output quite a bit. So a lot of things obvious in retrospect, but there's so many things to try. It's kind of good to sort of know what the best practices are.

• 37:20Speaker 01

  Yeah, I guess one of the things that's kind of weird is early on, we found chain of thought to be really helpful. Models should think through how, why it's using a tool before it goes to call it, set the arguments and, you know, classically, that makes sense. The latest fine tunings of models don't do that. Tool use is just call the tool. Which is saves on, you know, tokens. And if you're just trying to look up the weather, it's a good way to go. But it seems that they moved away from that, which is kind of interesting. And I wonder if that's something we'll see, you know, tick tock back as we move to long, long token sequences before doing things. So maybe smarter tool use. One thing I saw that's pretty neat is this tool use by clicking on the Anthropic's computer. Like, it's another way to look at tool use. It's kind of cool. More interactive, more, I don't know, human centric. And, you know,

• 38:06Speaker 01

  being more human centric seems to be a good strategy as we build these systems. Being able to work in a human world where, you know, there's a browser and you click on stuff. And that's how you interact with tools instead of calling a specific API. Which I hadn't expected, you know, I didn't see that coming. And it's a, it's a cool direction.

• 38:22Lukas Biewald?

  What about, you had another interesting topic on your Substack around reducing hallucinations. Do you want to talk about those results? They seem very practical.

• 38:33Speaker 01

  I'm trying to remember which one this was.

• 38:35Lukas Biewald?

  The challenges of reducing hallucinations. You're talking about using like domain-specific data.

• 38:40Speaker 01

  Ah, yeah. So I guess there are kind of two cuts to this. So one, when we think of hallucinations in our models, I don't like the word, but I just, you know, people like hallucinations, the way to describe you generating the wrong thing.

• 38:57Speaker 01

  Maybe more for me, I like to think like you're in a specific domain and there's ways to write very credible things that are wrong. And those are things I want to avoid. I want to like, I want to align with the domain, but I also want to be correct in that domain. One of the things that struck me, in fact, one of the things that I fought, but I turned out to be wrong about, was fine tuning. So we found that fine tuning our models definitely helps in this particular context. So I can adapt to the kind of vocabulary style of domain through fine tuning. And that does help reduce these errors from hallucination.

• 39:29Speaker 01

  Context obviously matters, so getting the right stuff in context matters a lot. Filtering your context down, we found to be pretty useful. This came out both in the tool use work, but also just more generally in kind of rag or work on Raft. So being selective about what you show to a model makes a big difference in the quality of the answers. So it doesn't try to copy the wrong information into the answer.

• 39:51Lukas Biewald?

  What's the intuition behind, I guess like classically, you know, from my ML background, I'd sort of expect fine tuning to be like generally helpful. But I've seen people have like a lot of different, you know, intuitions here like different than mine. Like, what would be the pro and con case for fine tuning and improving like the specific results?

• 40:13Speaker 01

  Let me, I'll argue why I didn't want fine tuning to be good. Totally. In context, learning is beautiful in that I don't need to adjust my model to use it for new tasks. In some sense, like we're in a race to build AGI, but we already have it. Like the idea that I can take one model and do anything with it. I mean, maybe not human level intelligence, but you know, I could do a lot of things with one model. It's pretty amazing. I don't need to update the model to do new tasks. In some sense, I wanted to believe that we would fine tune to change the behavior of models. But to cook new knowledge in, we could do that just by reading a book, by inserting the knowledge in the context of the of the model. I was wrong, and partly because I think the way I read something,

• 40:56Speaker 01

  having practiced doing that for a while and having practiced the style of, you know, answering questions in a particular Java syntax can make a big difference. Being able in this, this kind of one interesting example of fine tuning that probably in retrospect should have made sense is I've practiced a lot of hugging face function calls. If I don't retrieve the right documentation, but you ask me to do something, I can kind of wing it. Because I've gotten a lot of practice, fine tuning against that particular task. And so this fine tuning helps when there's kind of a very specific domain that's not out in the world. It is critical that you don't just fine tune blindly on your data, that you construct new tasks to reflect the kinds of thing you want the model to do in the long run. And this is kind of a, you know, it's a cautionary tale too.

• 41:41Speaker 01

  So if I, I want a model to generate JSON for a function call. I don't give it a bunch of documentation, say particular documentation. I give it a question documentation and the output JSON that I want it to produce. The consequence of that is it no longer does anything else. So I have a model, it just tells JSON and I want to joke JSON. That's what you're going to get. And so you do, you do tend to fit a model to a very specific domain in the process of doing that, which could be problematic if you want them all to do other things. And that's where like this movement towards compound AI systems to me is kind of exciting. So it means I can fine tune to have a really good function caller, but I can still have a nice friendly chatbot that will still talk to you and the function caller can sit there and call tools when the main chatbot needs help with the tool.

• 42:23Speaker 01

  And so this idea of decomposing things down, specializing my models, I think it was mentioned in this blog, is a way to kind of control hallucination, but also allow me to specialize and perform better at different tasks.

• 42:37Lukas Biewald?

  Can you describe how that system works? Like, like it's, there's something on top, like deciding which model to send the question to?

• 42:43Speaker 01

  So it, we don't have a great answer. So in at RunLM, we have a very specific pipeline. So, you know, given the task, we forgot what kind of code I'm going to do. I look at code as a documentation that I might use a different model that's specialized. We're dealing with that kind of thing. Based on a decision I can make earlier on from the question. More generally, that's an open question for me. Like, how do I abstractly mix a function fine tuned model with a general chat fine tuned model with a rag optimized model? And I think when I get back to this kind of world of multi-agent systems, I think this is where, you know, the exciting next frontier of AI is going to be is how do I compose these, these models that are specialized in different domains? We have very different system prompts to get them to work together effectively.

• 43:27Speaker 01

  I will say again, what we've done at the company, what I think a lot of people are doing right now is to have like some routing model that makes some basic decision and delegates to models accordingly. But, you know, that's, that's step one. It would be so much cooler if they could like talk together and say, you know, that would not me. Pass that on to this other person or, you know, I'd like some, you know, feedback from the code expert before I answer this question. And so having more direct interaction between models is, you know, pretty exciting.

• 43:50Lukas Biewald?

  Totally.

• 43:53Lukas Biewald?

  A little, that's a little bit of a segue into Route L M. Did you want to talk about how you set that up?

• 43:59Speaker 01

  Yeah, so Route L M was a project the students had built out of the arena, which in retrospect was pretty obvious and brilliant. Like, we're figuring out which model is best answering a question based on, you know, different types of questions. We should be able to use that to select between models. We weren't the first of this. Colleagues at Stanford had also started looking at, you know, how to build deluxe chatterish, you know, selection tactics for models. In fact, I think back in like 2015, 2016, maybe, or maybe it was 2017, we were doing work on like routing inside of computer vision models. So the field of routing across models is, you know, pretty, pretty well studied. Route L M was playing to the advantage of the arena. We have all this amazing data to, you know, help make decisions about which model to use to get better answers.

• 44:44Speaker 01

  And you could imagine doing even cooler things in the future. And, you know, there's students looking at that now. It's like, can I like predict an entire leaderboard based on your question? And do things, you know, maybe I have a cost performance trade off that I want to play. Maybe I should mix models in a way. So routing is kind of selecting models. Can I combine them or have them work together to answer a question?

• 45:02Lukas Biewald?

  And if you just wanted the optimal performance, like, how much better can you do? Like, I have this sense that lately, you know, O1 is really dominating leaderboard, although I'm looking at it and it seems like Gemini is at this moment in time, late November.

• 45:15Speaker 01

  Gemini is coming back.

• 45:16Lukas Biewald?

  Yeah, it seems like it's coming back.

• 45:16Speaker 01

  Google OpenAI competition is awesome.

• 45:19Lukas Biewald?

  But, but like, how much, how much higher can you go by like optimal routing?

• 45:24Speaker 01

  You know, I don't think we got a significant boost over say the art. A lot of the opportunity comes in maybe saving costs right now. But I do think in the future, there is a chance that you could refine questions. You could have models think and work together to get to a better answer.

• 45:41Speaker 01

  So I don't like, I don't think we've gone as far as you could with that. And again, this is, one of my students said this, like, if all we, if we were stuck with the models we have today, I think we'll see a decade more of exciting research. And that student, who's now the CEO of Litta, another startup from the lab, was right. I think he's going to be right. He'll be proven right in 10 years that even the models today, how we put them together will make an enormous difference in what they can do tomorrow.

• 46:06Lukas Biewald?

  Wow, a decade of improvement is

• 46:08Speaker 01

  Yeah.

• 46:09Lukas Biewald?

  Is a bold

• 46:09Speaker 01

  It's a pretty bold statement. Maybe it'll be tough, but I bet we could do a few years. I think he'll be right about that. It's hard to make predictions in this field.

• 46:18Lukas Biewald?

  Totally, totally. Where do you think other productive lines of inquiry? Like, I didn't see a lot, you know, like researching on like, kind of prompt optimization. Like, do you think that's an interesting area of research?

• 46:30Speaker 01

  I'm not doing a lot there. We had done some early work, Project Tempura, using kind of RL techniques to do like soft prompt optimization. I haven't done as much since then. My colleague Mate has been working on Project DS Pi. That's pretty cool, doing kind of prompt tuning, doing few-shot optimization. I think there's a lot we could do. There are things people could do there. For me, I'm a little more excited about how we manage the context in a kind of repeated conversation, which is a form of prompt optimization. But the idea that, you know, I might filter what information is presented to the model at each step. The MemGPT, kind of the Leta project, it's more exciting to me to think of like, maybe there's like a memory management system

• 47:14Speaker 01

  that's consciously modifying my prompt to make it better. So that, you know, I as a model can make better decisions, you know, practically maybe a human too. You have too much information on your plate. Here's what you need to know for this email. Great, that would be wonderful. And so like giving the, servicing the right information at the right time makes a big difference for humans and for AI. And that sort of flavor of prompt optimization is kind of exciting to me.

• 47:37Lukas Biewald?

  I guess the memory management kind of breaks the model of your your arena mode, right? Like, I do feel like it's hard to tell how much customized results I'm getting at this point, but clearly there is a fair amount.

• 47:52Speaker 01

  Yeah, great, great question. So the arena right now treats people as, you know, stateless. So you come to the arena, you're a new person. ChatGPT, Claude, like all these services start to have a memory. And they even tell you, remembering something, that's cool.

• 48:07Speaker 01

  That's important to doing better in a, you know, a long-term conversation. That context that you develop over time is pretty critical. How we surface that is important. The arena is certainly not testing that right now. It will be, maybe. We're looking at adding agents and kind of accounts so we could start to track these things over time and help you have a better experience with the arena and also, you know, better model how these, you know, how context plays a role.

• 48:34Speaker 01

  It's a good point. We're not playing a lot with memory in the arena. I think it is an area where these models will evolve, where we will get wins. I think when we build agents and they start to work together and they have to remember the context of what they're doing, it's important.

• 48:54Speaker 01

  Yeah, it's an exciting next direction.

• 48:57Lukas Biewald?

  Are there other leaderboards that you look at and admire for other topics?

• 49:03Speaker 01

  It's a good question. So I still turn to the basics, MMLU. I want to know how things are performing. The recent math benchmarks have become pretty useful for me to get a sense of how things are performing, especially through this test time compute, the long chain of thought reasoning.

• 49:19Speaker 01

  What else are we looking these days? For me, like the company, we actually don't use a lot of leaderboards internally. And that's a good thing. Like I think their leaderboard is the arena is a great way to get an overall picture. But if you're looking at the arena to figure out what your company should be doing,

• 49:38Speaker 01

  you might be doing it wrong. You should be looking at what your internal leaderboard is showing. Like people should start collecting this data and start to try to understand how a model is performing for them. It's critical because they will change. Yeah, so yeah, we look at the arena, but also for a lot of the tasks I'm looking at, we have very specialized boards we'll look at.

• 49:57Lukas Biewald?

  What about Sweetbench? I feel like that has a lot of purchase.

• 50:00Speaker 01

  Yeah, we started looking at that. Sweetbench results are terrible. It's exciting because it's a hard task. I think, you know, working on agent systems, they don't work. The multi-agent systems agents, like we're not there yet. There's a lot to do to make them better. And in fact, right now, for me, I'm trying to understand how do they fail? Like, do you have a good understanding of why they don't work? When they don't work, what do they do wrong? And like our best hypothesis right now is that a lot of failure is getting lost, getting stuck and then doing the same thing over and over again until you timeout. Or like, I figured out what the bug is. Done. We didn't solve it. Oh, I forgot. I was supposed to do that, too. And this idea that like I've worked all this, you know, this context up to figure out the problem.

• 50:44Speaker 01

  And I figured out the problem. And I forgot what to do is sort of getting back to the story of memory. I think like being able to say, here's the steps I'm in. I finished that step. Success. But actually, now I have to go back to the beginning. And be able to guide the model, you know, through its its thought process is pretty critical.

• 51:02Lukas Biewald?

  Look, I thought we'd end with, you know, RunLM where you're like, actually, like, you know, kind of putting these things in production. Do you want to talk about like, you know, what the use cases are and how far along this this product is?

• 51:14Speaker 01

  Yeah. So RunLM, we're building essentially, right now it looks like a support bot. But actually, it's pretty exciting. We're building something a little bit bigger than that. We're building an AI that both interfaces with customers to help them solve technical problems, help them understand the product that you're developing. You know, use, say, weights and biases. So I want to know, how do I use weights and biases to do this? Can I use weights and biases to do this? What is weights and biases? And be able to ask the dumb questions that you wouldn't ask a real person. Be able to ask the questions that you were like, I'm trying to do this with this data. And this endpoint, what's the code to do that? Which you wouldn't ask a support person, but an AI is like, yeah, I'm happy to work with you on your code.

• 51:55Speaker 01

  And so with the RunLM, we've been very focused on doing that right now. But what I think is super cool is weights and biases can go, hey, what are my customers confused about? I want to ask you, you know, where are they confused? And when they're encountering this problem, can you encourage them to look at this other thing, too? We have a new thing coming out. They should try that. And so being not just an interface for customers to a company to understand the problems, but also an interface for a company to its customers to interact with that experience. And filling more of the role of a, you know, of a support engineer. They can talk to customers. They can track issues. We're adding the ability to like, interact with your support system, with your development interfaces. We've done a lot of work with code, so like, you don't have to document everything amazingly well. We'll actually just check your code.

• 52:39Speaker 01

  See, how does it work? Oh, yeah, that's the here's how to answer that question. What am, it's funny, at RunLM, we've added support for some of the research projects. To my group, I run the another project VLM, which is a serving platform for models. So a lot of users out there and my grad students recently told me like Run LM is supporting their, their, you know, support interface, they're running their, their docs.

• 53:00Lukas Biewald?

  That's awesome.

• 53:01Speaker 01

  And when people get stuck with how do I do stuff with VLM, they go to Run LM and it actually answers better than we were answering. It's like we're in, we didn't document a lot of stuff, it was a research project. And so like they don't need the documentation, they just get, you know, it goes and looks at the code and says, here's the thing you need to do differently. And like, wow, that's amazing. So yeah, this idea of being able to do more than just answer questions, be able to engage customers, provide a voice to the customers back to the company. Make sense of, you know, just not just the documentation, but the code. We also recently had the ability to find bugs in your documentation. So like your documentation disagrees your code and we can flag that. And people are getting stuck on this thing. You should take a look at that. And this kind of bidirectional voice is, you know, something I'm pretty excited about that we're doing. That's been a lot of fun.

• 53:44Lukas Biewald?

  No, it's super cool. Is it, um, do you develop that, is there anything different about developing like a customer facing product than like a research? Paper oriented project?

• 53:58Speaker 01

  Oh yeah, there's a lot. I mean, certainly the core elements have a lot of the flavor of research. We are trying lots of stuff and trying to collect data constantly. We have again, a whole suite of things that we test internally. So we can say, hey, oh, one mini came out. Should we use that one instead? And being able to agilely try new things, kind of like research, but also having good metrics in place, being a little more conservative about what we ship to customers. Takes some work. I think

• 54:28Speaker 01

  part of, you know, maybe what we haven't done well at Run LM is communicate our bigger story. And that's something, I guess is ironically something we tend to do more in research, but in the company we have done less of. And something we could probably do better, like telling this kind of new way of thinking about how an AI product is more than just a, you know, a support interface, but an interaction. On the technological side, I mean, we're certainly following what's happening in the research. There's a sense of skepticism, a lot of things like even if my research that we tried, they don't always work. And so we do try stuff that like, yeah, it worked in research, but it didn't work for this thing. And so being able to try and fail at stuff is a big part of, you know, building the product.

• 55:07Lukas Biewald?

  Awesome. Well, this is really fun. Thank you so much for your time.

• 55:10Speaker 01

  Thank you.

• 55:11Lukas Biewald?

  I think this is one of the most practical interviews I've done, so thank you very much.

• 55:15Speaker 01

  It's been fun.

• 55:18Lukas Biewald?

  Thanks so much for listening to this episode of Grading Descent. Please stay tuned for future episodes.