⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Watch the full show on YouTube: https://www.youtube.com/watch?v=Mp0gxUunnWI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Alessio Fanelli?

  Hey everyone, welcome back to another Lit and Space Lining Pod. This is Alessio, partner and CTO at Decibl, and I'm joined by my co-host Wix, founder of SmallAI.

• 0:11Speaker 01

  Hello, hello. And today we are excited to talk about agent evals and leaderboards. Pratik, I think I'm missing out the intro, but Pratik Bhavsar, you're from Galileo Labs and you worked on agent leaderboards this year.

• 0:29Speaker 02

  Hey folks, it's so cool to be here and I'm excited to chat with you on everything about agents today.

• 0:38Speaker 01

  Yeah, awesome. Yeah, so I think, you know, I first got notice of this thing in our Discord when Erin Staples, who I think works with you guys, posted it and said that she was also contributing to it. I think the broad thing is, there's a big shift in evals this year, I think, from just model evals maybe, which quickly gets saturated. I think Almarina also had some controversies. And I think more generally, people want agent evals anyway, where they are evaluating the ability of, you know, models to do real tasks. And instead of like testing knowledge, they're actually, you know, doing two calling

• 1:22Speaker 01

  and utilizing context very well, that kind of stuff. So, yeah, I think one of those things where, this is one of those things where I think there's a lot of fresh area and potential for progress if you can do a credible eval. And the other thing I like to see is an eval that has a cost awareness component. Because I think, you know, people don't really stay very honest with like, oh yeah, like, we spent a lot more tokens, or we spent a lot more money on this agents and we did better. But it's not really clear, like, if the other comparison could also just do the same thing, like, just think harder and also have the same performance.

• 2:03Speaker 02

  Hundred percent, yeah. So, we have been hearing from our customers that they are now moving towards agents and it become very clear to us that last year we did a RAG evaluation because that's what we are discussing more with the customers. But we quickly understood that this is going to be a year of agents. And what we have understood from our previous experiments is that it's not necessary that what you see as top models in let's say L M Arena or other specific evaluation or general evaluations. They might not be also the same ranking for other tasks like agent tick task. So that was our goal, very fundamentally speaking, that we want to get very specific insights apart from just performance.

• 2:47Speaker 02

  And cost was the other thing that comes up a lot with enterprise customers. And that was the second focus essentially. And then there is S L M versus L L M kind of debate. Are small models also catching up? And there is the big one, open source versus private. So those were the dimensions we were interested in.

• 3:04Alessio Fanelli?

  And in the eval itself, the big thing that you highlight is obviously tool selection and tool use. And then single versus multi-turn interaction. Maybe for for people that are just learning about the leaderboard now, just give a high level overview of what are like the key things that you test for.

• 3:21Speaker 02

  Sure. So, um. Agent leaderboard is kind of a eval harness specifically created to test L L M's on hundreds of domains. Uh, we know that there are certain research papers from top companies and research institutes from Berkeley, from Salesforce, from other companies which are. Targeting certain domains for evaluation, but what our goal was that let's take all of these benchmarks because it's possible that certain models are good at certain data sets. There might be issues of leakage also.

• 4:00Speaker 02

  So we wanted to create a as much possible holistic benchmark by taking on data sets, slices on single turn, multi-turn.

• 4:10Speaker 02

  And also on like different kind of edge cases, like there is some information missing for the parameter. The tool itself is missing and long context and all those problems all at once called composite scenario. All those things we wanted to take from this different data sets, we did filtering on it for as hard as possible. Um. And then we created fourteen small data sets uh and then evaluated L M specifically for tool calling. Uh. And we use a used a gallius metric called tool selection quality, which is an L L M judge based metric. Uh powered by G P T four O and it has multiple judges so that we get high performance.

• 4:50Speaker 02

  And that's how we kind of get the score for each sample and we aggregate for a data set and then we do average over the data set. And that's how we get the overall score for the ranking, but yes, customers, uh sorry, readers can also go through particular. Performances or certain data sets as well. Um. Apart from that, uh we were pragmatic about uh selection in a way that we were very interested in how the small models perform as well as. Open source model performs because we know that there can be interesting insights. Historically, what we have seen from our previous initiatives is that, okay, maybe the best G P T or best Claude is the top model.

• 5:34Speaker 02

  But there might be very small gap with the model just below it and then it becomes a cost performance trade off. So that users can kind of select a model which is a slightly worse but also gives them a similar performance. And there can be some hidden models that nobody is talking about because it's all about the latest top releases and not about like the mid tier or slightly lower tier. Uh. Models uh from other providers and sometimes it's just they haven't got enough traction on social. So they might be just missing from the discussion. So we tried to find this kind of models uh and. Get insights as much as possible. So that that was the whole initiative. Uh we launched in February this year.

• 6:17Speaker 02

  Uh previously, we had done rag specific uh evaluations. Now, a small update on the agent leaderboard also, like we are going to do a V two of it. Because we find certain issues with this evaluation. And let's talk about this more at the end of the podcast, like what's next coming. Uh we can dive more into that.

• 6:42Alessio Fanelli?

  Um. What are some of the surprising results? For me, it was Mistral small being the best open source model above four O and deep sea V three. Um. Anything else that that jumped out to you?

• 6:55Speaker 02

  Oh, yes. So first and foremost, I would say that I was expecting GPTs to perform the best. Because they kind of pioneered the tool calling, function calling, support, and they have been working on that for so long. Um. There were some criticism around Gemini models, although they were catching up. And I was very surprised to see when we released this leaderboard. Of course, there has been some shaking of the leaderboard also later on. But when we released, we found that Gemini models are performing really good. And they were extremely cost efficient at the time compared to the second and second model like GPT four at the time.

• 7:34Speaker 02

  And so it became kind of a no brainer that if anybody wants top performance, they can just directly go and go with the Gemini models. And they just not only released like the pro model, they also released the flash version later on and flashlight. But the flashlight is a very interesting model. Uh in in the blog, we have the models when we release this leaderboard. But in the on the original leaderboard, we also have flashlight, which nobody talks about, I think, even now. But I think it's got a very decent score, which is like in the top five models. It's extremely cheap model like it's so dirt cheap that it makes a lot of sense. Only model that is cheaper than it is and performant is from Amazon, uh which also nobody talks about.

• 8:16Speaker 02

  So we found this more interesting models uh that worked surprisingly well. Then another surprise for me was that the reasoning models were not performing well enough. Uh they had certain kind of limitation when we probed into it, like why are they scoring less overall? They were like the O one, the O four, O three, they. We're not performing well specifically for multiple tool call outputs that you have this request. And it requires either the same tool being called multiple time. Or different tools being called all at once and they were outputting only single tool call.

• 8:55Speaker 02

  Um. And because our evaluation required that, okay, all the tools should be outputted at once. That's the instruction as well in the prompt. Uh. It was not following and it was getting low score. So because of certain inherent characteristic of specifically O series model. Uh. We did not find them well to be performing on this kind of data set. We had like multiple data set on the multiple tool output. Uh. And that brought down the score for this model. And of course, they are extremely costly, right? So then it became. Uh. It it becomes straightforward that you should not be using this for tool calling and instead go for the regular models probably. Um.

• 9:36Speaker 02

  Apart from that, on the open source side, I would say that I was very impressed by Mistral's entry into the ecosystem. Um. Three point one just straight away went up and it also very cost efficient. So I think not people might have known at the time and in the launch blog also, there was not much discussion on tool calling, but the model performed very well on our benchmark.

• 9:59Speaker 01

  You mean meta, right? You you said Mistral, but you meant meta?

• 10:02Speaker 02

  No, I meant Mistral only. Yes. Mistral three point one performing really well uh on our benchmark. Another, of of course, the other surprise was that all the Lama models were not performing well on our benchmark. Uh. Three point three and even the Lama four all were really performing extremely poor. So they are like.

• 10:22Speaker 01

  Oh, no.

• 10:24Speaker 02

  And I I that was unbelievable, like it it felt like they were not tuned for tool calling at all in some way. Um. So, I mean, historically, it's all historical, right? There were certain biases and we are like meta one. The Lama one was great, Lama two was great, three also was great, and then suddenly something happened and it went downhill. So these are like surprising results for us, I would say that. The leaderboard Claude three point five was somewhere in the middle. Um. It was really good at certain other things, but for tool calling it was not awesome. But then when we released the leaderboard and just in a week, they launched three point seven. And that went straight up and nobody has beaten it so far. We are not evaluated the latest one.

• 11:10Speaker 02

  Um. Like the Quinn models, uh. Also the latest Mistral models and not the O three pro models which just came out in last few days. You're not evaluated, but it it held the fort for many, many weeks. So that was very impressive release, I would say.

• 11:29Speaker 01

  Awesome. Um. You know, I think uh that's a very good overview, uh very good findings. I think people can use this leaderboard to pick out what base model that they want to use for their agents. Um. And I think that's a really good uh starting point. Uh. I wanted to dive in a little bit on a little bit, a few of the metrics that you chose. Uh. So first of all, did you just, did you just rerun um all these evals, basically the BFCL, the X Lamb? Or do you, um, do you take it off of whatever they publicly report?

• 12:03Speaker 02

  No, we actually took the data slices from the original dataset. And then ran it ourselves. Uh. We got the outputs uh and then we evaluated it with our metrics. So it's fresh. Everything is fresh.

• 12:17Speaker 01

  Okay, everything's fresh. Yeah, and as long as it's like, you know, comparable and um it ba- basically like, do you think that these are, do you have any comments on just like BFCL versus X Lamb versus, you know, your uh your own TSQ thing? Um. How do they, how do they differ in methodology? Like, what should people know about their pros and cons?

• 12:42Speaker 02

  Uh so I feel like they are very similar in nature. It says that uh the data set is from different domains. And certain data sets have multi-turn conversions available while certain do not have it. I do not remember exactly which one has it and which one doesn't, but there were these discrepancies. And we really enjoyed, like I really loved Tau Benchmark also because it had particularly hard evaluation data sets and a lot of models had some struggle over there.

• 13:14Speaker 01

  I think Clay, uh Clay from Sierra actually just released an update to Tau Bench for some reason.

• 13:19Speaker 02

  Um. Oh, he said.

• 13:21Speaker 01

  Yeah, Alessio's best friend now.

• 13:24Speaker 02

  Ah, yeah. So he, he, he released Tau Squared. Tau Squared, wow. That was not a one.

• 13:33Speaker 01

  Dual, it was yesterday. I just happened to see it. Um.

• 13:39Speaker 01

  To coordinate, guide and assist the user.

• 13:44Speaker 01

  Yeah.

• 13:44Speaker 02

  Wonderful.

• 13:45Speaker 01

  So, so Tau Bench, uh I actually didn't know it was from Sierra, uh but it's, it's actually very, very influential. Yeah, can you talk more about Tau Bench?

• 13:57Speaker 02

  Oh yes, so essentially Tau Bench, when it was released, I mean yes, the Tau Bench is for two, uh two domains. It tests agentic capabilities for two domains, which is airline and retail. And their setup is that they have very long prompts in the beginning with a lot of requests. And then you have these tools defined and the tools can interact with certain databases for read and write operations. And it is predefined that once the whole agentic task is done, then they will validate on the final uh edits made on and they will check for the expected value in the data sets.

• 14:39Speaker 02

  Uh in the database, which are basically CSV style of files. Uh and in that fashion, they will validate whether the task was done or not. So it's, it's, it's pretty well defined such that.

• 14:53Speaker 02

  You get a deterministic evaluation all the times. That's the beauty of this and it's it's domain specific, which I think is missing from other evaluation data sets. And that is also one of the themes for our V two. Uh, because I think people don't just come from a general mindset, right? People in the industry come from, let's say, I'm from healthcare. I want to see if my, if this model's work for healthcare or not. People are coming from investment, finance, insurance industry and they want to know that is, are the models ready for my domain or not? Because we all know that things work on the general domain side and when you just put it on the internal data, things start to blow up with the internal terminologies and the cases that happen.

• 15:36Speaker 02

  So I think that was the beauty that they specifically targeted domains. I don't know if they might have expanded domains in this because two was, I felt like less, but it was a decent start. It was more about the methodology rather than the domains I fit at the top.

• 15:50Speaker 02

  It was very hard.

• 15:51Speaker 01

  Yeah, it did. Yeah. So Tau squared, you know, just kind of looking at it now, Tau squared doesn't seem like it expanded domains. Uh, it, it just, they, they expanded the way that they interact with the user.

• 16:03Speaker 02

  Mm hmm.

• 16:04Speaker 01

  So some kind of, uh, oh, they, they added a telecom domain. Okay, there we go.

• 16:09Speaker 02

  I see.

• 16:11Speaker 01

  But I, I, I feel if, it, it seems like the emphasis was a more sort of, um, co-piloting uh, rather than, uh, two call, just, you know, they, they call it dual control. Yeah, we'll, we'll see, we'll see what this is. I, I, it's, it's obviously very new, so. I just, I just happened to see it the other day. Um, okay.

• 16:30Speaker 01

  Yeah, yeah, definitely put that in your, uh, V two.

• 16:33Speaker 02

  Absolutely.

• 16:33Speaker 01

  And it, yeah, I, I also wanted to go back to, um, TSQ, I think that's the other sort of maybe, uh, novel part. Um, I think a lot of people have queries about L M S judge and, um, yeah, just tell us more, like, you know, do you have any calibration errors? Um, how, how, uh, do, do, should more people adopt this kind of approach?

• 16:59Speaker 02

  Indeed, I agree. L L M's have been notorious with their own biases, right? We have seen historically, they have issues with long context and they have issues with certain kind of outputs, they prefer, let's say, lengthy outputs and stuff. Um, what we, so this, our philosophy in the company is that any metric that comes out, which is a Galileo metric, we do extensive research on it. Uh, we create very hard data sets internally, um, and we evaluate it, evaluate them. We annotate them and check for correlations. And only once we feel comfortable, then only we release it and which is usually

• 17:43Speaker 02

  based on the correlation scores like A U R O C scores that we find. If they are high enough, only then we release it, otherwise it goes through a rigorous process of prompt optimization. So we

• 17:57Speaker 02

  allow people to use any L L M with the metric. Uh, and so because L L M's keep on improving, the metric also keeps on improving. And then the prompts is what we kind of uh, the prompt is where all the research goes through and we have seen tremendous value in that. And yes, people would definitely have skepticism, a lot of people, a lot of our customers also have that skepticism. Some are still stuck on blue and root scores, which is surprising because they think that L L M's are not there yet. But we, what we try to do from our side is essentially we try to follow the best practice of prompting. Uh, we try to exploit the reasoning as much as possible from this L L M's.

• 18:39Speaker 02

  And then we use basically self-consistency by using multiple outputs and then taking a majority voting over there. People can, people have flexibility on these things, they can select the L L M's they feel they want to use. If they want to use much more powerful L M, go ahead, you can customize it. If you want to change the prompts, go ahead, customize it. If you want to increase the number of judges, go ahead and customize it. So we allow full flexibility to people to modify this L L M as judges so that they can get the best performance possible. Uh of course, people should always validate with some small annotated data set before they go ahead

• 19:21Speaker 02

  and just make it their default evaluation. So that is always a part of our P O Cs and they they get clarity and confidence only then they will be they are using it. So I feel I have basically I have been a full stack M L engineer previously um and I've worked a lot with annotators. And I have seen a lot of biases and issues with annotators as well. Um it's just so time consuming that you can't afford to kind of wait for annotation to get completed. So it's just much, much better to add more instructions, do some initial validation and get the clarity that, okay, this is where it works. If it doesn't work, yes, go back to annotation. But I feel like it has started working a lot one year back.

• 20:04Speaker 02

  Uh because we I I was running this experiments like we were working with Halosomission Index two years back, uh where we are using our own similar L M S judge methodology. And we had certain issues and there were some correlation issues at the time. But eh last year it got pretty strong and this year I feel it's so strong, I L L M S judges is so strong that you can use it. I've gone through like hundreds of explanations, like our L L M's also generate explanations that users can read to validate whether it actually went to this correct reasoning or not. And I have gone through many explanations in the during this V two experiment that I'm doing right now and also the V one.

• 20:46Speaker 02

  And I did not find any issues. So uh I feel like when we so that's why we released all the data set. People can go and run the experiment with the V two. Also, we are going to release the data sets. People can just take them and run it on our uh cluster. It's free, our uh product is free. So anybody can just sign up and start using it. They can just put the open A I key. And they will get the clarity. Okay, it's working or not. We want to be as open as possible by still being a closed source company. So I feel like it's working and people should definitely pay attention to it.

• 21:21Alessio Fanelli?

  What's the gap between the best prompts and like the worst prompts? Like when you run, as you iterated on the prompting of the L L M as judge, did you see models change in like position in the leaderboard? Or were the results mostly the same, just like different scores?

• 21:39Speaker 02

  So I was not working on fine tuning the L L M as a judge. It is done by a separate team member. And once the metric is ready, only then I start using it. But that was, they went through like a months of rigorous process of prompt tuning. And it's a very structured prompt. It's like an art in itself, as you know, where we have certain sections for certain things on certain instruction. And what we do is like, it's this very simple philosophy. We start with a base prompt with what we think is correct. And then we see where it's failing. And then we try to add those instructions again, defining the failure cases in a general way to the prompt of the L L M as just,

• 22:24Speaker 02

  and see if it's following it or not. And that iteration goes on for multiple times, till we are sure that we are getting the performance that we need. So it's, it's very common sense oriented approach, I would say. And it's just plain English writing with just following the right way of writing prompts.

• 22:43Speaker 02

  And of course, the leaderboard, if you are using different judges with different models, it can, there can be heavy shakeup of the leaderboard also. So that's why we use the best L L M available at that. When we are running, when we are starting or near completion of the experiments, we try to use the best L L M available. So that it should not be a situation where you're trying to evaluate 3.7 with G P T four O mini, right? It will be really risky proposition. Because G P T four O mini can't find. Uh, cannot evaluate the hard outputs that three point seven might be doing correctly, right?

• 23:23Speaker 02

  Um, so, and so for this V two, I'm using the three point seven because it is one of the top reigning models right now. And I've seen really good performance from it. So we essentially swap these models as much as possible when we are running the experiment. Otherwise, it looks way, we have seen situation previously where we were using probably G P T four turbo or something. Or even G P T three point five long back, I'm talking like two years back. Um, and we had to use G P T four turbo at the time because with the three point five, it was something else entirely. Because it can't find errors.

• 24:01Speaker 01

  Um hum.

• 24:05Speaker 01

  Okay, um, awesome, uh, maybe give a little preview of the thinking for V two. And uh, maybe that that will be uh, where we end up, but uh, yeah, I'm just kind of curious.

• 24:17Speaker 02

  Sure, us, what we found issues with V one is. Uh, it has a lot of domains, but it does not have a domain vertical in itself. That, okay, if somebody is interested on how this model perform in, let's say, telecommunications. They can just select filter and then see which, what's the performance from each model? What are the gaps? What's the cost related to them? Um, so first thing first, we are doing domain specific. Um, which means that this are like new data sets from scratch. And why are we doing this? Uh, is of course, the first problem is that they are not available easily. Secondly, it's because, which is, I would say, the main reason is that we don't want leakage at all.

• 25:03Speaker 02

  To be possible in any of the L L M so far. So we want that this are new data sets, so we guarantee that it's going to be performing. As unbiased as possible. Then we saw with we want that the scores are saturating. We see that the best score has already reached point nine five. And so if we are going to evaluate even better models that are coming out, they will be definitely in the range of point nine to one. Which I feel is not very informative for anybody to take a call. Because there can be some noise also in this evaluation. So we want the benchmarks to be harder. Uh and this tool calling to be more complicated and realistic at the same time.

• 25:47Speaker 02

  Uh this DFCL, if you just open and check it, uh you would be like, okay, this is okay. But you don't want to use it for realistic evaluation because it's like some math. Problem statement is given, some simple tool is given, just like let's say some exponential tool, logarithmic tool, multiplication and stuff. And you are just like evaluating on that. So I was not particularly happy with that, but that's the best we could got get at the time. Um and that's why we mixed up all the data sets available at the time to kind of average out the things. Uh because it's very specific, right? If some model is good at math, then it might perform really great for DFCL. But if some model doesn't understand the terminologies of industry. Uh then it will not perform well for Tau.

• 26:32Speaker 02

  So we kind of mixed up the data set during that time. Um so two things, domain and one is increasing the hardness. Third thing is multi-turn evaluation. So this whole evaluation is going to be, let's say, a support bots. The whole theme is evaluating support agents with a lot of tools, like we are having twenty tools. They are going to have five domains, uh some like banking, finance, telecommunications, insurance, healthcare. Um and we'll be having certain categories also like. Coordinate the tools and then kind of adversarial scenarios, customers manipulating. Uh the bot to kind of do certain things.

• 27:19Speaker 02

  And uh yeah, multi-turn, coming to multi-turn, in the multi-turn, what we are doing is that this time, last time it was more like there is an input with one message or multiple messages, and then give one output. That was how we are evaluating. This time it's like a realistic scenario, that we start with a scenario. A scenario has a first message and there is like goals to be accomplished. This scenario is given to a user simulator. Now, user simulator is basically a prompt, which gets the whole history of the chat and it's supposed to generate a new message. And that simulation is ran by GPT four point one mini model.

• 28:01Speaker 02

  Now, there is another simulator which I call the tool simulator. Why we selected tool simulator? Essentially because of ease of running this simulation. As we saw with Tau, Tau has like tool definitions and databases with CSV file kind of thing with read write operations going on. Now, I wanted to have like many tools created. We have like twenty tool per domain and we have like five domains, so it's like hundred tools. I did not have, do not have the time to kind of write this tool with all these databases. So I kind of created a tool simulator again with a GPT four point one mini model, which takes into the input as a tool call.

• 28:46Speaker 02

  So it gets the tool call, it gets the response of the tool call schema, and then it is supposed to give an output. So rather than like interacting with an API or calling certain databases, it is going to generate an output based on the history of the conversation and the tool call information that is given. And the main one that we are evaluating is basically the agent. The agent we are swapping all the LLMs to evaluate. And this agent basically is accomplishing the task given by the user. And then user gives a task, the agent either gives a response or gives the tool call. If it's a tool call, it's given to the tool simulator.

• 29:28Speaker 02

  will give an output and then that output will be used by the agent to give a response to the user. So what we do in this is we for every scenario, we have like at least like five to seven goals to be accomplished. Uh they are pretty complex like Tao and they are all they all need to be accomplished. Otherwise, it's the task is not accomplished. So this time you have to get done many things at once. Which makes the thing hard and it's possible that you might miss out on certain things. And we see that the best models are good at it, but certain models are really struggling on it.

• 30:08Speaker 02

  And they can go into iterations for no reason. Um and so the scores are coming out to be lower, much lower like. The pro- the main problem point nine five is like solved that, okay, the scores are lower. We are getting domain specific insights, uh and then it's a very realistic scenario where you are having multiple turn. And then only finally, when everything is done, then the task is accomplished. So what's how do we evaluate the performance in this case and we use two metrics. Uh we use the older one, the tool selection quality, which is just telling us whether the tool selected was correct or not. But other thing which is super important from a business point of view and practical point of view is that whether the agent completed all the task or not.

• 30:54Speaker 02

  Right? Uh so for that, we have the action completion metric, uh which will check that whether all the requests. That were given by the user, were all of them accomplished and were they accomplished correctly or not? So for every sample we'll get, okay, this is how good it is at tool selection, this is how great. The model was at accomplishing the task. So in that fashion, we'll get the scores for every. Scenario and then we'll aggregate the scores for all the data sets to get the final score. So this time we are kind of focusing more on action completion metric, I would say, which is telling us

• 31:34Speaker 02

  that the user asked for these five things and whether the agent completed all the five things or not. So that's how we are going to evaluate it. We'll also have the tool selection quality metric, but that doesn't tell us the whole story. It just tells a very granular story. It's a debugging metric, but this is like an overall performance metric for the user. And then so essentially a user workflow will be like in the realistic production workflow is that they have this agent working and in certain scenarios, the action completion is lower. So they'll go there and then check what went wrong and also they might see some tool selection issues and they will debug them.

• 32:14Speaker 01

  Got it. Um. Perfect. I think that's a that's a very comprehensive recap. I like the I like the new direction for V two. It sounds like a lot of work. So good luck with that.

• 32:27Speaker 02

  It's difficult to tune prompts to get the right direction of what you want to achieve because you want to be realistic and we the the hard part for me that I've spent a lot of time was generating hard scenarios. Uh, because you take a hard, you take the best model to generate hard scenarios and you then see if the best model can accomplish those tasks or not. Like let's say I generate by three point seven and then check on four point one. And then it's able to accomplish. So I had to kind of do a lot of iterations on like finding where the model fails and how to trigger this kind of scenarios and that took a lot of time. So but I think we have

• 33:10Speaker 02

  we have reached a certain stage where we feel confident that yes, there are realistic and hard scenarios and model is not able to accomplish all the time. So this is going to like reset. This is going to calibrate the expectation of the users of this algorithms. Because our last leader was saying that models are really great at tool calling. So it's like see, but they actually not, right? They are making mistakes. And this is going to recalibrate the expectation of the users that be careful because they are still not perfect in real scenarios.

• 33:44Speaker 01

  Awesome. Okay, I think that's about all we have time for. Anything else Alessio?

• 33:49Alessio Fanelli?

  No, thanks for joining us, Pratik.

• 33:53Speaker 02

  My pleasure, this was awesome.