Metrics Driven Development

How do you systematically measure, optimize, and improve the performance of LLM applications (like those powered by RAG or tool use)? Ragas is an open source effort that has been trying to answer this question comprehensively, and they are promoting a “Metrics Driven Development” approach. Shahul from Ragas joins us to discuss Ragas in this episode, and we dig into specific metrics, the difference between benchmarking models and evaluating LLM apps, generating synthetic test data and more. Sponsors: Assembly AI https://www.assemblyai.com/?utm_source=practicalai&utm_medium=podcast – Turn voice data into summaries with AssemblyAI’s leading Speech AI models. Built by AI experts, their Speech AI models include accurate speech-to-text for voice data (such as calls, virtual meetings, and podcasts), speaker detection, sentiment analysis, chapter detection, PII redaction, and more. Featuring: Shahul Es – GitHub https://github.com/shahules786, LinkedIn https://www.linkedin.com/in/shahules, X https://x.com/shahules786Daniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Ragas https://ragas.io/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard, and I'm really excited to talk to another founder today in the AI space. Today, we have with us Shahul from Ragas. He's one of the co-founders. Welcome, Shahul. How are you doing?

• 1:04Speaker 02

  Hey, I'm doing good. Hi, Daniel. Hey, folks. How are you doing?

• 1:08Daniel Whitenack?

  Yeah, yeah. Well, thanks for joining at a late hour in India. Appreciate that. But yeah, I would love to hear a little bit, maybe for those that aren't familiar with what you're doing, maybe share a little bit about what that is, and also maybe how you came upon the types of problems that you're solving with your current work.

• 1:33Speaker 02

  Sure. That's a very good place to start. So I go Ragas. Ragas is an open service library for evaluating LLM applications. And what we are trying to do with Ragas as an open source library is to provide the developers or engineers who are building LLM applications, the tools and the workflows that are necessary to automate or partially automate the process of evaluation through using different techniques, different methods and techniques that we bring out through Ragas. And how we came up to the idea of Ragas is, basically, me and my co-founder Jitan, we have been working in ML for the past six,

• 2:15Speaker 02

  seven years. And when LLMs came out, we were already working with natural language models. Jitan mostly worked on the inference and infrastructure part of it, and myself, I was working as an applied researcher. So we were, you know, practically, we loaded, we were practically, you know, doing a lot of experiments with it. We were part of different open source initiatives, building L L M's and also using L L M's to build different frameworks at that point. Even Langchain and Lamb index was coming out as, you know, one of the earlier frameworks at that point. This is around early two thousand twenty three. And we were also working with different applications and stuff. And RAG was one of the most popular application that went into production very soon with L L M's.

• 3:00Speaker 02

  And this is one of the first things that L L M's actually opened up as a possibility for, you know, enterprises to build something on top of them, which can save a lot of time and money for them. So we were also building RAGs. Uh, and we initially, we after a couple of experiments with, uh, some clients, we found out that, okay, we are able to build these L L M applications, this RAG application using any L L M. Uh, it has different moving components and going forward, there will be a lot more moving components to any L L M applications. Like it will be something like a common system where there are, as you can see now, it's like not only RAG, now now it's like two use cases.

• 3:41Speaker 02

  There are different versions to, you know, that that constitute a common system kind of stuff. And we thought, okay, you know, why don't we build something, some evaluation metrics, evaluation metrics that can be, uh, used to understand the quality of any RAG application that a that any engineer is building. And we ourselves found out that, you know, going through these, uh, answers or going through these intermediate results manually is not a scalable approach. I could rephrase it to say that it's a very boring thing to do and nobody, uh, is really keen to do it. Uh, this is something everybody will push into someone else's responsibility, but, uh, it is something very important too.

• 4:20Speaker 02

  So we thought of identifying methods or, uh, coming up with solutions that could help developers do the evaluation, but also save them a lot of time while doing, uh, it. So it should not be something like, uh, you know, it it is basically an evaluation of L L M application regardless of. Rag or any agency workflow is basically very tedious process that takes a lot of time if you go through manually. You want to make sure that this tedious process, this manual process that takes a lot of time is cut down into one by ten of the time and it should get you the same insights as you go as manually. So that's how we came uh with the, you know, initial uh initial MVP of progress and we have uh

• 5:03Speaker 02

  we released an open source uh library uh like made of twenty twenty three and since then we have been continuously recreating and we have been getting organic growth and usage from that from there and that's that's what we are trying to do.

• 5:17Daniel Whitenack?

  Yeah, that's awesome. I noticed that you very specifically refer to the evaluation of L L M applications, not necessarily the evaluation of L L Ms. Could you explain like what might be the difference between those mindsets for people that are maybe getting into this? Maybe they're looking at, they have looked at benchmarks, let's say, for like a leaderboard for L L Ms, right? And there's a certain level of, you know, evaluation or benchmarking there. But then um you're talking about the evaluation of L L M application. So could you help us understand kind of some of the differences there?

• 5:58Daniel Whitenack?

  Some of what needs to be thought of at the application level versus at the model level?

• 6:05Speaker 02

  This this goes to the, you know, ideology of A as a consumer uh product as of now because pre L L Ms, nobody, everyone, most of the enterprises or most of the startups who are building uh A powered applications used to have their own models and they will obviously also have a team of writers and this was building and managing these models. So in that spectrum, the building of the L L M itself uh or the, you know, model itself and also the evaluation of the model itself was the responsibility of the researcher, the edge slash engineer slash data scientist who were working on it.

• 6:44Speaker 02

  But now it's known that people who are building an L L M application, so A A applications is not really building their own models. They are consuming models from an external endpoint or even a conference model that are being released and building applications on top of it. Now, this actually forms a wide spectrum where at the left end of the spectrum, there are people building their L L M itself. They have a separate list of targets, loss functions, metrics and benchmarks we evaluate on. And towards the right end of the spectrum, there are people who doesn't really care about how the L L M is built itself.

• 7:21Speaker 02

  They only care about, you know, can this thing which I'm consuming using an A P I do what I'm trying to do here using this A as a technology. Now, when a researcher or an organization like who builds L L M evaluates L L M, they don't really know what is the exact use case for which this A A or L L M is going to be used against. Or the type of data this will be used against. No. When the evaluation is done at the researcher or the L L M builder part of the spectrum, they are limited to the evaluation or, you know, testing of this L L M's capabilities on a general purpose basis.

• 8:01Speaker 02

  They are not really tailoring this evaluation or testing to your application because they don't really have a control over what you are trying to build with it. They can only say that, okay, this is a general capability of the model. And you know that even the general capabilities of the models and benchmarks are highly leaked into the training data and even that is questionable, but that's a different question. This is what a researcher or a L L M builder does at his end. But when it comes to a L L M application builder, he can't really, if someone will say a rag or a tooling agent on top of an L L M, he can't really say that, okay, the L L M builder has said X and X accuracy, so I should get X and X. That would be a wild assumption to make.

• 8:45Speaker 02

  So what we are trying to do here is giving that giving that user a application builder who is at the right end of the spectrum, the power, the required tools to evaluate his application without knowing so much about ML or getting into so much jargon or anything. We wanna make it as easy as possible, so that as intuitive as possible, because most of the people who are building A applications are not from an ML background. They are from a software engineer background or application building, application background, where their their specialty is building and scaling this application, not building and scaling the models. So we are trying to be intuitive as possible. We also want to make sure that we do not take a lot of their time while doing the evaluation.

• 9:28Speaker 02

  So we wanna make sure that their time is valuable and we wanna make sure that we, you know, we do the heavy lifting of this evaluation for them.

• 9:35Daniel Whitenack?

  Yeah, that makes a lot of sense. And and also I'm realizing there's this kind of spectrum that I I was thinking about while you were talking. On the one side, you have kind of data scientists or researchers who are building out benchmarks or metrics for models. On the other end of the spectrum, you have maybe software engineers who are used to writing unit tests or integration tests for their software. And then like what we're really talking about is integration of L L M's into software applications or into certain workflows. You talked a lot about kind of this distinction between L L M benchmarks and evaluating L L M applications.

• 10:18Daniel Whitenack?

  Could you talk a little bit about the differences, maybe there are software engineers in the audience that maybe they're used to writing unit tests and integration tests for their software. Now, as a consumer, like you said, they're integrating some L L M functionality or maybe a reasoning, you know, a chain of reasoning with L L M's into their software. From a practical standpoint, what are the new types of things that they might need to consider that are different maybe from the way that they've unit tested in the past or written tests in the past now that they're working with these L L M workflows?

• 10:56Speaker 02

  Sure, that's a very interesting question. So when it comes to the application builder slash software engineer who is building with AI, as you said, most of them are already familiar with unit tests and integration tests that they, you know, regularly write for their software, softwares. Now, the major difference is that, uh, I also have observed many of the software engineers, uh, who are relating to evaluation, using this analogy, okay, you know, this is testing, I am praying, I will, you know, learn or understand evaluation using the, you know, my understanding of the testing. So, so the thing is that this is a fundamental thing about analogy itself. So now you are using analogy to understand one thing,

• 11:38Speaker 02

  but it could be that these two things can have different physical and chemical, different properties here. For example, now, uh, for when it comes to evaluation and versus traditional software testing, traditional software testing is mostly a discrete space where, you know, you have an input, you have an expected output, and if you give this input, you are supposed to get this expected output from the software. There is no variation in that that satisfies the, if the test. Now, you know, if the logic is basically to add one plus one, you should get, uh, if the whole logic is basically addition, if you give input as one plus one,

• 12:18Speaker 02

  the output should be two. There is no other possibility that exists that would, you know, satisfy that test case is correct. But when it comes to natural language, there is more of a continuous space. Like, if you have a, an L L M that does the same thing, let's say one plus one, uh, or, you know, addition, L L M could even say two in natural language or two in the decimal point numbers, right? So both are actually correct. So here, there are like a continuous space where the output cannot be exactly matched against or you are sorted against, or you should have an understanding, but there is a In this continuous space of outputs that are possible, there's obviously a subset that is actually can be regarded as correct.

• 13:01Speaker 02

  Obviously, if the LLM gives an answer as three in natural language or something, that means wrong, that is out of, out of the way, right? So, out of work and out of work, work again. So what a software engineer should really understand is that when you deal with ML, when you deal with, when you integrate A into your application, you should try to think about it as in a continuous space rather than a very discrete space of or in a black and white manner, you know, it's yes or no, it's, it could be that it is like in the middle of, it could, there are like a lot of space in the gray area here, right? So, uh, you know, that is what a software engineer should understand. And then there is also a non deterministic part of the whole, uh, you know, uh, A thing.

• 13:42Speaker 02

  Basically, if you have a system which is a part which, which is part of core and part of AI, the system is going to be somewhat non deterministic, whatever you do. So this is also something that traditional software is not like that. Traditional software is very deterministic. So you have an input, you will have an output, given that the intermediate states remain to see. So here again, uh, you have this non deterministic thing to take up core for. So that's also one thing. These are the two major differences that exists between software testing, traditional software testing and, you know, this A application or combined A application testing that makes it different from each other.

• 14:17Daniel Whitenack?

  Yeah, super interesting. And I, I know in the software world, there's all of these sort of frameworks for development around, you know, test driven development or data driven development or, or these different things. I noticed in Ragas in your, in your core concepts and your documentation, you talk about metrics driven development for these LLM applications. So for those that are maybe developing LLM applications out there, could you describe a little bit the, the mindset of metrics driven development, what you mean by that? And, and then maybe we can get into a few more of the, uh, the details of, of Ragas itself and, and how you enable that, that framework.

• 15:01Speaker 02

  Yeah, uh, matrix driven development is a highly, you know, uh, it's a concept that we get from the test driven development itself. So the idea, we want to educate developers, like software developers who are more familiar with metrics. Uh, but they are familiar with test, uh, testing to, you know, what we're trying to do here. So as I said, metrics versus test is, is like a matrix is like something that delivers a value or, you know, that helps you understand the performance of some application in a scale of, uh, let's say zero to one or something. Now, whenever you have an application, if you want to upgrade or, you know, change, let's say you are building a whole, a lot of application that consists of

• 15:43Speaker 02

  different agent workflows plus a workflow, et cetera. And now let's say you wanna change one single prompt or one single function call or something, or even the retriever. How would you understand the effect of this change in your pipeline? It's a question that metrics driven development tries to answer. So if you have a metric, if you have a way to objectively quantify or understand the performance of your system before and after this change, you can also understand and analyze the systems, uh, you know, uh, responses or the behavior of the system using these numbers. For example, if you are, you know, switching out the retriever and you, let's say you had a set of metrics that effectively

• 16:27Speaker 02

  quantifies the performance of your system, you could actually switch on the retriever or switch out any function calls or something and then run the Andrea pipeline once again for the given test set and understand the change in those metrics in particular dimensions. And okay, let's say, once you observe this change, you could again easily deep down, okay, you could easily fetch these samples for which the change is reflected in the maximum. And then you could easily analyze and understand the matter, that, that actually uh, helps a lot, helps and reduces a lot of time when it comes to debugging and, you know, testing these applications. So that's the idea of, uh, matrix driven development.

• 17:23Speaker 03

  What's up friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Assembly AI is where you can turn voice data into insights, chapters, transcripts, summaries, and so much more with their leading speech AI models. So Dylan, give me a glimpse into what you're doing with speech AI models at Assembly AI.

• 17:45Speaker 04

  So at Assembly, we're building industry-leading speech AI models for various tasks like speech to text, streaming, speech to text, speech understanding, to help developers easily convert voice data, whether it's live or pre-recorded, into super accurate text. And then to help developers extract a ton of information and metadata around voice data, or even around the text that they just were able to convert from that audio data. So these are things like picking out entities or PII that was spoken in voice files, or summarizing voice and audio data down into custom summaries. It's things like being able to detect how many speakers spoke and who said what, and what the names of different speakers were.

• 18:28Speaker 04

  So we bundle all those things into a super simple API with really great docs that developers can just sign up to for free to start, use the API, build into their apps, and then build these really cool AI apps and products and workflows and automations on top of voice data with.

• 18:44Speaker 03

  I dig it. Okay. Can you take me a little deeper into the opportunity for developers? Because it seems like there's a lot of voice data out there, and there's a lot of trapped value in that voice data.

• 18:55Speaker 04

  There's so much voice data being created on the internet now. Podcasts, videos, phone calls, voice messages, audiobooks, virtual meetings, it's crazy. And you can now transform and understand all this voice and audio data in ways that were not even possible a year, 18 months ago. So what we're seeing with the help of these new AI models that we're creating at Assembly, developers and organizations are just racing to build all these new applications, workflows, automations that leverage the voice data they have either within their organization or... Their product to build really cool new products and services workflows that are just like taking off in the market. So at Assembly, we're building the industry leading

• 19:38Speaker 04

  models for all those different apps and workflows, whether it's speech to text, or speaker diarization, or speech understanding capabilities to summarize voice data, or extract entities from voice data, or mask PII from phone calls for various types of automations that might be built. And we're exposing that through a super simple, super scalable API that's just constantly being updated and constantly getting better. And so we're seeing a crazy amount of developers and companies just built really cool apps and services on top of our API every day. It's really only just getting started, especially with the model updates that we have planned over the second half of the year that are coming out. They're really excited to launch to the developers on our API.

• 20:18Speaker 03

  Okay, constantly updated speech AI models at your fingertips, well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser, or you can get started with a $50 credit at assemblyai.com-practicalai. Again, that's assemblyai.com-practicalai.

• 20:59Daniel Whitenack?

  There's a lot that you've already integrated into Raghas, which are really interesting, and maybe some people have heard of, or some people haven't, like faithfulness and context recall, noise sensitivity, aspect critique, summarization score, and more that I'm not listing. I'm wondering if you could maybe just share a couple of those that you think are maybe most utilized, or maybe interesting from your standpoint, just to give a sense of people, like what types of metrics are we talking about here?

• 21:30Speaker 02

  I think it would be beneficial to answer that in an abstract, you know, for Campbell. If I say uh metrics, the whole concept of uh V E devising or giving uh that out of these metrics is that it's not extremely hard to come up with these metricses. But it's a it is when a software developer or a a application developer thinks about evaluation. Because he's already fam- familiar with, you know, this concept of putting up metrics, designing the right flows and everything. It could take him like two to three days to figure out, you know, the right metricses. So what Ragas does is, if you are an application developer, okay, you are the you are developing this particular type of application. You can come to Ragas and you can go to Ragas metricses.

• 22:15Speaker 02

  And there you will find that in uh enough, you know, workflows, enough uh parts and documentation that can help you navigate according to user's cases, according to requirements, that can land you up in the right metrics that you should use. And the documentation should will also provide an intuitive understanding of how this matrix is calculated underneath. So these are the two value props that we provide there. Uh it's not that a developer can easily, you know, a developer or a data scientist could easily make up these metricses thereon. But sure, if it is mostly if you are a software developer or application, there it could be very hard for you to figure out, you know.

• 22:54Speaker 02

  I have to think around all this stuff. So we are effectively taking that load of of you and providing you the, you know, uh the ways to easily navigate and understand the right metricses to evaluate your application, tell it to your application. Uh now we are also uh expanding, we are also expanding the visual metricses to two use cases and identity workflows. And we are also going to, you know, reformulate the whole, the whole around that particular so that people can easily find the metricses. What are some metricses can be headlamp based or non-headlamp based. These are all, you know, this, you know, whenever a developer comes to Ragas, he obviously asks a lot of questions. Or even think about metrics, he will ask a lot of questions like, for example, okay, what is my use case?

• 23:37Speaker 02

  What are the parts in my application that I want to evaluate? And then what? Should I go for a L L M based metrics which has high correlation with human judgment, but also has its own lack of, its own issues like, you know, non-determinism and everything, or non-L L M based metrics which are traditional, it has less correlation with human agreement or human judgment, but it is, you know, uh, more reproducible. So, all these questions, all these doubts both pop up in mind when, and I think about metrics, we are trying to abstract away all that into Raga's metrics and provide the developer a way to think about metrics. And with metrics, if they're also, when, when they adopt a metric from Raga's,

• 24:18Speaker 02

  they will also have a related list of items or, you know, features that they can use because they adopted the metrics from Raga's. For example, let's say you designed a metric for language, English. Now, let's say you are evaluating for Portuguese or in Spanish, you'll have to, you know, you know, convert it to the other language or whatever, you know, whenever you're evaluating different language, you'll have to do that. Now, in Raga's school, if you're using a metric from Raga's, Raga's could take care of that. Then, then there are also issues like if you're building an L L M based metric, that is, which is the trend here and most of the other, the L L M based metrics, because that has high correlation with human judgment.

• 24:59Speaker 02

  Then, okay, let's say the L L M, you know, you are evaluating the L L M uh, based metrics and you are finding that the L L M based metrics is actually performing, i- it's actually not performing well enough in your case. You should have a way to align these L L M, you know, the people will have different expectations for different met- uh, metrics. For example, let's say you are trying to do something like traitfulness. And traitfulness is basically the mode of hallucination or that is happening in the generated answer, given a context. And if you look at it, different developers from different domain has different strictness towards hallucination.

• 25:40Speaker 02

  I could have a statement like, I have a blue car and versus I have a car and I could, for me, maybe in my domain, I could say that these are two equal signals. But for someone who is working in fintech or something, it might not be equal signal. So, there is also a domain kind of bias when it comes to metrics or in judgments, because developers from the different domain expects different level of strictness and everything. So. Bringing out this alignment with these metrics depending upon your domain is also one extra thing that we are trying to tackle here, which is called, basically, we call it as a metric alignment, that is trying to align large language model judges to your, you know, your specific measurements using the feedbacks that you can give to LLMs.

• 26:25Speaker 02

  It is also an upcoming feature in RAGs.

• 26:27Daniel Whitenack?

  I want to just dig into a couple of those things. So one is like you talk about alignment, but also there's like the idea, it's very important in this case for me to be thinking about data and examples from my domain in terms of how I'm evaluating these. So I'm just looking through, you know, just to make things for concrete for people, looking at your example around answer relevance. And there's data samples in there that have a question, an answer, and then a set of context, and then you can evaluate with RAGs using the metric answer relevancy. So obviously, there's a data component there. And in that component, there's, you know, answers that are there and context that are

• 27:12Daniel Whitenack?

  there. So one question would be like, how much and what type of data will people need to configure such that they can appropriately evaluate their LLM applications? And are there metrics that require sort of data upfront or metrics that maybe are reference free that wouldn't require data upfront? What's the perspective there in terms of kind of, I guess, the cold start? Like if people are starting an LLM application from scratch, they haven't run it in production. What's the data burden and the path towards both getting this data in place and getting the alignment in place?

• 27:51Speaker 02

  So regarding the data, the number, the number of samples people generally use to evaluate is around 100 to 500 when it comes to offsite evaluation. So it really depends on how you formulated your test data. So for example, you could have the same results or say kind of dragging when you have 100 samples at a point, or in many use cases, even 100 may not be enough to, you know, include all that kind of use cases or all the kind of distributions that you see in production. It really depends on the variety of items that you see in production, you know, that you expect in production. Then let's say you have a very niche, you know, use case and a variety is very, you know, uh, very skewed.

• 28:34Speaker 02

  Your test data can be very small, yet still serve the purpose. But if your use case is very broad and you have, you see a wide variety of users coming into your application or a wide variety of request, your test data uh has to be broad enough to include all the different distributions in the data sets itself. So that's about uh the number. And I I did, it's not really about the concrete number, but again, uh basically making sure that you understand what are the different distributions of queries coming into your into your system and also making sure these distribution of queries are also represented in the test data set so that you know when you change a probe or when you change a prompt, which type of queries are being affected.

• 29:17Speaker 02

  You know, it it could be that when you change a particular retriever or particular tool or something, when you start out a particular tool or something, it could be that a a specific set of queries are being affected. So overall pipeline almost never gets affected, but even if you wanna do a huge uh change, where for the small changes, it's mostly a subset of queries, a subset of distribution that gets affected. And it is very important that you are in this series items. Uh so that's about the, you know, uh how to formulate a data set itself for evaluating uh the applications. And regarding the second question of uh metrics and reference free and the with reference metrics, we usually provide both reference free and reference with reference metrics.

• 30:00Speaker 02

  But there is a big shock coming to reference free metrics itself because reference free metrics are basically, you know, you know, we could estimate things there, but there is an error to estimation with reference free metrics. For example, answer relevancy or, you know, you could estimate with some way that if the given answer was correct or something, but if you don't really have the exact thing. Some way or another, it's really impossible to say that, okay, the N L M application arrived at the right thing at the end of the day. So there, what we provide, even with production data, it's very hard to acquire a test dataset. Because again, with production data, that is very, very messy. It's a common thing that people, people, uh, if you ask any researcher or, you know,

• 30:44Speaker 02

  and a guy who has put a mouse into production, mouse into production and the, and the method of acquiring a test dataset. Obviously, they are gonna look at production data, but the way in which test data are being acquired is, there is a long way to go from production data to formulating test, test data, because production data is incredibly messy. Because it's an uncontrolled environment. It's not an human-identified controlled environment. People can come and say anything there and they don't, they have like zero consequences, isn't it correct? So the thing is that production data becomes incredibly messy depending upon the application. If you have, let's say, if your application is serving an internal set of users, like an internal company employee or something, still, you have an amount of control there. But if your application is strictly B to B to C,

• 31:29Speaker 02

  and you are opening up the whole application to, you know, anyone, and anyone using your application. There can be users who are basically trolls, who just like to use your system and, you know, leave out, you know, false feedbacks and stuff. So production data being incredibly messy, it's, there is a long way from going from production data to, you know, and to a good test dataset. So there again, what we are trying to do is providing a way to synthetically create these test, test datasets. Now, this synthetic creation of test datasets will be grounded on things like production data. Uh, then there are your internal documents that should be, you know, taken into account when creating a synthetic test dataset. Because again, you are trying to create a test dataset that's very detailed. Your use case is not a generic one. So.

• 32:15Speaker 02

  There are two points where we ground it, basically the set of internal documents or whatever you have. Then you ground your application and also the production data where the user engages with your application. And when we, when, this is again, uh, uh, one of the upcoming features, uh, the test data generation is already there. Uh, but we are also trying to extend it to a way that we could ground these into the from production data. Basically, we call it seeding from production data. Basically, if a user has been already using Gagas test generation, but he wants to take motivations from production to imitate more, uh, more behavior in the test data set, that is also happening in production.

• 32:56Speaker 02

  Now, there is like a lot, a lot of things, uh, that has to be done to make sure that, to first understand what is happening in production. You know, there can be different distributions, again, as I said, different set of users. We have to understand different set of queries that's coming in, different set of interactions that's happening and everything. And the once then we understand that, we could also have a way to synthesize these type of data points using L L N. Now, uh, the developers, you will not be to annotate these data points, but to verify these data points. Once this data or test data is synthesized, uh, by Gagas, you could export it to a simple, you know, uh, UI that, uh, UI tool.

• 33:37Speaker 02

  And then you could simply, or even Excel sheet, you do, most of the users could export this data to an Excel sheet and basically they go through it. And then once they go through it, they can easily, you know, cut out the bad data points that they think is L L N messed up or something. Because again, we don't really, we can't really guarantee hundred percent efficiency while we're synthesizing these data points. So what we are trying to do is improve the efficiency, let's say if you generate other data points. Our goal is to make sure that all the data points are equally valid and good. But it might not happen like that in every use case. So developer can take elements of his, his or her time and go through this data set manually. It is a fairly quicker process than annotating these, annotating or creating these data sets.

• 34:20Speaker 02

  Uh, because creating these data sets would easy take a day or two, or even a lot of money when you, when you, uh, give it to a human or other human translators. So developer basically can go through this, you know, data points that's been synthesized and then cut out the points that he thinks that is not valuable. So again, that is again, one, one, this, these are actually the real reason which we save a lot of developer time in evaluation. Because again, formally, it's a data set. It's very, it's a very, you know, uh, coca cola, it's a very time consuming, boring process that nobody wants to. And it's mostly falls upon one developer, the thing to do it, and it's a messy process. So these other, you know, type of innovations, what we are trying to bring in the evaluation spaces of

• 35:01Speaker 02

  to save a lot of time, uh, of the developer. And this ideology or this philosophy is why, why we are being getting this organic growth, because we, uh, it is very kind of hard to come up with these kind of solutions. But when we come up with this, the kind of solutions, there are, you know, a lot of developers who are, you know, who wants, who badly wants this. And they are, you know, they are the, uh, people who, uh, motivate us to, uh, continually bring these kind of innovations to the evaluation space.

• 35:29Daniel Whitenack?

  Yeah, that's great. I, I love, uh, I love the innovations around synthetic data use in, in evaluation and utilizing kind of the, the L L M's and these models to help in the evaluation process. But in a way that's, that's validated and still, still fitting with improvement over time. So, yeah, um, as we kind of, I close out here, this has been a fascinating discussion. But, um, maybe just to close out here, what, what are some of the things that you're excited about moving into the next, next six months or so to either explore, or maybe it's things that you see happening in the A I ecosystem more broadly?

• 36:10Daniel Whitenack?

  What, what really excites you about the direction that people are going with their, uh, L L M applications?

• 36:16Speaker 02

  So with L L M applications, uh, with early twenty twenty three or, you know, mid twenty twenty three, RAG was a big thing. And if you remember the time when RAG became popular, there was again a lot of limitations. There were only four K context or, you know, L L M was hallucinating and everything. I think with agent decode tool use cases, we are at the same level as of now, you know? We have been trying to bring more and more, you know, uh, tool use cases. Tool use cases are actually incredibly useful when when it comes to building a whole L L M A application experience because combined with internal knowledge, internal knowledge is what you can infuse with RAG and then taking actions can you i- is

• 36:59Speaker 02

  something that in infused with a tool use case, tool bindings. So if in a tool binding, I'm very excited to see, you know, the next class of models uh performing better on tool use cases. Again, now uh even the these recent uh models uh have have been bringing abstractions or, you know, um being uh using tool binding to facilitate this, but still it's a little bit shaky as of now. But the next class of models, I don't expect it to be better. And then the whole thing, uh RAG plus tool use case. We don't see a lot more enterprises adopting and adopting and using L L M applications at, you know, at different capacities.

• 37:39Speaker 02

  And it's saving a lot of, you know, time and resources for both, you know, people who are at the back of these applications or also people who are interacting with these applications. So that's something I'm really excited uh when it comes to on the very next six months of catalog applications and so on. And also, I think when it comes to the frameworks or libraries that are being built around the applications. I'm seeing more and more better abstractions these days when it comes to, you know, uh these frameworks and libraries. Uh because people now has almost one and a half year of, you know, building with L L M application experience. Now that experience is yielding more and more, you know, understanding of what kind of what is the best abstraction to be

• 38:21Speaker 02

  used to build these applications and everything. And then there is an overall agreement that is coming up uh with, you know, what is, you know, how how to format our goals, how to, you know, build these combo systems and stuff because earlier, uh the first early early twenty twenty three and everything, people are really, really confused on how to build these applications, how to use it and everything now that. The clarity, I can see that the, you know, more clarity is happening at that end too. So at the more, at the model building stage, again, the bottom building spectrum, they are again having more clarity on, you know, things like, you know, the behavior of the models.

• 38:56Speaker 02

  Uh, the type of data either more and more, you know, papers and more and more this is happening at the, you know, data processing and pre-processing stage, you know, what is the type of data needed to train uh the higher quality and best quality elements and stuff. Because we see, we think that uh we have mostly settled on the architecture itself from that point of L L M point of view. Now, the main thing people are, you know, working on this data, again, when it comes to data, people are again now uh mostly, we will finish up the free data that's available on the internet very quickly. And now it's again, it's in the data that has a really good chance of improving these models itself.

• 39:31Speaker 02

  So the idea of models output being used to feed models and improving the models itself on different use cases is itself a fascinating thing regarding the model building stage. When it comes to evaluation, there needs to be more and more innovations happening in the evaluation uh space itself, whether as an open research or as an open approach to, you know, bringing in all the time involved in testing and assessing these applications itself. Uh, because that's one big uh pain point on why the enterprises or, you know, uh big companies could. A big barrier for these big companies to adopt uh L L M application, L L M for A A's in their system.

• 40:14Speaker 02

  Because these people, they have a high responsibility. So when, when with high responsibility, you have to do these testing and evaluations. And there is a very good way of doing it and a great of way of doing it, uh it becomes a pain point. And that's something we are also trying to bring in, you know, we are trying to, you know, formulate all that research, all the innovations that are happening in the evaluation space to come together and build an open source standard for evaluating L L M applications itself. So that there is an agreement between everyone on how to evaluate L L M applications. And that's the long-term mission of the company.

• 40:49Daniel Whitenack?

  Awesome. Yeah, well, thank you, Shahul, for taking time to join and again, joining at a late hour where you're at. I'm really excited about what you're doing with Ragas. And this is a really interesting space that we'll be interested to follow. So thanks for taking time and hope you can have a good rest of your week.

• 41:08Speaker 02

  Sure. Thanks, Daniel. This was fun chatting with you and I hope your users learn something from this conversation.

• 41:16Daniel Whitenack?

  Yeah, thanks. Bye bye.

• 41:25Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire changelog community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.