Towards high-quality (maybe synthetic) datasets

As Argilla puts it: “Data quality is what makes or breaks AI.” However, what exactly does this mean and how can AI team probably collaborate with domain experts towards improved data quality? David Berenstein & Ben Burtenshaw, who are building Argilla & Distilabel at Hugging Face, join us to dig into these topics along with synthetic data generation & AI-generated labeling / feedback. Sponsors: Fly.io https://fly.io/ – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun https://fly.io/speedrun/ to get started in minutes. WorkOS https://workos.com – A platform that gives developers a set of building blocks for quickly adding enterprise-ready features to their application. Add Single Sign-On (Okta, Azure, Google, Microsoft OAuth), sync users from any SCIM directory, HRIS integration, audit trails (SIEM), free magic link sign-in. WorkOS is designed for developers and offers a single, elegant interface that abstracts dozens of enterprise integrations. Learn more and get started at WorkOS.com https://workos.com/Eight Sleep https://www.eightsleep.com/practicalai – Take your sleep and recovery to the next level. Go to eightsleep.com/PRACTICALAI https://www.eightsleep.com/practicalai/ and use the code PRACTICALAI to get $350 off your very own Pod 4 Ultra. You can try it for free for 30 days - but we’re confident you will not want to return it. Once you experience AI-optimized sleep, you’ll wonder how you ever slept without it. Currently shipping to: United States, Canada, United Kingdom, Europe, and Australia. Featuring: Ben Burtenshaw – GitHub https://github.com/burtenshaw, LinkedIn https://www.linkedin.com/in/ben-burtenshaw, X https://x.com/ben_burtenshawDavid Berenstein – GitHub https://github.com/davidberenstein1957, LinkedIn https://www.linkedin.com/in/davidberenstein, X https://x.com/davidberensteiChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Argilla https://docs.argilla.io/latest/Distilabel https://distilabel.argilla.io/latest/Synthetic Data Generator UI https://huggingface.co/spaces/argilla/synthetic-data-generatorHugging Face and Argilla meetups https://lu.ma/argilla-event-calendarUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:44Speaker 01

  Okay, friends, I'm here with Annie Sexton over at Fly. Annie, you know, we use Fly here at Changelog. We love Fly. It is such an awesome platform, and we love building on it. But for those who don't know much about Fly, what's special about building on Fly?

• 1:00Speaker 02

  Fly gives you a lot of flexibility, like a lot of flexibility on multiple fronts. And on top of that, you get, so I've talked a lot about the networking, and that's obviously one thing, but there's various data stores that we partner with that are really easy to use. Actually, one of my favorite partners is Tigress. I can't say enough good things about them when it comes to object storage. I've, I never in my life thought I would have so many opinions about object storage, but I do now. Tigress is a partner of Fly, and it's S3-compatible object storage that basically seems like it's a CDN, but is not. It's basically object storage that's globally distributed without needing to actually set up a CDN at all. It's like automatically distributed around the world.

• 1:45Speaker 02

  And it's also incredibly easy to use and set up, like creating a bucket is literally one command. So it's partners like that that I think are this sort of extra icing on top of Fly that really makes it sort of the platform that has everything that you need.

• 2:00Speaker 01

  So we use Tigress here at Changelog. Are they built on top of Fly? Is this one of those examples of being able to build on Fly?

• 2:07Speaker 02

  Yeah, so Tigress is built on top of Fly's infrastructure, and that's what allows it to be globally distributed. I do have a video on this, but basically the way it works is whenever, like, let's say, a user uploads an asset to a particular bucket. Well, that gets uploaded directly to the region closest to the user. Whereas with a CDN, there's sort of like a centralized place where assets need to get copied to, and then eventually they get sort of trickled out to all of the different global locations. Whereas with Tigress, the moment you upload something, it's available in that region instantly. And then it's eventually cached in all the other regions as well as it's requested. In fact, with Tigress, you don't even have to select which regions things are stored in. You just get these regions for free. And then on top of that, it is so much easier to work with.

• 2:51Speaker 02

  I feel like the way they manage permissions, the way they handle bucket creation, making things public or private, is just so much simpler than other solutions. And the good news is that you don't actually need to change your code if you're already using S3. It's S3 compatible, so like whatever SDK you're using is probably just fine. And all you got to do is update the credentials. So it's super easy.

• 3:14Speaker 01

  Very cool. Thanks, Annie. So Fly has everything you need. Over 3 million applications, including ours, here at Change Log, multiple applications have launched on Fly, boosted by global anti-cust load balancing, zero configuration private networking, hardware isolation, instant wireguard VPN connections, push-button deployments that scale to thousands of instances. It's all there for you right now. Deploy your app in five minutes. Go to fly.io. Again, fly.io.

• 3:56Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, where we're building a Private Securigen AI platform and I'm joined as always by Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Great today, Daniel. How are you? It's a, it's a beautiful, beautiful fall day and you know, a good day to take a walk around the block and think about interesting AI things and Absolutely. clear your mind before getting back into some, some data collaboration, which is what we're gonna talk about today. Chris, I don't know if you remember our conversation. It was, it was just me on that one, but with Bing Sun Chuah,

• 4:41Daniel Whitenack?

  who, who talked about broccoli AI, the type of AI that's healthy for organizations. And in that episode, he, he made a call out to Argilla, which was a big part of, of his solution that he was developing in a, in a particular vertical. I'm really happy today that we have with us Ben Burton Shaw, who is a machine learning engineer at Argilla and also David Berenstein, who is a developer advocate engineer, working on building Argilla and is still a label at Hugging Face. Welcome, David and Ben.

• 5:17Speaker 04

  Thank you. Great to be here.

• 5:19Speaker 05

  Hi. Thanks for having us.

• 5:21Daniel Whitenack?

  Yeah. So like I was saying, I think for some time, maybe if you're coming from a data science perspective, there's been tooling maybe around data that manages, you know, training data sets or evaluation sets, or maybe ML Ops tooling and this sort of thing. And part of that has to do with preparation and curation of data sets. But I, I found interesting, I mentioned the previous conversation with Bing Sun. He talked a lot about collaborating with his sort of subject matter experts in his company around the data sets he was creating for, for text classification. And that's where Argilla came up.

• 6:03Daniel Whitenack?

  So I'm wondering if maybe one of you could talk a little bit at a, at a higher level. When, when you're talking about Data collaboration in the context of the current kind of AI environment, what does that mean generally? And how would you maybe distinguish that from previous generations of tooling in maybe similar or different ways?

• 6:26Speaker 04

  So data collaboration, at least from our point of view, is kind of the collaboration between both domain level experts that really have high domain knowledge, actually know what they're talking about in terms of the data, the inputs and the outputs that the models are supposed to give within their domain. And then you have the data scientists or the AI engineers and this side of the coin that are more technical. They know from a technical point of view, what the models expect and what the model should output. And then the collaboration between them is, yeah, now even higher because nowadays you can actually prompt LLMs with natural language and you actually need to ensure that both the models actually perform well and also the prompts and these kind of things. So the collaboration is even more important nowadays.

• 7:10Speaker 04

  And that's also the case for, still the case for text chat models and these kind of things, which we also support within our job.

• 7:17Daniel Whitenack?

  I guess maybe in the context of, let's say there's a new, a new team that's exploring the adoption of AI technology, maybe for the first time. Maybe they're not coming from that data science background, the sort of heavy ML ops stuff, but maybe they've been excited by this latest wave of AI technologies. How would you go about helping them understand how their own data, the data that they would curate, the data that they would, that they would maybe collaborate on is relevant to and where that fits into the certain workflow? So yeah, imagine someone may be familiar with what you can do with ChatGPT or pasting in certain documents or other things.

• 8:02Daniel Whitenack?

  And now they're kind of wrestling through how to set up their own domain-specific AI workflows in their organization. What would you kind of describe about how their own domain data and how collaborating around that fits into common AI workflows?

• 8:17Speaker 05

  Yeah, so something that I like to think about a lot around this subject is like machine learning textbooks. And they often talk about modeling a problem as well as building a model, right? There's a famous MAMA and MATA cycle. And in that, when you model a problem, you're basically trying to explain and define the problem. So I have articles and I need to know whether they are a positive or negative rating. And I'm describing that problem. And then I'm going to need to describe that problem to a domain expert or an annotator through guidelines. And when I can describe that problem in such a way that the annotator or the domain expert

• 8:59Speaker 05

  answers that question clearly enough, then I know that that's a modeled and clear problem. And it's something that I could then take on to build a model around.

• 9:08Speaker 05

  In simple terms, it makes sense. And so I think when you're going into a new space at generative AI and you're trying to understand your business context around these tools, you can start off by modeling the problem in simple terms by looking at the data and saying, okay, does this label make sense to this articles? If I sort all these articles down by these labels or by this ranking, are these the kinds of things I'm expecting? Starting off at quite low numbers, right? Like single articles and kind of building up to tens of hundreds. And as you do that, you begin to understand and also iterate on the problem and kind of change it and adapt it as you go. And once you've got up to a reasonable scale of the problem, you can then say, all right, this is something that a machine learning model could learn.

• 9:54Daniel Whitenack?

  I guess on that front, maybe one of the big confusions that I've seen floating around these days is the kind of data that's relevant to some of these workflows. So it might be easy for people to think about a labeled data set for a text classification problem, right? Like here's this text coming in, I'm going to label it spam or not spam or in some categories. But I think sometimes a sentiment that I've got very often is Hey, our company has this big file store of documents. And somehow I'm going to fine tune quote unquote

• 10:37Daniel Whitenack?

  a generative model with this blob of documents and then it will perform better for me. And there's two elements of that that are kind of mushy. One is like, well, to what end for what task, right? What are you trying to do? And then also, how you curate that data then really matters. Is this a sentiment that you all are seeing? Or how for this latest wave of models? How would you describe if a company has a bunch of documents? And they're in this situation, they're like, hey, we know we have data. And we know that these models can get better. And maybe we could even create our own private model with our own domain of data.

• 11:19Daniel Whitenack?

  What would you walk them through to explain where to start with that process? And how to start curating their data, maybe in a less general way, but towards some end?

• 11:29Speaker 04

  I think in these scenarios, it's always good to first establish a baseline or a benchmark. Because what we often see is that people come to us or come to like the open source space. They say, okay, we really want to fine tune a model. We really want to do like a super extensive RAG pipeline with all of the bells and whistles included. And then kind of start working on these documents. But what we often see is that they don't even have a baseline to actually start with. So that's normally what we recommend also. Whenever you work with a RAG pipeline, ensure that all of the documents that you index are actually properly indexed, properly chunked. Whenever you actually execute a pipeline and you would store these retrieved documents or these based on the question and the queries

• 12:13Speaker 04

  in RGLA or any other data annotation tool, you can actually have a look at the documents, see if they make sense. See if the retrieval makes sense, but also if the generated output makes sense. And then whenever you have that baseline set up, from there, actually start iterating and then kind of making additions to your pipeline. Shall I add re-ranking potentially to the retrieval if the retriever isn't functioning properly? Shall I add a fine tuned version of the model? Should I switch from the latest Llama model of 3 billion to 7 billion? Or these kind of things. And then from there on, you can actually consider maybe either fine tuning a model if that's actually needed. Or fine tuning one of the retrievers or these kind of things.

• 12:53Chris Benson?

  Is you're saying that as you're speaking from this kind of profound expertise you have. And I think a lot of folks really have trouble just getting started. And like you asked some great questions there. But I think some of those are really tough for someone who's just getting into it. Like which way to go and some of the selections that you would go with that. Could you talk a little bit about the kind of go back over the same thing, but um kind of make up a little workflow? You know, that's kind of hands on on just like you might see this. And this is how I would decide that just for a moment, just so people can kind of grasp kind of the thought process you're going. Cause you kind of described a process, but if you could be a little bit more descriptive about that. I think when I talk to people, once they get going, they kind of go to the next step and go to the next step and go to the next step.

• 13:38Chris Benson?

  But the first four or five big question marks at the beginning, they don't know which one to handle. So.

• 13:44Speaker 05

  I can add some practical steps on to that that I've worked with in the past. Did I say that?

• 13:49Chris Benson?

  That'd be fantastic.

• 13:50Speaker 05

  Yeah. So one thing that you can do that is really straightforward is actually to write down a list of the kinds of questions that you're expecting your system to answer. And you can get that list by speaking to domain experts, or if you are a domain expert, you can write it yourself, right? And it doesn't need to be an extensive exhaustive list. It can be quite a small starting set. You can then take those questions away and start to look at documents or pools and sections of documents from this lake that you potentially have and associate those documents with those questions. And then start to look if a model can answer those questions with those documents.

• 14:31Speaker 05

  In fact, by not even building anything, by starting to use say chat GPT or hugging chat or any of these kind of interfaces. And just seeing this very, very low, simple benchmark to is that feasible?

• 14:45Speaker 05

  Whilst at the same time, starting to ask yourself, can I as a domain expert answer this? And that's kind of where Argilla comes in at the very first step. So you start to put these documents in front of people with those questions and you start to search through those documents and say to people, can you answer this question? Or here's an answer from a model to this question in a very small setting. And you start to get a basic early signals of quality. And from there, you would start to introduce proper retrieval. So you would scale up your, you would take all of your documents, say you had a hundred documents associated with your 10 questions. You put all those hundred documents in an index and iterate over your 10 questions and

• 15:30Speaker 05

  see, okay, are the right documents aligning with the right questions here? Then you start to scale up your documents and make it more and more of a real world situation. You would start to scale up your questions. You could do both of these synthetically. And then if you still started to see positive signals, you could start to scale. And if you started to see negative signals, I'm no longer getting the right documents associated with the right questions. I personally would always start from the simplest levers in a RAG setup. And what I mean there is that you have a number of different things that you can optimize. So you have retrieval, you can optimize it semantically, or you can optimize it in a rule-based retrieval.

• 16:15Speaker 05

  You can optimize the generative model, you can optimize the prompt. And the simplest movers, the simplest levers are the rule-based retrieval, the word search, and then the semantic search. So I would first of all add like a hybrid search. What happens if I make sure that there's an exact match in that document for the word in my query? Does that improve or improve my results? And then I would just move through that process.

• 16:55Speaker 01

  What's up friends? I'm here with a friend of mine, a good friend of mine, Michael Greenwich, CEO and founder of WorkOS. WorkOS is the all-in-one enterprise SSO and a whole lot more solution for everyone, from a brand-new startup to a enterprise and all the AI apps in between. So, Michael, when is too early or too late to begin to think about being enterprise-ready?

• 17:19Speaker 07

  It's not just a single point in time where people make this transition. It occurs at many steps of the business. Enterprise single sign-on, like Sammler, Auth, you usually don't need that until you have users. You're not going to need that when you're getting started. And we call it an enterprise feature, but I think what you'll find is there's companies, when you sell to like a 50-person company, they might want this. Especially if they care about security, they might want that capability in it. So it's more of like SMB features even if they're tech-forward. At WorkOS, we provide a ton of other stuff that we give away for free for people earlier in their lifecycle. We just don't charge you for it. So that AuthKit stuff I mentioned, that identity service, we give that away for free up to a million users, 1 million users.

• 18:01Speaker 07

  And this competes with Auth0 and other platforms that have much, much lower free plans. I'm talking like 10,000, 50,000, like we give you a million free. Because we really want to give developers the best tools and capabilities to build their products faster, you know, and to go to market much, much faster. And where we charge people money for the service is on these enterprise things. If you end up being successful and grow and scale out market, that's where we monetize. And that's also when you're making money as a business. So we really like to align, you know, our incentives across that. So we have people using AuthKit that are brand-new apps, just getting started, companies in Y Combinator, side projects, Hackathon things, you know, things that are not necessarily commercial-focused, but could be someday. They're kind of future-proofing their tech stack by using WorkOS.

• 18:45Speaker 07

  On the other side, we have companies much, much later that are really big, who typically don't like us talking about them, their logos, you know, because they're big, big customers. But they say, "Hey, we tried to build this stuff, or we have some existing technology, but we're sort of unhappy with it." That built it maybe has left. I was talking last week with a company that does over a billion in revenue each year. And their skim connection, the user provisioning was written last summer by an intern. Who's no longer obviously at the company and the thing doesn't really work. And so they're looking for a solution for that. So there's a really wide spectrum. Well, we'll serve companies that are in a, you know, their office is in a coffee shop or their living room, all the way through. They have a, you know, their own building in downtown San Francisco or New York or something. And it's the same platform, same technology, same tools on both sides.

• 19:30Speaker 07

  The volume is obviously different and sometimes the way we support them from a kind of customer support perspective is a little bit different. Their needs are different, but same technology, same platform, just like AWS, right? You can use AWS and pay them $10 a month. You can also pay them $10 million a month, same product.

• 19:44Speaker 01

  Or more, for sure.

• 19:45Speaker 07

  Or more.

• 19:47Speaker 01

  Well, no matter where you're at on your enterprise ready journey, WorkOS has a solution for you. They're trusted by perplexity, copy.ai, Loom, Versal, Indeed, and so many more. You can learn more and check them out at workos.com. That's W O R K O S dot com. Again, workos.com.

• 20:23Daniel Whitenack?

  I'm guessing that you all, you know, the fact that you're supporting all of these use cases on top of Argeela on the data side. Makes me think, like you say, there's so many things to optimize in terms of that rag process. But there's also so many AI workflows that are being thought of, whether that be, you know, code generation or assistance or, um, you know, content generation, information extraction, but then you kind of go beyond that. David, you mentioned text classification and of course there's image use cases. So I'm wondering from you all, do the, at this point, you know, one of the things Chris and I have talked about on the show a bit is, you know,

• 21:06Daniel Whitenack?

  we're still big proponents and, you know, believe that in enterprises, a lot of times. There is a lot of mixing of, you know, rule-based systems and more kind of traditional, I guess, if you want to think about it that way, machine learning and smaller models. And then bringing in these larger gen AI models as kind of orchestrators or, you know, query layer things. And that's a story we've been kind of telling, but I think it's interesting that we have both of you here in the sense that like you really, I'm sure there's certain things that you don't or can't track about what you're doing, but just even anecdotally out of the users that you're supporting on on Argilla, what have you seen in terms

• 21:49Daniel Whitenack?

  of what, what is the mix between those, you know, using Argilla for this sort of maybe what people would consider traditional data science type of models like text classification and or image classification type of things and these maybe newer workflows like RAG and and other things? How do you see that balance? And do you see people using both or one or the other? Yeah. Any, any insights there?

• 22:15Speaker 04

  I think we recently had this company from from Germany, Allemains, over at one of our meetups that we that we host. And they had an interesting use case where they collaborated with this healthcare insurance platform in in Germany. And one of the things that you see with large language models is that these, yeah, large language models can't really produce German language properly, mostly trained on English text. And that was also one of, one of their issues. And what they did was actually a huge classification and and generation pipeline, combining a lot of these techniques where they would initially get an email in that they would classify to a certain category, then based on the

• 22:58Speaker 04

  category, they would kind of define what kind of email template, what kind of prompt template they would use. Then based on the prompt template, they would kind of start generating and composing one of these response emails that you would expect for like a customer query requests coming in for the healthcare insurance companies. And then in order to actually ensure that the formatting and phrasing and the German language was applied properly, they would then, based on that prompt, regenerate the email once more. So prompt an LLM to kind of improve the quality of the initial proposed output. And then after all of these different steps of classification of retrieval, augmented generation of an initial

• 23:39Speaker 04

  like generation and a regeneration, they would then end up with their eventual output. So what we see is that all of these techniques are normally combined. And also a thing that we are strong believers in is that whenever there is a smaller model or an easier approach applicable, why not go for that instead of using one of these super big large language models? So if you can just classify, is this relevant or is this not relevant? And based on that, actually decide what to do, that makes a lot of sense. And also, one of the interesting things that I've seen, one of these open source platforms, haystack out there using is also this query classification pipeline,

• 24:23Speaker 04

  where they would classify incoming queries as either a key terminology search, a question query, or actually a phrase for an LLM to actually start prompting LLM. And based on that, actually redirect all of their queries to the correct model. And that's also an interesting approach that we've seen.

• 24:43Chris Benson?

  Quick follow up on that, and it's just something I wanted to draw out because we've drawn it out across some other episodes a bit. You were just making a recommendation, kind of go for the smaller model versus the larger model. Could you, for people trying to follow, and you know, there's the divergent mindsets, could you take just a second and say why you would advocate for that? What the benefit, what the virtue is in the context of everything else?

• 25:08Speaker 04

  I would say like smaller models are generally hostable by yourself, so it's more private.

• 25:15Speaker 04

  Smaller models, they are more cost efficient. Smaller models can also be fine tuned easier to your specific use case. So even what we what we see a lot of people coming to us about is actually fine tuning L L Ms. But even the the big companies out there with huge huge amounts of money and resources and dedicated research teams still have like difficulties for on on fine tuning L L Ms. So whenever you instead of within your retriever augmented generation pipeline and fine tune like the L L M for the generation part, you can actually choose to to fine tune one of these retriever models that that you can actually fine tune on consumer grade hardware. You can actually fine tune it very easily on any arbitrary data scientist developer device.

• 26:00Speaker 04

  And then instead of like having to deploy anything on one of the cloud providers, you you can start with that. And for in in a similar reasoning for for a RAG pipeline, whenever you provide an L M with with garbage within such a retriever management generation pipeline, you actually also ensure that there's yeah, less relevant content and the the output of the L M L M is also going to be worse.

• 26:24Daniel Whitenack?

  Yeah, I've seen a lot of cases where I think it was Travis Fisher who was on the show, he advocated for this hierarchy of how you should approach these problems. And there's like, you know, maybe seven things on his hierarchy that you should try before fine tuning. And I think in a lot of cases, I've seen people maybe jump to that. They're like, oh, I forget which one of you said this, but, you know, this naive rag approach didn't get me quite there. So now I need to fine tune, when in reality, there's sort of a huge number of things in between those those two places. And you might end up just getting a worse performing model, depending on how you go about the fine tune. One of the things, David, you kind of walked through these different, the the example

• 27:08Daniel Whitenack?

  of the specific company that had these workflows that involved a variety of different operations, which I assume, you know, to Ben, you mentioning earlier, starting with a test set and that sort of thing and how to think about the tasks. I'm wondering if you can. Specifically now talk just a little bit about Argilla. Specifically, people might be familiar generally with like data annotation. They might be familiar, you know, maybe even with how to upload some data to quote, "fine tune" some of these models in an API sense, or maybe even in a more advanced way with Qlora or something like that. But could you take a minute and just talk through kind of

• 27:52Daniel Whitenack?

  Argilla's approach to data annotation and data collaboration? And like, it's kind of hard on a podcast because we don't have a visual to show for people, but as best you can, you know, help people to imagine, you know, if I'm using Argilla to do data collaboration, what does that look like in terms of what I would set up? And what, who's involved? What actions are they doing? That sort of thing?

• 28:16Speaker 05

  Argilla, there's two sides to it, right? So there's a Python SDK, which is intended for the AI machine learning engineer. And there's a UI, which is intended for your domain expert. In reality, the engineers often also use the UI and you kind of iterate on that as you would, because it gives you a representation of your task. But there's these two sides. The UI is kind of lightweight. It can be deployed in a Docker container or on Hugging Face Spaces. It's really easy to spin up. And the SDK is really about describing a feedback task and describing the kind of information that you want. So you use Python classes to construct your dataset settings.

• 29:01Speaker 05

  You'll say, okay, my fields are a piece of text, a chat, or an image. And the questions are a text question, so like some kind of feedback, a comment, for example. A label question, so positive or negative labels, for example. A rating, say between one and five, or a ranking. So example one is better than example two, and you can rank a set of examples. And with that definition of a feedback task, you can create that on your server in your UI. And then you can push what we call records, your samples, into that data set. And then they'll be shown within the UI.

• 29:47Speaker 05

  And your annotator can see all of the questions. They'll have nice descriptions that were defined in the SDK. They can tweak and kind of change those as well if you need in the UI, because that's a little bit easier. You can distribute the task between a team. So you can say, okay, this record will be accepted once we have at least two reviews of it. You can say that some questions are required and some aren't. And they can skip through some of the questions. The UI has loads of keyboard shortcuts like with numbers and arrows and return. So you can move through it like really fast. It's kind of optimized for that. And different sort of screen sizes. One thing we're starting to see is that as LLMs get really good at quite long documents, that some of the stuff that they're dealing with is like a multi-page document

• 30:32Speaker 05

  or a really detailed image and then a chat conversation. And then we want like a comment and a ranking question. So it's just like a lot of information in the screen. So the UI kind of scales a bit like an IDE. Like so you can drag it around to give yourself enough width to see all this stuff. And then you can move through it in a reasonably efficient way with the keyboard shortcuts and stuff.

• 30:54Daniel Whitenack?

  Interesting. And what do you see as kind of the backgrounds of the roles of people that are using this tool? Because one of the interesting things from my perspective, especially with this kind of latest wave is there's maybe less data scientists, kind of AI people that their background and more software engineers and just, you know, non-technical domain experts. So how do you kind of think about the roles within that? And what are you seeing in terms of who's using the system?

• 31:29Speaker 04

  For us, I think it's, yeah, the, from, from the SDK Python side, it's really still developers. And then from the UI side, it's like anyone in, in the team that needs to have some data labeled with domain knowledge. Often, these are also going to be like the, the AI experts. And one of the cool things is that whenever an AI expert actually sets up a dataset besides these fields and questions, they can actually come up with some interesting features that they can add on top of the dataset. They are also able to add like semantic search, like attach records or semantic representation of the records to one of the records, which actually enables the, the users within the UI to label way more efficiently.

• 32:11Speaker 04

  So for example, if someone sees a very representative example of something that's that bad within their dataset, they can do semantic search, find the most similar documents, and then continue with, with the labeling on top of that. Besides that, you can also, for example, filter based on model uncertainties. So let's say that your model is very uncertain about an initial prediction that you have within your UI. And it's really interesting for the domain expert or for the data scientist to, to go and, and have a look at that specific record or this like range of uncertainties. And then based on that, the labeling or like the, the data curation or whatever you would like to call it, becomes way more engaging and way, way more interesting.

• 32:55Speaker 04

  And on top of that, another thing that we are starting to explore is actually using this AI feedback and synthetic data within our JLA as well. And, and that's actually one of the other products that we're working on and it's called Tisci Label. So nowadays, what you can do with L L M's is also actually use L L M's to evaluate like questions, for example, to evaluate whether something is labeled A, B, or C. Whether something is a good or a bad response and you see all kinds of like tools, open source tools out there. And that's also a thing that we are looking at for integrating with the UI where instead of doing this more from a data science SDK perspective, users without any technical knowledge would actually be able to tweak these guidelines

• 33:40Speaker 04

  that Ben highlighted earlier and then say, okay, maybe instead of taking this into account, you should focus a bit more on like the harm that potentially is within your data or the risks that are within your data. And then you would be able to prompt an L M once again to kind of label your data and then you wouldn't directly need the Python SDK anymore.

• 34:02Chris Benson?

  I was thinking about as you were describing that I work in a large organization and we certainly have a lot of domain experts in the organization I work at that are either non-technical or semi-technical. And as users, they will sometimes find it intimidating, you know, kind of getting into all this as they're starting a project. Could you talk a little bit about what it's like for a non-technical person to sit down with our Gila and start to work in a productive way? What, what is that experience like for them? Because it's one thing, like the, the technical people kind of just know they dive into it. They're going to use the SDK, they've used other SDKs, but there can be a bit of handholding for people who are not used to that. Could you describe the, the user experience for that non-technical

• 34:45Chris Benson?

  subject matter expert coming in and what labeling is like and just kind of paint a picture of words on what their, what their experience might be like?

• 34:54Speaker 05

  Yeah, I mean, one thing I guess I'd start off by saying is that our Gila is kind of the latest iteration of a problem that has existed for a long time in machine learning and data science, right? About collecting feedback from domain experts. And it's kind of gone through spreadsheets and various other tools that were substandard and really bad user experiences where domain experts were, were asked for information, that information was extracted, and then models have been trained really poorly on that information. So as a field, we kind of know that it's something that we have to take really seriously. And, and that's kind of what our Gila is built on top of, right? That's part of our DNA as a product was like optimizing.

• 35:41Speaker 05

  The feedback process as a user experience problem. And so when the user sits down to use Argilla, the intention is that all of the information should be right there in front of them, inside their single like record view. So what that means is they've got a set of guidelines that are edited in Markdown. They can contain images, links to various pages or other external documents if they need, and they can just kind of scroll through that. It's always there, it's always available. They've then also got like basic metrics, so they'll know how many records they've got left, how many they've labelled. They can view their kind of team status and see what's going on. And then on the left, they have their fields which they can scroll through,

• 36:25Speaker 05

  and on the right, they'll have a set of questions. As I said, they can move through these in keyboard shortcuts, and they can switch the view so that they can scroll infinitely, or they can move into a kind of page swiping. Which, yeah, if you're looking at really small records with like a couple of lines and you're just assigning a simple label to, you can do that in bulk. So, as we said, you could do say semantic search, give me all the records that are similar to this one and I'll bulk label those. Or you could search for terms inside those records and you could bulk label those. And then, once you're finished, you'll know about it.

• 37:01Speaker 04

  And one of the interesting things that I've done personally quite often is sit together with the domain experts and their AI engineers to kind of walk them through how to configure Argilla most usefully for both of them. And then the domain experts come with a lot of things to the table, like, I want to see this specific representation. What if we could do this? What if we could do that? Then the AI engineers think about like the data side of things. Is this possible from our point of view, from our side? And then me as a mediator, so to say, can I make the most out of the Argilla configuration? And that's also how we see this collaboration process going, where domain experts really work together also with AI engineers,

• 37:44Speaker 04

  because AI engineers or machine learning engineers actually know what's possible from the data, what it means to to get high quality data for fine tuning a model. Because whenever a domain engineer comes up with something that's useful for for them in terms of labeling, doesn't mean necessarily that it's actually proper data that's going to come out of there in terms of fine tuning a model. And that's also a part of, I guess, the collaboration that we're talking about.

• 38:25Speaker 01

  What's up, friends? I've got something exciting to share with you today. A sleep technology that's pushing the boundaries of what's possible in our bedrooms. Let me introduce you to Eight Sleep and their cutting edge Pod4Ultra. I haven't gotten mine yet, but it's on its way. I'm literally counting the days. So what exactly is the Pod4Ultra? Imagine a high-tech mattress cover that you can easily add to any bed. But this isn't just any cover. It is packed with sensors, heating and cooling elements, and it's all controlled by sophisticated AI algorithms. It's like having a sleep lab, a smart thermostat, and a personal sleep coach all rolled into a single device.

• 39:07Speaker 01

  It uses a network of sensors to track a wide array of biometrics while you sleep, sleep stages, heart rate variability, respiratory rate, temperature, and more. It uses precision temperature control to regulate your body's sleep cycles. It can cool you down to a chilly 55 degrees Fahrenheit or warm you up to a good, nice, solid temperature of 110 Fahrenheit. And it does this separately for each side of the bed. This means you and your partner can have your own ideal sleep temperatures. But the really cool part is that the pod uses AI and it uses machine learning to learn your sleep patterns over time. It uses this data to automatically adjust the temperature of your bed throughout the night

• 39:52Speaker 01

  according to your body's preferences. Instead of just giving you some stats, it understands them and it does something about it. Your bed literally gets smarter as you sleep over time. And all this functionality is accessible through a comprehensive mobile app. You get sleep analytics, trends over time, and you even get a daily sleep fitness score. Now, I don't have mine yet. It is on its way. Thanks to our friends, Over8 Sleep, and I'm literally counting the days I get it, because I love this stuff. But if you're ready to take your sleep and your recovery to the next level, head over to 8sleep.com-practicalai and use our code PRACTICALAI to get 350 bucks off your very own Pod4Ultra.

• 40:34Speaker 01

  And you can try it free for 30 days. I don't think you want to send it back, but you can if you want to. They're currently shipping to the US, Canada, United Kingdom, Europe, and Australia. Again, 8sleep.com-practicalai.

• 41:09Daniel Whitenack?

  I want to maybe double click on something that David you just said in sort of passing, which I think is quite significant. And I don't know if some people might have caught it, but when you were talking about the still label, you also talked about AI feedback. So AI feedback and synthetic data. So I'd love to get into those topics a little bit. Maybe first, coming from the AI feedback side, I think this is super interesting because, you know, Ben, you talked about how this is a kind of more general problem that people have been. Have been looking at in various ways from various perspectives for a long time in terms of this data collaboration labeling piece.

• 41:49Daniel Whitenack?

  But there is this kind of very interesting element now where we have the ability to utilize these very powerful, maybe general purpose, instruction following type of models to actually act as labelers within the system, or at least generate drafts of labels, or feedback, or even preferences and scores and all of those sorts of things. So I'm wondering if one of you could speak to that. Some people might find this kind of strange that we're kind of giving feedback to AI systems with AI systems, which seems circular and maybe like, why would that work?

• 42:33Daniel Whitenack?

  Or just sort of, maybe that's kind of produces some weird feelings for people. But I think it is a significant thing that is happening. And so, yeah, either of you would want to kind of dive into that. What does it specifically mean in AI feedback? And how are you seeing that being used most productively?

• 42:51Speaker 05

  So when we create a data set, either manually or with AI feedback or AI generation, we have all the information there to understand the problem. We have a set of guidelines, we have a set of labels, definitions of those labels, with documents and definitions of those documents. We give those to a manual annotator, or we'll go out and collect those documents and we'll give those documents to the manual annotator. And we're trying to describe that problem so that the person understands it to create the data. We can essentially take all of the same resources and give those to an LLM and get the LLM to perform the same steps. So there's two parts to that. There's a generative part where the LLM can generate documents.

• 43:32Speaker 05

  So let's say we've got 100 documents in our data set, but we want 10,000. We can say, generate a document like this one, but and add variation on top of that. And we can iter- fan out our data set, our documents from 100 to 10,000. We could then take those same documents, or a pool of documents from elsewhere, and we could get feedback on that. So that could be qualitative feedback. Tell me which of these documents are relevant to this task. Tell me which of these documents are of a high quality, are concise, are detailed, these kind of attributes, so we could filter down our large dataset or our generated dataset to the best documents.

• 44:16Speaker 05

  We could also add labels, so we could say, "Tell me which of these documents relates to my business use case or not," these kind of things. Apply topics to these documents. And then we can, in doing so, create a classification dataset, right, from those labels. Or we could, in one example, take a set of documents and use a generative model to generate questions or queries about those documents. And we could use that to create a Q and A dataset or a retrieval dataset where we generate search queries based on documents.

• 44:49Chris Benson?

  When you're doing that and you're generating the datasets with another, you know, model, how much do you have to worry about hallucination playing into that? It sounds like you have a good process for kind of trying to catch it there, but is that a small issue? Is that a larger issue? Any guidance on that?

• 45:08Speaker 05

  That's one of the main issues, definitely. Like, it is probably the main issue. And so, really, it's about both sides of that process that I described, that generating side and that evaluating side. So you get the large language models to do as much as possible to expose hallucination by evaluating themselves. And typically, you're getting larger models to evaluate, so that they're a more performant model and they should hallucinate less. The task of identifying hallucinations is not the same as generating a document. So typically, LLMs are better at identifying hallucinations and nonsense if you give them the context, then they are not generating it.

• 45:50Speaker 05

  And so, you combine that within a pipeline, and then you would take that to a domain expert in a tool like Argilla. And so that's really why we have these two tools, right, Distilabel and Argilla, because kind of without Argilla, Distilabel would suffer from a lot of those problems, right?

• 46:10Daniel Whitenack?

  Yeah, and I guess that brings us to the second tool, the Distill Label, which I know has some to do with this synthetic data piece as well. And I'm really intrigued to hear about this because I also see some of what you have on the documentation about what are people building with Distill Label. I do note a couple of data sets like the Open Hermes data set, the Intel Orca DPO data set. These are data sets that have been part of the lineage of models that I've found very, very useful. So first off, thanks for building tooling that's created really useful models in my own life.

• 46:52Daniel Whitenack?

  But beyond that, yeah, David, do you want to go into a little bit about what Distill Label is and maybe even tie into some of those things and how it's proven to be a useful piece of the process in creating some of those models?

• 47:07Speaker 04

  I think, yeah, the idea of Distill Label kind of started to done like half a year ago, more or less, or maybe a year ago, where we saw these initial new models coming out like Alpaca and Dolly from Databricks, Alpaca from Stanford, where there were like data sets being generated with OpenAI frontier models, being evaluated with OpenAI frontier models, and then published and actually used for fine tuning one of these models. So apparently there were research groups or companies kind of investing time in this. But what we also saw is when we would kind of upload these data sets into Arjilla, actually start looking at the data that there were a lot of flaws within there.

• 47:51Speaker 04

  And then whenever like Ultra Feedback, which is one of these specific papers that really started to scale this synthetic data and AI feedback concept came out, we thought, okay, maybe it's worth to look into like a package that can actually help us facilitate creating data sets that we can then eventually fine tune within Arjilla. And that's when we started to work on the initial version of Distill Label. So it's kind of like application framework, like Lama Index or LangChain, if you're familiar with those, but then specifically focused on synthetic data generation and AI feedback. So what we try to do is organize everything into this pipelining structure.

• 48:34Speaker 04

  Where you have either steps that are about basic data operations, tasks that are about prompt templates or prompting and from templates, you can think about either providing feedback, maybe rewriting some initial input that you provide to that prompt template, or maybe like ranking or like generating from from scratch or these kind of things. And then these tasks are actually executed by L L M's and these are then all like fit together within within a pipelining structure. And the thing for these tasks is that nowadays, we actually look at all of the most recent research implementations or most recent research papers, and we try to implement them whenever they come out and are actually relevant for synthetic data generation.

• 49:20Speaker 04

  So you really go from that kind of finicky prompt engineering, so to say, to well evaluated prompts that we've implemented. And the nice thing about our pipeline structure is also that we run everything asynchronously. So there's multiple like L L M executions being done at once, which will really speed up your pipeline. And on top of that, we also cache all of the intermediary results. So as you can imagine, calling the OpenAI API can be quite costly. And whenever you run a pipeline, there can go, a lot of things can go wrong. But whenever you actually re-run our pipelines within this label, you actually have these cached results already there. So you would avoid kind of incurring additional costs whenever something within the pipeline breaks.

• 50:06Daniel Whitenack?

  Yeah, that's that's awesome. And I know that one element of this is the kind of creation of synthetic data for further fine tuning L L M's, you know, to increase performance or or maybe to some sort of alignment goal or something like that. But also, I know from working with a lot of, you know, healthcare companies, manufacturers, others. That are more security, privacy conscious in my day job. Part of the pitch around synthetic data is maybe also creating data sets that might not kind of poison LLMs with a bunch of your own

• 50:48Daniel Whitenack?

  sort of private information that could be sort of exposed as part of an answer that someone prompts the model in some way and this data is embedded in the in the data set and all of that. So yeah, I would definitely encourage people to check out this still label. And you said it's been around for half a year. So how have you have you seen the kind of usage and adoption so far?

• 51:11Speaker 04

  The usage and adoption has been quite, quite good in terms of the number of data sets that have been released. So you mentioned the Intel ORCA DPO dataset, which was an example use case of how we were initially using it, where we had this original data set that had been labeled by by Intel employees with preferences of what would be like the preferred response to a given prompt. And we actually used this label to kind of clean that based on asking, prompting LLMs ourselves to reevaluate these chosen rejected pairs within the original data set, filtering out all of the ambiguity. So sometimes the LLM wouldn't align with the original chosen rejected pair.

• 51:56Speaker 04

  And based on that, we were actually able to scale down the data set by 50%, leading to less training time and also leading to a higher performing model. And that was one of the really famous examples that kind of inspired some, some people within the open source community to actually start looking at this label, to start using this label to generate data sets. There are some Hugging Face teams that actually have been generating millions and millions of rows of synthetic data using this label. And that's pretty cool to see that that people are actually using it at scale. And besides that, there's also these like smaller companies, so to say, but like Alamine, the German consultancy, the German startup that I mentioned before, using it

• 52:41Speaker 04

  to also rewrite and resynthesize emails within like actual production use cases.

• 52:48Chris Benson?

  That's really fascinating. You guys are pushing the state of the art in a big way. With with the work that you've done in Distilabel and Argilla, where do you think things are going? Like when you're when you're kind of end of whatever your task is of the day and you're kind of just letting your mind wander and thinking about the future, where do each of y'all go in terms of what you think's going to happen, what you're excited about, what you're hoping will happen, what you might be working on in a few months or maybe a year or two? What are your thoughts?

• 53:21Speaker 05

  I suppose for me, it's about two main things. And the first would be modalities. So moving out of text and into image and audio and video, and also kind of UX environments. So that maybe in Argilla, but also in Distilabel, that we can generate synthetic data sets in different modalities and that we can review those. And that's a necessity and something that we're already working on and we've already got features around, but we've got kind of more coming. And then the second one, which I suppose is a bit more far fetched, and that's a bit more about kind of tightening the loop between the various applications. So between Distilabel, Argilla, and the application that you're building, so that you can deal

• 54:05Speaker 05

  with feedback as it's coming from your domain expert that's using your application and potentially Argilla at the same time. So we can kind of synthesize on top of that to evaluate that feedback that we're getting and generate based on that feedback. So we can add that into Argilla and then we can respond to that synthetic generation, that synthetic data. And then we can use that to train our model, this kind of tight loop between the end user, the application, and our feedback.

• 54:33Speaker 04

  Yeah, and for for me, it kind of aligns with what you mentioned before, man, like the multimodality, smaller, more efficient models, things that can actually run on a device. I've been playing around with this app this morning that you can actually load a local LLM into, like a small, smaller like QN or a Llama model from Meta. And it actually runs on an iPhone 13, which is really cool. It's private, it runs quite quickly. And that thing that I've been wanting to play around with is these speech to speech models, where you can actually have real time speech to speech. I'm currently learning Spanish at the moment. And one of the kind of difficult things there is not being secure enough to actually talk to to people out on the streets and and these kind of things.

• 55:17Speaker 04

  So whenever you would be able to kind of practice that at home privately on your device, kind of talk some Spanish into an LLM, get some Spanish back, maybe some corrections in English, these kind of scenarios are super cool for me whenever they would be able to

• 55:31Daniel Whitenack?

  Moi Bueno.

• 55:32Speaker 04

  Come true.

• 55:33Daniel Whitenack?

  Yeah, this is Moi Bueno. And yeah, I've been been really, really excited to to talk to you both and would love to have you both back on the show sometime to update on those things. Thank you for what you all are doing, both in in terms of tooling and, you know, our G L and hugging face more broadly in terms of of how you're driving things forward in the community and especially the the open source side. So thank you both. Thank you for taking time to to talk with us and hope to talk again soon.

• 56:04Speaker 04

  Yeah, thank you. And thanks for having us.

• 56:07Daniel Whitenack?

  Thank you.

• 56:15Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire changelog community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.