Trends in data labeling

Any AI play that lacks an underlying data strategy is doomed to fail, and a big part of any data strategy is labeling. Michael, from Label Studio, joins us in this episode to discuss how the industry’s perception of data labeling is shifting. We cover open source tooling, validating labels, and integrating ML/AI models in the labeling loop. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. SignalWire https://signalwire.com – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com https://signalwire.com and use code SHIPIT for $25 in developer credit. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFeaturing: Michael Malyuk – LinkedIn https://www.linkedin.com/in/mikhail-malyuk, X https://x.com/michaelmalyukChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Label Studio https://labelstud.io/Heartex https://heartex.com/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We put a lot of effort into making sure that the actual interface inside Label Studio is very intuitive and easy to follow. Because as a data scientist, we always focus on ourselves, but we also put specific focus on a non-tech savvy users. Because those users can actually have a lot of very specific domain interesting knowledge. And you want to be capturing that, but if you build a tool that is very complicated, they would spend a lot of time figuring out how to use that. So we are trying to make it as simple as possible, but yet powerful.

• 0:37Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:21Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 1:56Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:21Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International, and I'm joined, as always, by my co-host, Chris Benson, who is a strategist at Lockheed Martin. How you doing, Chris?

• 2:37Chris Benson?

  Doing very well. Glad to be here.

• 2:39Daniel Whitenack?

  Yeah, you were just telling me before we started recording that you had an animal rescue emergency right before.

• 2:45Chris Benson?

  I did, and so I'm just glad to show up. I'm glad to be here today. It's a good day for me.

• 2:51Daniel Whitenack?

  Yeah, yeah, for sure. And you know what? Even more good news. So I don't know if you remember 2019, Chris.

• 2:58Chris Benson?

  That's like pre-pandemic, isn't it?

• 3:00Daniel Whitenack?

  Pre-pandemic.

• 3:01Chris Benson?

  That's a history book thing now.

• 3:03Daniel Whitenack?

  I mean, yes, but pre-pandemic 2019, we had this great conversation about data labeling with Michael Malyuk of Label Studio. And Michael's back with us. How you doing, Michael?

• 3:16Speaker 00

  Hey, guys. Doing great. I mean, it seems like we're post-pandemic now, so we somehow survived.

• 3:23Daniel Whitenack?

  Yeah, it's crazy that it's been two years since we talked to you.

• 3:26Speaker 00

  Right, right.

• 3:27Daniel Whitenack?

  Different world, right?

• 3:28Speaker 00

  Yeah, it's insane. Like, we first were on a podcast pre-pandemic, now we're on a podcast post-pandemic. So if ever we make the third podcast, I'm kind of wondering what would happen in that timeframe.

• 3:40Chris Benson?

  I don't know, but I'm so envious of you because we're still having it here in the American South. I live in the Atlanta metro area, and it's right now as bad as it was in the worst of the previous thing. So we have some work to do here, but I am incredibly envious of you fellows being in areas where things are getting a little bit better.

• 3:58Daniel Whitenack?

  Well, yeah, I mean, who knows what the future holds? Who knows what the next, like you say, Michael, the next two years will hold. But I mean, the one thing that's true is people will need to label data. I think that that's something we can all agree on.

• 4:13Chris Benson?

  That was such a good transition.

• 4:14Daniel Whitenack?

  You know, I tried.

• 4:15Chris Benson?

  Oh, my gosh. Yeah, that was great. I love the way he did that.

• 4:20Daniel Whitenack?

  So I'm curious, Michael, just from your perspective, as you're day in and day out in data labeling world, like, and through the pandemic, what has shifted or changed? Or do you see like maybe certain trends happening over the past two years in data labeling? What are some of those shifts that you're seeing?

• 4:40Speaker 00

  Yeah, good question. It depends how you look at like the whole markets. What I guess we are seeing from our side when we are talking to our customers and to potential customers is just previously, two years ago, five years ago, the, uh, the market, they were thinking about the data labeling as something that, uh, they just need to get done as quickly as possible, not paying too much attention into the, some of the gritty details of the labeling process. I think is now changing is that our companies are realizing that labeling is actually a way of making your raw data set into what I call a liquid asset. And because of that, they're starting to pay more and more attention around into the

• 5:25Speaker 00

  process and everything that goes into the process, all the reports, the analytics, the metrics inside the labeling process. I think previously it was kind of, hey, we just need to get our data labeled and go to the next step as quickly as possible. Now it's something where they are actively investing resources into building the internal process and practices around labeling. I mean, we probably knew that for quite some time now. The accuracy of the model kind of more or less directly correlates with the quality of your labeled data sets. And I think that now becomes more and more like this common knowledge for the whole market, for the whole ML markets.

• 6:07Chris Benson?

  Do you think that that's a sign of, of just kind of this whole industry that we're in maturing?

• 6:12Speaker 00

  Yep.

• 6:12Chris Benson?

  And recognizing that people are moving from that like, because I remember, I'm totally guilty of that, of just get it done.

• 6:18Speaker 00

  Right.

• 6:18Chris Benson?

  I want to get through my labeling so I can get to my training. And that's the exciting part, you know, the way we were thinking back then. And now people are recognizing that labeling is kind of strategic to their business?

• 6:27Speaker 00

  Yeah, I think it's if you look at the whole ML kind of market, it's pretty new. And there was a lot of really good developments in the infrastructure world. And because of that, the infrastructure for ML kind of becomes a commodity more or less right now. And where your actual strategic investments may be laying in is the labeling, how you process the data and how you prepare the data. Especially if you're working with a data that requires some sort of expert knowledge to be labeled. So everything around how you capture those expert knowledge into the way of labeling data, that's a very liquid asset that you have on your hands after you are done, right?

• 7:07Chris Benson?

  Back when we talked the first time and kind of the world was always about the GPUs and everybody was buying the latest GPU coming out. But there's no competitive advantage because that's available to everybody. And the same thing, whereas the way you label is something that you can create strategic advantage for your business that's based on your expertise.

• 7:26Speaker 00

  Right, exactly.

• 7:27Chris Benson?

  If you have a great tool to do that with. And so I think that's really changed mindset wise since we talked last.

• 7:33Speaker 00

  Yeah, exactly, exactly. And that's a great point because when you start paying more attention into the process and how you set up the process, the software that enables you to do that becomes very important. And from the software perspective, it needs to provide you with all sorts of reports around the labeling process. So you can understand all the metrics where we are like and where we're going. So you have very tight control over that.

• 8:00Daniel Whitenack?

  And those metrics around the data labeling process, do you see customers kind of putting the importance on the quality of the labels? Like how consistent they are, how correlated they are with this or that? Or is it more important sort of quantity? Does what I'm asking make sense? Like in terms of quantity and quality, how have you seen people thinking about balancing those two things recently?

• 8:29Speaker 00

  Yeah, good question. I think it depends on where the company is in terms of their ML adoption. What is the product that we are building? Is it just out of the R and D stage and it's something new and we need to test it out? So the quantity is more important over the quality? Or is it something that we are using, for example, in trading on the markets? Because there, the quality becomes very correlated with the dollar impacts that your model is going to make. Right? So it depends on the model and like, what is the company's, like, their actual business use case? I think over time, quality will dominate over the quantity.

• 9:09Daniel Whitenack?

  And you mentioned culture a couple of times, like the culture around data labeling and also like experts within an organization participating in labeling. Something that I've sort of both struggled with and tried to move forward with, but it's been a struggle is figuring out like all of the best practices around like how you present a labeling task to your labelers, how they view the task, how there can be so much variability between labelers, how like different groups need sort of like different types of instructions or onboarding and this sort of thing. Like, how does a company approach that type of situation?

• 9:53Daniel Whitenack?

  And maybe start to build up a bit of that culture and like a bit of that knowledge internally around like expertise in data labeling, I guess?

• 10:01Speaker 00

  It's a very, very good question and a very complicated one.

• 10:07Daniel Whitenack?

  I would love to know the answer because I don't quite yet know the answer and I need to, probably soon.

• 10:14Chris Benson?

  I think you asked a fantastic question there, Daniel, because I myself really want to understand that that's the hard part when you're getting in.

• 10:23Daniel Whitenack?

  It's a complicated set of answers, maybe, but yeah. You're closer to this than we are, Michael.

• 10:29Speaker 00

  You're very right, like, because when it comes to the data that is subjective in its nature, that's where you have all those struggles with labeling that, because there can be so many different scenarios. For example, one of the more or less obvious things when you're dealing with a subjective data, you want to distribute the same sample at your label into multiple people and look at their consensus, right? So what can happen next? You distribute the same sample to three people, you look at their consensus, all three of them agree, right? But that does not necessarily mean that they label it correctly, because they may label that based on their knowledge that they, all three of them have, but those knowledge are incorrect.

• 11:10Chris Benson?

  So there's a bias there?

• 11:12Speaker 00

  Yeah, it is a bias, yeah. They basically, all three of them are biased, and they're biased in a wrong way. So there needs to be a verification step after that. But that's kind of on the, when you actually start labeling data, right? What comes before that is what you mentioned, instructions for the data labeling. What comes even before that, what we have found out, working with the large organizations, is the teams inside large organizations, they can run multiple data labeling projects, they may use the same semantically named labels, but the actual name of the label is different. So you can think about all the varieties, how you can use the label name for the first name and last name. It

• 11:54Speaker 00

  can be F name, first name, can be just name, and things like that. So even on the company level, when you start and you launch the labeling project, it can be already inconsistent with how other teams inside the organization are thinking about it, right?

• 12:10Chris Benson?

  That begs the follow-up. If you're in a company that is wanting to move, you may have dabbled in deep learning, but there are so many organizations out there that are still not fully in, they're not at a mature level, they're exploring it still, and they're trying to understand how to be productive without wasting lots of money and folks on that. And so how should a company be thinking strategically? So not just the practitioner who's doing the labeling themselves, but if you're leadership and you're wanting to invest in workflow and an infrastructure to support it, how should you be thinking about that in terms of what your company is trying to achieve, so that you get the best out of your practitioners when you have them go in and actually do the thing?

• 12:54Speaker 00

  Yeah, great question. So what I think from the company perspective, from the leadership perspective, take very small steps towards the goal. I think validating the use case with a very little labeling, as little as possible is the first step, and understand the process, basically the process that gets you to a consistent, high-quality labeling in the end. And this process would be different for every organization, right? And it depends on a number of things. One of those is what type of data we are labeling? Is it subjective? Does it require a subject matter expert, et cetera, et cetera? And another kind of dimension is what resources do we have to do that? Do you want to include people from the operations teams?

• 13:39Speaker 00

  Is it going to be a data scientist labeling for the most part? How are we going to be doing the verification step? Who is going to be doing that? So understanding this process to get you to a consistently labeled results, I think this is a critical part in thinking strategically about the data labeling.

• 13:56Daniel Whitenack?

  And one of the things that I've sort of run into a few times recently is even just like the concept of data labeling is like it's so common to us as like data scientists or practitioners and we totally get right away why we need this and like often the huge value that it provides. But then like convincing other groups within your organization about like an investment in this area and thinking about this area, it's actually more difficult than I expect in many cases.

• 14:31Chris Benson?

  That's a good point.

• 14:32Daniel Whitenack?

  Yeah, I mean, I don't know if it's they think, oh, we're investing in AI, not data labeling, or you know, whatever the like thought process is or.

• 14:40Chris Benson?

  It's this label thing, yeah.

• 14:41Daniel Whitenack?

  Yeah, I don't know. As like the CEO of a data labeling company, I'm sure you have many of these conversations with like maybe non-technical people about this process. Any tips for like explaining sort of the value of this, an investment in this area to people that are maybe not data scientists or aren't doing the sort of, aren't using the data themselves?

• 15:05Chris Benson?

  It's so central and yet it's kind of invisible to people who aren't actually doing the work.

• 15:10Speaker 00

  It's more or less like that because you kind of, when you think about the data labeling and you think about the budget that you would need to invest into that, you kind of start thinking, oh, well, maybe we can figure something out.

• 15:24Speaker 00

  So the way I'm thinking about it and the way I usually talk about it with our potential customers is we are right now in a pretty unique stage when ML and AI is being integrated almost into every organization in the world, one way or another. Products are being built based on the ML models. And for the companies, one of the easiest way to improve their model performance is actually improve their data labeling processes. And improving your model's performance would mean that your product becomes more competitive on the markets. If your product is more competitive, you can capture more data with your products and you

• 16:08Speaker 00

  can improve your model even further. I look at it from the way why Google dominates the search space, because they were the one to figure out the algorithm that would make their engine stronger with more of webpages that they crawl, right? And the same thing with the data labeling. You can improve the models and make the models stronger, more competitive, capture more data and keep improving. Basically, over time, I think those companies that are investing into the data label and now would dominate their specific markets.

• 16:42Chris Benson?

  That makes a lot of sense when you think about it. I think it's something that is that outside of our tiny community, it's not really understood very well.

• 16:49Speaker 00

  Right.

• 16:50Chris Benson?

  The effect of labeling can either make a dataset really, really useful and really productive across multiple things. And yet, if you don't do it well, you can end up with very poor results with the same data.

• 17:03Speaker 00

  Right.

• 17:03Chris Benson?

  I think going to Daniel's comment a few minutes ago, it's I don't think that's well understood in a lot of executive ranks.

• 17:26Speaker 02

  Signal Wire is realtime video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast quality ultra-low latency video that's proven and trusted by Amazon, Ring Doorbell, Zoom, and others. See why the future of video communication is being built on Signal Wire. They have easy-to-deploy APIs, SDKs for the most popular programming languages, and expert support from the OGs of software-defined telecom tech. Try it today at signalwire.com and use code AI for $25 in developer credit. Just visit signalwire.com, that's signalwire.com, and use code AI

• 18:08Speaker 02

  to receive that 25 bucks. Once again, that's signalwire.com, code AI.

• 18:26Daniel Whitenack?

  So I think we've done a good job at really diving into the sort of value of data labeling and how people are thinking about it now, which is fascinating. But I'm curious, on the tooling side, I definitely want to get into Label Studio and what has Progressins in because I have some thoughts and questions there, too. But in terms of this space in general, how are you seeing this space of like tools around data labeling kind of grow and shift over these past couple of years?

• 18:55Speaker 00

  It's pretty crazy. It's very crowded market in some sense. There are a lot of tooling. Even two years ago, when with our previous podcast, it was already pretty crowded. Now there are kind of more players and more tooling around the data labeling tools, which I think is interesting. So there are like all sorts of data exploration tools as well that are now available. On the other hand, it's very exciting to see a lot of smart people putting in a lot of effort into building all that ecosystem at the moment.

• 19:27Daniel Whitenack?

  Yeah, so with that, why don't you tell us a little bit about Label Studio itself? In case people haven't caught the previous episode, but also a lot, you know, a lot has been updated since then. And I've got a little bit of a secret to tell, you know, Chris, I'm not unbiased in this conversation because I am a Label Studio fan and user. So our organization uses Label Studio and I've used it on a bunch of different things over the past couple of years. So I am personally very happy that we had that conversation two years ago because it saved me a lot of work over the past couple of years. You know, looking at Google Sheets where people have tried to label something or, you know, something like that.

• 20:10Daniel Whitenack?

  Anyway, tell us a bit about what Label Studio is and how people can use it, maybe.

• 20:15Speaker 00

  Yeah, sure. So Label Studio, it's an open source data labeling tool. One of the key features that was a key feature two years ago and is still one of the key features today is that it's the most flexible data labeling tool. So instead of us giving you the interface that we think would work for your use case, you use a configuration language that we have created to build the interface for your specific data set. It's multi-data type, so it supports a variety of data types. If you want to label for text, for audio, for images, and it's also multimodal.

• 20:54Speaker 00

  So you can put the text and the audio and multiple images on the same screen and label them at the same time. To install, there is our website. You can Google Label Studio. As I said, it's open source software, so there is a pip package, there is a Docker container. All of that, you can get up and running in a couple of minutes.

• 21:17Daniel Whitenack?

  Yeah, I love the fact that I can just launch the server, right? So, Chris, I'm gonna brag on Michael's work here for a second. I can launch the Label Studio server and then like it's available and then I can build different data labeling projects within the one server, right? So it's not like I have to like preconfigure. My labeling task and then like launch it, I can like launch it and then like build my labeling task in the interface, which is kind of fun because I'm not like a UI person by any means. So being able to do that and the sort of customization has been cool for us, I know, because like, for example, the last one I set up was a question answering dataset labeling.

• 22:01Daniel Whitenack?

  And even if there's like a pre-configured thing for question answering, it's like there's a question, maybe a context or a passage from which you're answering that question and you like select the answer in the text or something like that. But for our task, we actually had nine different contexts and we had to select the answer in each one of the nine different contexts. So like in another tool that wasn't like this, I would sort of have to shove all of that together and maybe like the same text and then like create little like markers to separate them and do pre and post-processing. But here I could just add like nine different text blocks to like label. So I think for me, that customization is like a really key feature.

• 22:45Daniel Whitenack?

  Maybe you have stats on this in terms of like what people are using or maybe you don't, but you have templates, but do you see a lot of people like in your Slack channel and other things, what are some of the creative ways that you see people customizing these tasks?

• 23:00Speaker 00

  I think one of the powerful features of Label Studio is that you usually just need around 10 lines, well, depending on the number of labels that you have. But sometimes you just need 10 lines of this configuration language to build a pretty complicated interface. And that's what we usually see in people coming up with the interfaces with like 10, 20, 30 lines of code. But at one point, I was on this interesting call with one of our potential customers at the moment and they just shared their screen. They used 400 lines of configuration language to basically almost build like a web portal inside Label Studio.

• 23:41Speaker 00

  It was crazy just to look at that, like all the different things that they have put in there.

• 23:47Daniel Whitenack?

  Like how much time they've spent in that window making that thing.

• 23:51Speaker 00

  Right, right, right.

• 23:53Chris Benson?

  That sounded sort of like almost another mea culpa from Dan right there, like I've done that, you know, kind of like, oh yeah.

• 23:59Daniel Whitenack?

  We won't get into that.

• 24:02Speaker 00

  But it was interesting to see like, what are some of the like extreme use cases that people have done with the tool?

• 24:09Chris Benson?

  So I'm gonna follow up, since I gotta tell you, Michael, I don't think I've ever seen him this just purely excited in one of our episodes about something. So I'm very impressed. Oh yeah, I'm a total fanboy. He totally. And the listeners can't see, but we can see each other in the thing. And yeah, totally, man. I want to convert what the excitement that Daniel has over to our listeners that haven't used it before. Can you take that and kind of talk a bit about your workflow for someone who hasn't had a chance yet and they're sitting there in their car, they're in, you know, traffic right now, and they're going, when I get to the office, this is, I'm gonna open this up and go do it. What should they expect? What should they be thinking about? What's the workflow look like?

• 24:50Speaker 00

  Yeah, so in a nutshell, it's a web app, right? So this is something that runs in your browser. This is something that you can launch on your EC2 instance, or whatever the story you're running on your laptop. It doesn't need a connection to the internet. You go to your browser and you can start creating the data label and projects. As I said, it's very flexible, so you can configure it specifically for your data sets, no matter what your data sets consists of. Images, data set, if you're doing the computer vision, the NLP tasks, named entity recognition, audio segmentation, all a variety of the use cases. And then what's also interesting is you can start connecting the machine learning

• 25:30Speaker 00

  models to help you do the annotation. And the flow of that is very interesting, because you do the annotation, a batch of annotations, and you retrain your model. And then you do another batch, and you retrain your model again, and you get the new predictions out. And you keep improving the model predictions this way. It almost feels like teaching a child to do a certain specific task for you, right? And this is something that I personally find very exciting, because for me, it was like, okay, I can see the actual improvement of the model in its predictions in the real time, right? And this is just one of the setups,

• 26:12Speaker 00

  how you can set up the label studio. Another one might be that if you're running some sort of human in the loop types of like ML pipeline, where if the model is not sure in its prediction, you send that to a person. And the person would be labeled inside Label Studio as soon as the label that gets back into their retraining phase. Right? And that's another use case. But we put a lot of effort into making sure that the actual interface inside Label Studio is very intuitive and easy to follow. Because we, as a data scientist, we always focus on ourselves, but we also put specific focus on a non-tech savvy users.

• 26:51Speaker 00

  Because those users can actually have a lot of very specific domain and interest in knowledge. And you want to be capturing that. But if you build a tool that is very complicated, they would spend a lot of time figuring out how to use that. So we are trying to make it as simple as possible, but yet powerful. I hope that got somebody interested.

• 27:14Chris Benson?

  I bet you did.

• 27:15Daniel Whitenack?

  Yeah, and maybe that's part of like going back to the hard questions that we discussed earlier about like building that culture of data labeling, getting instructions right, setting up the task right, doing verification, all of these sorts of very hard things. If you don't have to add to that, like an interface that people like is very unfamiliar and they can't like figure out how to use, then that's at least one less thing to like complicate those muddy waters. Yeah, so like a web app and a browser where people can just sort of click things. That's a nice scenario, I think. And I'm sure you've looked at sort of user experience, UI type things over time, Michael.

• 27:57Daniel Whitenack?

  What has that experience been like as you've developed the front end of Label Studio? And maybe things that you thought would work and didn't work, or maybe things that you integrated and people love, but you didn't think would would have been as big of a deal?

• 28:10Chris Benson?

  He's smiling before he answers.

• 28:13Speaker 00

  Many, many.

• 28:17Speaker 00

  What I think is interesting, one other reason why we wanna give you this flexibility of building the UI is because you can keep only the relevant parts that are relevant to your data sets. And that minimizes the error on the annotators. So they don't have something that they actually don't need to use in terms of the UI elements. Another thing on the UX and UI design, I wanna really praise our open source community. So we have had dozens of UX sessions with our open source users. We basically just invite them and we ask them to click around and show them the prototypes. They've been very, very helpful in designing the application.

• 29:01Speaker 00

  We have right now more than, well, not more than, but are close to 2,500 people in our Slack. And some of those community members are very, very helpful. And we can do ourselves only as much, but then the other bigger part comes from all the contributions in terms of like how we may think about certain cases from our community. On the mistakes part, we have done a lot of things that didn't really work. But we were quickly identifying those things that are kind of not really user friendly and just removing them. So we have this tendency to like to really understand what doesn't work. Try not to clutter the application.

• 29:45Speaker 00

  Because again, we try to make the tool as simple as possible, but yet to have all those powerful features, which is more of an art than a science, I guess.

• 29:56Chris Benson?

  So I'm curious, you've got me excited all over again. I remember the excitement of this because I went, I haven't been doing as much hands on as Daniel has, but I went and started using it as well afterwards. I just don't know it as well. I'm wondering though, as you've done these improvements over the last couple of years and you're looking at a world where this is just gonna become more and more embedded in business. And so it's not, it's not this off thing. And the amount of data as people are finally starting to learn how to collect their data and not have It transient and gone and and they're storing it and the volume of that data goes up by orders of magnitude. And to some degree, for use cases, the quality of the data may vary tremendously on that.

• 30:39Chris Benson?

  How do you envision the process of labeling going forward in the years to come? As the problem, you know, it's solved in the sense of what you've done so far, but it's an ever changing problem. And so you're constantly going to have to chase that down. How are you seeing that that curve into the future?

• 30:58Speaker 00

  Yeah, great question. I think in the future, well, there are different vectors. First of all, with ML adoption itself, which is going to have more and more use cases that ML can support. Meaning that from the labeling perspective, we would need to do a lot of the multimodal labeling and just different varieties of labeling that we are not thinking about right now. Then from the perspective of the volume, we're going to have a lot of the really well pre-trained models. They're going to help us label a vast amount of data automatically. And then basically what people would be concentrating on are their edge cases, the main specific cases.

• 31:41Speaker 00

  And just the cases where there are not good pre-trained common knowledge models available yet. So I think with a more ML adoption in the business, we'll see that the need for the labeling is only going to grow. Just because from the fact that ML is going to support more and more use cases within the business.

• 32:01Daniel Whitenack?

  Do you get any sort of weak signals off of what people are doing with Label Studio? In terms of the different domains of machine learning, AI, in terms of like people's focus on NLP or people's focus on audio or like new things with computer vision? Do you happen to see like trends in that with like new people coming into Slack wanting to do all of the sudden? Or, you know, something like that?

• 32:28Speaker 00

  Yeah, it's a good question, but it's not easy to answer that too. Because what we see, for example, we are one of the very, very few tools that supports time series labeling. And majority of the people that are doing time series labeling, they use Label Studio. So we see a lot of those people, but then on the other hand, we see a lot of people who are doing labeling for computer vision. And they're well, not comparable, there is more people doing computer vision, but we have more competitors in terms of the computer vision data labeling tools. So I would not be able probably to say that there is a one dominant data type that we see people using the tool to label, again, because of the flexibility of the tool.

• 33:13Speaker 00

  And over time, I don't think that we see a lot of the change because it's the data types are distributed more or less equally, just because from the fact that people usually use a few of the data types at the same time on the same screen. You can think about audio and then there is a description for the audio and you kind of want to have the audio on the same screen just for the reference and you're labeling the transcript. So it's hard to pick something and identify the change just given the flexibility of the tool and what it provides to the end user.

• 33:43Chris Benson?

  Are there a lot of use cases for having kind of that multi-model approach? If you go back a couple of years and people would tend to be either really focused on, you know, NLP or really focused on, you know, labeling convolutional, but they tended to be one or the other in my experience. And what I've seen since then is a lot more integration and less emphasis on what it is. Are you seeing more of that in the tool? And, and how does the tool accommodate that workflow as it's evolved into this kind of mesh of different approaches together?

• 34:15Speaker 00

  Yeah, totally. So I think there is a clear trend for multi-model being on the rise. And from the tooling perspective, it's, yeah, because you're building the interface yourself, you can put as many data types on the same screen as, as you want. And because you have this ability, even if you don't do the multi-model labeling, you label, let's say, just, just a text, right? Having the audio for the context, if the text is not clear because the transcription was screwed. You still would want to do that. So the tool kind of naturally supports this type of use cases. But yeah, I agree that I, I, I think that we are definitely moving into more of the multimodal world.

• 34:55Daniel Whitenack?

  So we've talked a lot about the tool itself. It is an open source tool and I'm always curious to like in terms of people that are building a business around open source, whether that be a label studio or hugging face or, you know, this sort of grid AI people with Apache TBM or, you know, all of these different cases. It seems like people, of course, are always playing with that model and figuring it out how it works. So you've got the open source tool, but then also you're dealing with a lot of data and open data is very much a topic right now. So how do you think about building your business in this space around an open source tool?

• 35:39Daniel Whitenack?

  And has that been a struggle in terms of that balance?

• 35:44Speaker 00

  For the open source company, it's always a struggle because it's open source. You have to identify what you actually want to sell as a product. So that's always a complexity, but I think there are some really strong advantages to being the open source company. One of them is a community. You get a lot of insights from the community. They help you improve your software the way you would not be able to improve your paid product. Another one, because of the community, you have one of the largest community of testers of your software, meaning that your software becomes so well tested just because of the pure adoption of that, that you have one of the most stable tools out there, which I think is great.

• 36:29Speaker 00

  And those advantages, I think, at least to myself, they compound over those disadvantages where you have to struggle sometimes to identify if this needs to go into the open source or that goes to into the paid product. And then another one that I think is really important from the company perspective is all our developers, they have direct access to the users of their software and that makes them really Be happy when somebody is getting a lot of value out of it. And they talk about you on Twitter or somewhere else. And they just tell you how great is the tool that you have built for the developers. It's basically the best they can get. And they get very excited about that.

• 37:09Speaker 00

  So I think at least for myself, the advantages of being the open source company clearly beats all those disadvantages that we may have.

• 37:19Chris Benson?

  We've seen a lot of the tooling in the larger kind of AI ML space. Start as open source, which was very different. And I can say this because I'm on the older side from the way the software world started off. It was all commercial and open source kind of came in and it was small for a while, but it grew and took over. We've had the benefit in ML of being able to kind of start from that. Do you think that that will continue to be the model just because of the benefits that you just now drew out? That it allows you to kind of accelerate and be totally connected with your user base?

• 37:51Speaker 00

  I think there are going to be, I hate to hypothesize on the what future holds for us. Like if two years ago you would ask me about the future, I would probably not be able to predict the pandemic coming.

• 38:02Chris Benson?

  Nor did any of us.

• 38:06Speaker 00

  But I think it's going to be a mixture of both. I think there are going to be a market, a huge market for the commercial only closed source solution. And there are going to be a market for the open source. It's just, it's the different models and I think both of them have their advantages and disadvantages. So I'm kind of not thinking about commercial being kind of in the way of open source. I think they can just go in parallel.

• 38:31Daniel Whitenack?

  Yeah, and data labeling, I'm guessing, will always remain fairly challenging. There'll always be an opportunity, I think, for people to build a business out of those things just because it is so challenging.

• 38:45Speaker 00

  Right.

• 38:45Daniel Whitenack?

  Yeah, I like what you were mentioning earlier about like one of the things you're sort of keeping track of is people using sort of models to seed their labeling, so pre-trained models to do that. And that's definitely something that has benefited us. So it's cool to see that you're integrating some of that ML layer within Label Studio. I'm curious, as a user, maybe not as futuristic of a question, but what does the roadmap look like for Label Studio? What are those things that you're really like, you're getting a lot of requests for? Or you're really sort of at the top of your mind in terms of enabling in the tool?

• 39:23Speaker 00

  Yeah, I think I'll, first of all, we do have a public roadmap on our GitHub. I'll share one that I think probably the most exciting one. In the next couple of months, we're going to be releasing the support for video. Right now, you already can do the video classification type of tasks, but we'll be releasing the one where you'll be able to do the object tracking.

• 39:45Daniel Whitenack?

  That's cool.

• 39:45Speaker 00

  Yeah, this is going to be huge. With the introduction of this proper video labeling, we'll be covering all major data types. And that was our kind of goal for the labeling tool. So that's pretty exciting.

• 40:00Daniel Whitenack?

  Yeah, I mean, I know that there's like cloud video editing like tools and stuff now. I'm sure that that does definitely like put a bit, maybe more of a strain on the tool in terms of its performance than other things. Is that like, is that coming last because that major category that you're talking about coming last because of some of those challenges? Or is it just sort of like the growing need for that? And that's what sort of came next?

• 40:28Speaker 00

  It's a mix of both. It's both challenging from the, it's just the complexity of the video labeling is very high. Then we see more need in terms of our users asking us about labeling. And then we just started with images, audio and tags, just because the founders of the company, they're like, we all were coming from exactly these backgrounds ourselves. So the video labeling, for some reason, it comes the last one. But I think it's a It would be a good kind of thing to get all of them under the one hood. Yeah.

• 41:04Chris Benson?

  So it's interesting to me that you said that and they announced that because I think it puts pressure maybe on an area in software that, you know, wasn't really thinking of themselves as, as having to worry quite so much about ML. It's not central to the thing. It's ancillary in some, but there's a whole new arena that you're going to be moving into in that sense.

• 41:25Speaker 00

  Yeah, totally. And I think what's also exciting is because, again, you can put different data types and we try to make them talk to each other. There are very interesting use cases that can be uncovered by that. One of the use cases that was very much requested by our community is, for example, when you're doing the time series labeling and you want to have the reference with your video stream, right? Because when you think about labeling those plots, they may not provide you enough information by themselves to do the labeling. But when you have those video stream that is kind of acts as a reference for what the, for example, the robot arm was doing, then you can effectively label the time series.

• 42:10Speaker 00

  And I think this kind of merge of different data types on the same screen supporting each other just like opens another box of the use cases that would be available to the community.

• 42:22Daniel Whitenack?

  Awesome. In two years from now, when we are having this conversation again, what do you think we'll be talking about?

• 42:31Chris Benson?

  No more pandemics, though.

• 42:32Daniel Whitenack?

  Yeah, I mean, hopefully no more pandemics. Maybe the, the data labeling world will be fully taken over by, by Label Studio. But yeah, I certainly hope that we have that conversation.

• 42:43Chris Benson?

  I think he's going to have Steven Spielberg sitting next to him at that point.

• 42:49Speaker 00

  Maybe I already have him.

• 42:53Daniel Whitenack?

  Well, it's been super fun, Michael. I really appreciate. I'm looking forward to that conversation in a couple of years. So thank you so much for joining.

• 43:01Speaker 00

  Likewise.

• 43:02Daniel Whitenack?

  And we'll make sure our show notes include all the links to great Label Studio stuff. So all of you who are listening, definitely check it out and we'll talk to you soon, Michael.

• 43:11Speaker 00

  Thanks, guys. Thanks for having me.

• 43:16Speaker 02

  Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you at changelog.com, including our brand new show, Ship It with Care Hard in the Zoo, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra, and the people that make it happen. Yes, we focus on the people because everything else is in implementation detail. Subscribe now at changelog.com/shipit or simply search for Ship It in your favorite podcast app, you'll find it. Of course, the galaxy brain move is to subscribe to our master feed. It's all changelog podcasts, including Practical AI and Ship It in one place. Search changelog master feed or head to changelog.com/master and subscribe today. Practical AI is hosted

• 44:01Speaker 02

  by Daniel Whitenack and Chris Benson with music by Breakmaster Cylinder. We're brought to you by Fastly, Von Starfleet, and Linode. That's all for now. We'll talk to you again next week.