Open source data labeling tools

What’s the most practical of practical AI things? Data labeling of course! It’s also one of the most time consuming and error prone processes that we deal with in AI development. Michael Malyuk of Heartex and Label Studio joins us to discuss various data labeling challenges and open source tooling to help us overcome those challenges. Sponsors: DigitalOcean Managed Kubernetes https://do.co/changelog – DigitalOcean makes it super simple to launch a Kubernetes cluster in minutes. Developers can now run and scale container-based workloads with ease on the DigitalOcean platform. Learn more and get started for free with a $50 credit at do.co/changelog https://do.co/changelogKubeCon + CloudNativeCon (Amsterdam 2020) https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2020/?utm_source=changelog&utm_medium=podcast&utm_campaign=kceugotime – Get an additional 10% off EARLYBIRD registration pricing with the code KCEUGOTIME — ends November 8th! Learn more and register https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2020/?utm_source=changelog&utm_medium=podcast&utm_campaign=kceugotime. AI Demystified (FREE five-day mini-course) http://learnai.infinite.red/ – Get an introduction to the most important concepts, types, and business applications for AI and Machine Learning. This course is 100% free. Featuring: Michael Malyuk – LinkedIn https://www.linkedin.com/in/mikhail-malyuk, X https://x.com/michaelmalyukChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Heartex https://www.heartex.ai/Label Studio https://labelstud.io/Books “Paradigms of Artificial Intelligence Programming” by Peter Norvig https://www.amazon.com/dp/1558601910Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/changelog. Again, do.co/changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slide with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:47Daniel Whitenack?

  Well, welcome to another episode of Practical AI. I'm Daniel Whitenack, a data scientist with S I L International. And as always, I'm joined by my co-host, Chris Benson, a Principal AI Strategist at Lockheed Martin. How you doing, Chris? Doing great, Daniel. How's it going today? It is going good. And I think when listeners will be listening to this in the future, if we're to imagine into the future, you will be at the Nvidia conference. Is that right?

• 2:18Chris Benson?

  That is accurate. As listeners are downloading this, I'm probably in Washington DC. I'll have just finished doing some commentary for the Alpha Pilot race. And those of you who aren't familiar, we had a recent episode about Alpha Pilot, which is really cool, autonomous drone racing. And as of the point where you're downloading this on Tuesday, which may potentially be tomorrow for you, I will be conducting a sort of a fireside chat slash panel about Alpha Pilot and AI at the edge at Nvidia's GTC DC event on Tuesday evening. So I hope if you're listening to this and you happen to be at GTC, I hope you'll come attend. And whether you do or not, find me, I'll be there all week and

• 3:00Chris Benson?

  find me and let me know you're a Practical AI listener and let's connect. I can't wait to meet people there.

• 3:05Daniel Whitenack?

  Yeah, that sounds great. I can't wait to hear about how it goes. Well, today, you know, if I think about Practical AI, Chris, and I asked you, what is the most practical of Practical AI things? What would you say?

• 3:20Chris Benson?

  You already know what I'm gonna say. You're setting me up for that. It's labeling.

• 3:24Daniel Whitenack?

  I'm teeing you up for the right answer.

• 3:26Chris Benson?

  Oh my God, it's labeling data, the bane of my existence over the last few years in terms of doing AI. If we can get the data labeled, then I'm finally at a happy place where I can start doing training and have some fun.

• 3:39Daniel Whitenack?

  So today is all about the data and all about labeling the data. And we're joined by Michael Malyuk, who is CEO and founder at HardX and one of the contributors and maintainers of Label Studio. Welcome, Michael.

• 3:57Speaker 03

  Hello, hello. Thanks for inviting me.

• 3:59Daniel Whitenack?

  And we were able to grab Michael for an interview while he is doing his world travels. He's, I think, right now in Moscow, is that right?

• 4:07Speaker 03

  Yep.

• 4:08Daniel Whitenack?

  Awesome. Yeah. Well, thank you for joining us even in the midst of your jet lag.

• 4:13Speaker 03

  Oh, yeah.

• 4:14Daniel Whitenack?

  Yeah. So if you could just give us a little bit of an intro to yourself, your background, how you kind of got involved in AI, ML things, some of the things you've done, and how you ended up with a focus on data labeling.

• 4:30Speaker 03

  Sure. So I got into, it was quite, I think, interesting path for me because I have started with Lisp programming in Common Lisp. And then at some point, I got this book by Peter Norvig, who I think right now at Google, that is called "Paradigms of Artificial Intelligence", and it was kind of a mixture between Common Lisp and using Common Lisp to program AI. It was quite an old book, even when I got it. I think it was released in...

• 5:05Daniel Whitenack?

  Yeah, it sounds intense.

• 5:06Speaker 03

  Yeah, it was like 91 or 93, something like that. But it got me really interested into the whole concept about AI and how to program AI systems. And so from there, I kind of started to pick up all the, what actually needs to be built in production-level AI systems. And I'm here talking about more like math background, statistics background. And yeah, so that's what got me really, really, like, really interested into the whole AI concept, Common Lisp, 50 years old programming language.

• 5:41Daniel Whitenack?

  Well, my first programming language was Fortran and used it quite a bit. So I'm glad that I left it behind, but we all have, at least some of us have those roots in older languages.

• 5:55Speaker 03

  Yeah, it's like if we're talking about Common Lisp, it's like I still kind of big fan of the language. And it's a pity that it's not kind of catching up with and not becoming more mainstream these days.

• 6:07Daniel Whitenack?

  Is that how common is it to find people these days working in AI or machine learning and using Common Lisp?

• 6:15Speaker 03

  Not common at all, not common at all.

• 6:17Daniel Whitenack?

  Not common at all to give a pun?

• 6:20Speaker 03

  Oh yeah, the only one who I can actually remember is the person Gabor Melis, who has won a number of AI competitions. I think those were organized by Google. And he wrote all his programs in Common Lisp. And those was like very interested to study and to explore how he would approach things. Yeah.

• 6:41Daniel Whitenack?

  Yeah, that's cool. So you started out there. How did you end up coming to the point where you started thinking about founding your own thing that would impact the AI community?

• 6:52Speaker 03

  Yeah, sure. Yeah, so at the time, and it was something about a year ago, me and my then-to-be co-founder, we went to this hiking trip in the high-altitude mountains. And I think the lack of oxygen kind of contributed into us starting the company.

• 7:13Daniel Whitenack?

  Where was this?

• 7:14Speaker 03

  Himalayas.

• 7:15Daniel Whitenack?

  Oh, wow. What was the the max elevation?

• 7:20Speaker 03

  So I think I was giving up at almost 6,000.

• 7:26Daniel Whitenack?

  Yeah, 6,000 meters.

• 7:27Speaker 03

  Yeah, yeah, yeah, yeah. My co-founder, he went further, but the weather was really, really bad, so he had to turn back.

• 7:34Daniel Whitenack?

  So yeah, so as co-founders, you've been through a lot together.

• 7:38Speaker 03

  I'm so jealous. Yeah, I mean, it's probably as hard as it gets, like all those conditions, you know?

• 7:45Daniel Whitenack?

  Yeah. What is VC funding and fundraising around as compared to that?

• 7:49Speaker 03

  It's the easy trip. 2,000 meters.

• 7:55Daniel Whitenack?

  Yeah, nice. So the company is HardX and so, you know, not to, I think we've already spoiled the punchline here, but it has to do with data labeling. So how did you kind of get to a place where data labeling was something that you felt like you wanted to devote a lot of effort to?

• 8:13Speaker 03

  So we kind of, we were working on some algorithms in our spare time, just trying things. And by the time we decided to start a company, we all realized that at our past or current companies, the biggest issue that we had to hand on ourselves was the data labeling part. And we were talking to multiple data scientists, machine learning experts, and they have all agreed that it's kind of unsolved problem and more or less like a holy grail. If you're able to get your labels in the right moment, in the right place, then you basically end up with high-quality models. And by the time we realized that it's not only us who had these problems,

• 8:58Speaker 03

  but also other developers and other companies, then we decided to proceed and start the company around the data labeling.

• 9:06Chris Benson?

  So I for one, you know, you heard how I reacted to that in the very beginning about labeling, that being the bane of my existence. I for one, am thanking you very much for attending to that issue, because you know, everybody in the data science world, you know, wants to do the sexy AI training and such as that. And this is a problem that has needed serious attention for a long time. So as I ask you about it, I just wanted to start by saying, thank you very much.

• 9:35Daniel Whitenack?

  It's a problem I feel like have wider impact than like creating a single state-of-the-art model and publishing like a very hyped paper. Like this is the sort of problem that has a much wider impact.

• 9:50Chris Benson?

  Absolutely. So I guess I want to start off by just kind of, if you could kind of tell us a little bit about where does data labeling fit into the kind of the larger workflow that we're all engaged in, in the AI world? And you know, how does it relate to the AI problems that we are all working on? And you know, Daniel has been focusing on lots of stuff having to do with language stuff. And I've worked in robotics and different types of image classification stuff. So how does labeling fit into these workflows?

• 10:21Speaker 03

  Yeah, sure. So I think my personal take is that data labeling and annotation is basically the core of any AI-based product. Because if you are building on the labeled data, of course. Because if you're not able to go into your data sets and relabel some things that, for example, might have been labeled incorrectly or inaccurately, then you just end up with the models that are not working well. So my take is that data labeling, it's the first step that comes after collecting the data. And it's something from where everything basically begins.

• 11:05Speaker 03

  And I also like to think about it in a way that sometimes when you get the data sets or you collect the data sets, you're not sure what's inside of it. So it's not only that you need to label it, but while you're doing the labeling, you're also kind of exploring your data set, finding the edge cases or some examples that you have not thought about. So I think the data label is basically the core functionality or shall be a core functionality of any data science team or the team that builds AI-based products.

• 11:39Daniel Whitenack?

  Yeah, so in terms of like the types of data that people generally need to label or annotate, as Chris mentioned, there's of course a lot of different types of data that are relevant to AI. And in some cases, AI models might work on multiple types of data. But maybe you could just give some common types of data that people need to label or annotate, or maybe the most common ones that you run across?

• 12:07Speaker 03

  So I would say that most common ones, images, of course. So when you are placing a bounding box on the image, when you're doing semantic segmentation on the image. You can also think about even classifying images as a data labeling problem. So you're basically assigning a class to the entire image. What you see on the image, is it, for example, an image of a fruit or image of a person? Then text. So you can be classifying text, for example, for sentiment. You can be doing named entity recognition. Audio. Again, you can be doing speaker separation.

• 12:51Speaker 03

  You can be classifying audio for a number of classes or doing the multi-class classification. So those, I would say, most common ones. Then you, of course, have time series. Now you also have 3D spaces with all the leader sensors that are coming, and these data is coming from autonomous vehicles. And videos. So I would say these six data types are the most common ones. And inside each data type, you kind of have different types of labeling or annotations that you can apply to this particular data type. And that usually depends on what you're looking to achieve in the end. Yeah, and as you have studied, you can also mix those.

• 13:36Daniel Whitenack?

  Yeah, so the type of annotation that you perform is really tied to the task or the objective that you want your model to perform.

• 13:45Speaker 03

  Yeah, totally.

• 13:46Daniel Whitenack?

  Right, so it's like, if you want to pick out entities in text, which is what's done in named entity recognition, like you want to determine, are there any places or people that are mentioned in this text? Then you need to have data that you have manually labeled in some sort of gold standard way to help the model figure out, based on those examples, what the relationship should be between input data and the output of those entities. So there's a whole bunch of, I guess, there's probably infinite different like types of annotations that you could come up with because there's, you know, infinite different tasks that you might want to perform. Is that accurate?

• 14:28Speaker 03

  Yep, totally. Yeah, it depends on the data set that you have at your disposal. And then it depends on also what you're looking to achieve with your model. And you're absolutely right. You're labeling the data. And then, based on your labeling, the model kind of learns the patterns and try to identify the same patterns from the new data that comes in. And that's how you basically get a prediction from your model.

• 14:53Chris Benson?

  So it occurs to me that we've kind of thrown some terms around a little bit, and we've talked about labeling and annotation and recognizing that not everybody has had a chance to do this that might be listening. What does it actually mean to annotate that data? You know, when you're annotating or labeling data, what is it you're specifically doing? What is required to achieve that? And what is the practical benefit of going through that process?

• 15:20Speaker 03

  Sure. So the process itself, again, it's very dependent on the data set and what you're trying to achieve, like the problem that you're trying to solve. But in general, I would describe it as basically creating some metadata for every item in your data set. So this metadata, for example, for an image, this metadata might be a rectangle at a certain position with a certain class applied to this rectangle, which is basically called the bounding box labeling. And to do so, you need to have the tool that enables you to put this bounding box at the

• 16:01Speaker 03

  right position and to assign the label, the class that you want to this bounding box. And a lot actually depends on the tool, how accurately and how quickly you can do that in case your data set is super big and in case you're looking for very high quality labeling. Because in the results, the quality of your models, in most cases, is directly tied to how accurately your data is labeled.

• 16:30Chris Benson?

  Gotcha. And one of the things that you mentioned, by the way, being bounding box a couple of times, just wanted to note, kind of define that for a second. A bounding box, if you're looking at an image and you're trying to define the value of different parts of that image for purposes of training subsequently, then a bounding box is really just like it sounds. It's a geometric shape that you're assigning to different parts of the image to define the different areas of the image that you want the model to either focus on or not focus on.

• 17:00Speaker 03

  Yep.

• 17:01Chris Benson?

  I just wanted to note that.

• 17:01Speaker 03

  Yep.

• 17:13Speaker 00

  What is up practically eye listeners? We're working with Infinite Red to promote their free AI mini course. It's called AI Demystified. Learn more enroll at learnai infinited dot red. This free five day mini course is a great introduction to the most important concepts, types and business applications for AI and machine learning. Each day of the course includes a lesson, a quiz and an assignment to submit your learning. And after you've completed the course, you'll also get a certificate of completion for your LinkedIn profile or for your portfolio. If you've been feeling lost in the world of AI and hearing lots of buzzwords, then by the end of this mini course, you'll be able to speak intelligently about AI and machine learning and their practical business applications. Again, this course is completely free.

• 17:58Speaker 00

  Learn more and enroll at learnai dot infinite dot red. Again, learnai dot infinite dot red.

• 18:16Daniel Whitenack?

  Okay, Michael, so let's say that I'm convinced that I need to do data labeling and I'm convinced that I should put time into it. It is an important part of my AI workflow and one of the most important parts because it has this direct impact on the quality of my predictions. And let's say that I have 100,000 samples or more to label. There's obvious challenges around the like time consuming nature that it would take to label each of those 100,000 samples. Are there other challenges like I'm thinking in terms of maybe bias or like crowdsourcing this or like what sorts of challenges do people face when they're labeling data?

• 19:00Daniel Whitenack?

  Maybe other than the obvious one, like the time consuming nature of it.

• 19:04Speaker 03

  Oh, many, many. So time, yeah, definitely. If you have big enough data sets, a million of items, it's gonna take a lot of time to label it. Then quality. How do you verify that the results of the labeling and the actual labels are, for example, if we're talking about bounding boxes, that the bounding boxes are in correct positions? Biases. When different people label the same data set, you may end up with different results. So personal biases. Then even before that, you actually need to have a tool to help you do that. Because data sets are different types of annotations and labeling are very different.

• 19:46Speaker 03

  So you have to invest time into either creating your own tool or using something from the open source.

• 19:54Speaker 03

  And there are basically many more. I would say that two major ones are time and quality. Yeah.

• 20:01Daniel Whitenack?

  Yeah, so quality being around the verification of the data.

• 20:05Speaker 03

  Yeah.

• 20:05Daniel Whitenack?

  Right? So like if I crowdsourced, let's say, a million parallel translations between two languages, how do I know that those were actually good translations, given that I don't already have the model? So it seems like it's a sort of chicken and egg sort of thing. How do you deal with something like that?

• 20:28Speaker 03

  Yeah, to add to that, that works if you can crowdsource, right? So for example, if you're dealing with a data that requires kind of like domain-specific knowledge, for example, medical images, right? You can't crowdsource that.

• 20:45Daniel Whitenack?

  Yeah, and it's expensive to hire doctors, I imagine.

• 20:48Speaker 03

  Yeah. Then another one, if privacy is an issue, then you also can't crowdsource that. You need to have in-house data labeling team. So for the quality control, there are multiple ways how you can verify the results. One of them is you can kind of label fraction of your data set, verify it multiple times that it was labeled correctly. Then you can train a model on top of that and further use this model to verify the subsequent labels that are coming. Another one, you can distribute the same task to multiple annotators

• 21:31Speaker 03

  and verify their in-consensus between each other or not.

• 21:35Daniel Whitenack?

  So when you're talking about sort of bringing the model into this process, is that what people refer to as model in the loop versus out of the loop labeling? Where you actually kind of have a model that's trained on some of your data when you're trying to label more data and updating that? Is that what that means?

• 21:56Speaker 03

  I call it more or less like automatic labeling. And here the most important piece is how do you pick those items in the first place? Basically, if you have a very large data set, how do you pick those items that you want to label first? And using those labels, you kind of can build a model, but how do you pick the items? And so this field is called active learning. And active learning is basically a way to pick the items from your data set that provide you enough information about the data set as a whole. So you're analyzing the data set and picking exactly those items that you want to label first in order to be able to label the rest of the data set for you.

• 22:40Daniel Whitenack?

  So how are people currently approaching data labeling at this point? You know, what are the range of techniques and the tooling that you have that you would use for that, that you might have seen? And also, I guess, what's lacking in that at this point?

• 22:53Speaker 03

  Yeah, so I think right now there are two ways, basically. First one is using the services. So you just send your data sets to the service company and you get back the labeled data sets, the results. And second one is either building your in-house team or just using your data science team and using the tools to help them do that. So. The problem with the first one with the service companies is that you don't have control over the process, so you just get back the results and then it's your job to verify if results are of good quality or not. In most cases, you don't get good quality results.

• 23:35Speaker 03

  And then again, if you're dealing with a data that requires domain-specific knowledge, usually you can't outsource that easily. And privacy is another issue. With the second one, what we have found out that a lot of companies, they are starting with some sort of, using some sort of open source solution, just to get their data labeled and basically build the first version of their models. And what they find out is that they need to upgrade the tool and tweak the tool to their needs more and more before it becomes this monstrous tool that you kind of don't want and you don't have resources to support anymore. At this point, they're looking for something that is more production-ready and is ready to scale.

• 24:19Daniel Whitenack?

  So, uh, so in terms of like the, the range of things, I imagine that there's, so there, there's a ton of different model types of models and architectures that people use for these different tasks, like you've mentioned sentiment analysis, image classification. I imagine that the burden in different of these model types is, is heavier in terms of, of data labeling. And maybe, Chris, I, I know you've worked in like masking images for, for robot perception and that sort of thing.

• 24:55Chris Benson?

  Yeah, different types of CNNs.

• 24:56Daniel Whitenack?

  Yeah, that's much harder than, let's say, sentiment analysis in, in text where you just kind of say, is it positive or negative? As a result of that, are there types of problems or the types of models that you might want to create where there's already a lot of good data out there that's publicly labeled that you can use, like, let's say for sentiment analysis versus other problems where just due to the nature of how difficult it is to label, you're kind of stuck with doing it on your own?

• 25:23Speaker 03

  Yeah, I think, yeah, so for some problems, definitely, and like for the easier problems, in most cases, you can use transfer learning. So you basically start with a pre-trained model and then you label just a small fraction and you train the model, the transfer learning model with that data. And that works pretty well.

• 25:47Speaker 03

  But in most cases, you get the data, like especially if you're getting the data from the real world, then it's not that easy to use existing models for that.

• 25:58Chris Benson?

  So I guess this might be a good turning point to kind of talk about, you could tell us a little bit about your company and what Label Studio does and kind of how does the company and the product relate to each other and what are they?

• 26:11Speaker 03

  Sure. So yeah, the company name is Hardx and Hardx is a data labeling platform that makes entire data science teams more productive and helps build higher quality, safer and smarter models as a result. And we have open source product that is called Label Studio. So the difference between those is that Label Studio is just the frontend part. So you get the labeling interface where you can upload your data and go item by item and label it. And the company, the commercial offering is basically where you can also

• 26:53Speaker 03

  use our pre-trained models to help to go through the data set faster. You can invite your whole team to collaborate on the data labeling and exploring your data sets. And we have also extensive process for the quality control, helping you to verify that the results that you're getting are actually what you're looking for.

• 27:16Daniel Whitenack?

  Yeah, so this is like Label Studio, you can kind of think as the open frontend that anyone could use, you know, you could just get off of GitHub to help aid you in your annotation if let's say you want to start from scratch. But as you mentioned, starting from scratch isn't always necessary and isn't always practical or efficient, right? So the things that you mentioned, I see you mentioned like auto pre-labeling and native active learning. You already mentioned those things a little bit, but those are the things that kind of augment the processes. Is that right?

• 27:48Speaker 03

  Yep. And what I like about the open source, the Label Studio, it's the first open source data labeling tool that you cannot only download and run, you can also embed it into your own pipelines.

• 28:03Chris Benson?

  Ah, okay, so does that mean like you can run it non-interactively somehow, or how would that work out in practice?

• 28:09Speaker 03

  So in practice, many different ways. So you can use the tool to create the labels, and you can also use this tool to look at predictions of your model. So you can embed this tool into your pipeline, and you can verify what your model predictions are. You can ask your team members, or for example, the main knowledge experts, to provide the labels for specific items in your dataset. So there are multiple ways how you can embed and use it. And it's really flexible in the way how you can define the different types of tasks it can handle.

• 28:48Chris Benson?

  Could you describe some of those tasks as well? Just to kind of give us a sense of what all it can do and what's required for input and output on this?

• 28:56Speaker 03

  Sure. We originally started from the idea that we as the founders of the company, of the company, were coming from different machine learning backgrounds. So I was more concentrated on images and visual data, and my co-founders, they were concentrating on audio and text. So we had this idea about building the data labeling tool that is configurable. And so what we have created is basically a very high-level kind of components that you stick together. Think about it as you are building the webpage using HTML in the same way you are building your data labeling interface.

• 29:36Speaker 03

  It usually takes from five to 25 lines of HTML-like config language. And as a result, you can get data labeling tool that you can use to label audio, images, and text right now. And we will be adding video before the end of the year. And you can do many different things. So all that we have discussed, like basically bounding boxes, named entity, you can do all of them at the same time if you want to. And it's basically depending on the task and depending on the dataset that you have. You can configure it as a Swiss Army knife. You can configure it and tailor it for your particular needs.

• 30:29Speaker 00

  This episode is brought to you by CoopCon, Cloud Native Con, and you are invited to attend this flagship conference from the Cloud Native Computing Foundation, CoopCon, Cloud Native Con, North America 2019. It's happening November 18th through the 21st in San Diego, California. This conference gathers adopters and technologists from leading up a source in cloud native communities. Use the code KCNA PracticalAI19, again KCNA PracticalAI19, to get 10% off registration or check the show notes for a special link to register and a link to the Conventure Boss Letter. Again, check the show notes for links to learn more and register.

• 31:18Daniel Whitenack?

  All right, so every once in a while on this podcast, the topic intersects very nicely with a problem that I'm trying to solve in my own work, and this is one case. So in those cases, as Chris knows, I like to selfishly try to get the guests to help solve my problem on the fly. So one of those problems is I have not found a tool that will let me easily label reading comprehension data. So this is the case where you have like a question and a piece, a passage of text, and then the output that you want is answer that is drawn from that text, maybe a span within the article text or something like that. And there's no tool out there that, at least that I've found, that does that.

• 32:02Daniel Whitenack?

  So with Label Studio, could you kind of walk me through, let's say I came to Label Studio, and what would it take to set up that sort of interface with these Label Studio components? Would that be possible?

• 32:15Speaker 03

  Possible, yes. So I would suggest, yeah, basically installing it, then looking at the templates that we provide. Use the template as a starting point, and then looking at the tags that you can use for your particular problem. So based on your description, I think, yeah, it's very much doable, yep.

• 32:36Chris Benson?

  Yeah, so because I'm a data scientist and back-end person, and I don't know that much frontend, but maybe I've hacked on HTML before. Would that be something I could tackle? Like, how much frontend experience do I need?

• 32:51Speaker 03

  So it depends on how much you want. I would say if there are components that cover your use case, and we right now have around 20 different components, then you don't need any, almost zero knowledge about the frontend development. If you know how to stack up the HTML-like text, then you're good to go. But then, while we were creating the tool, we made it extensible in a way that you can create your own components. So, for example, if we don't have right now support for the video, you can create your own components to render video and connect it to, for example, checkboxes that you can use for classification.

• 33:36Speaker 03

  So in most cases, we try to cover, I would say, 80% of the most common cases with the components that we develop ourselves. But then we also give you an ability to extend it for your particular needs.

• 33:50Chris Benson?

  So if you're, you know, kind of extending that a little bit, if you're a data scientist or an AI developer, and you're trying to integrate Label Studio into your own data pipeline and pull data out for experimentation, how does that integration go? You know, and maybe draw an example or something like, you know, you're using a notebook with TensorFlow or, you know, whatever, PyTorch. And how does that look from a practical standpoint if I'm gonna sit down and use the tool?

• 34:17Speaker 03

  Yeah, so from the notebook example, we will be releasing the package specifically for the Python notebook that will make it super easy to initialize Label Studio inside the Python notebook and work with that. If you're looking into integrating it into your workflow, you would need to install the NPM package. You just need to initialize that with the data from your data sets. And then you create kind of a UI, how to visualize this data and how to label it. So basically, we split up the components that you have in Label Studio into two major ways. Those that are used to visualize the data and those that are used to label it.

• 35:02Speaker 03

  So you can think about it that if you're looking at the text documents, that's visualizing text. And then if you want to put the spans on the text during named entity recognition, that's another tag that is doing just that action. And you're also able to create to load your predictions from your current models if you have those. And you can also complete labeling in a different ways. So for example, if you want to have multiple people look at the same text and let them label it, you'll have two different results and then you can compare those results.

• 35:39Daniel Whitenack?

  And that kind of pushing of predictions into the tool, or let's say like for my training script, I want to pull the latest annotations out. I saw you mention some things about maybe a REST interface or something like that. Or like, how does that interaction work in the sort of the plumbing between the two? Once you've got Label Studio up and running, what's the most useful way of plumbing between an annotation tool and your training and inference in your experience?

• 36:09Speaker 03

  Again, two ways. First one, where we provide just the frontend part, right? It's basically NPM, it's JavaScript package. And you initialize the package and you send the data into the package. So there is basically, there is no API, nothing. It's as simple as that. And there is second part where we provide you with a data manager and we initialize the Label Studio frontend for you. So in that case, you just give us, give Label Studio the JSON-formatted file and we read the data from there.

• 36:47Daniel Whitenack?

  Cool. Yeah, this is still reasonably new in that you were just in the Himalayas thinking about it, you know, a year ago. But actually, it seems like if I'm looking at the GitHub on Label Studio, it seems like there's been some activity there and there seems to be a bit of a community developing. Have you been able to get contributions of components and start to interact with the community in that context?

• 37:14Speaker 03

  Yeah, community response thus far was great. And what people like about the Label Studio is that you can, basically in 10 minutes, you can build your own data labeling tool. That's one. Second one is that the UI is very simple. So because you only use what you actually need for your particular data labeling needs, you only use those components and the UI is super simple. So there is nothing that you actually don't need in the UI. And as I have mentioned, it's the first data labeling tool that you can actually embed into your applications and you can easily

• 37:52Speaker 03

  extend. We are getting, right now, we're getting more bug reports than contributions. So contributions are always welcome.

• 38:02Daniel Whitenack?

  You know, for some reason, that doesn't surprise me.

• 38:06Speaker 03

  Yeah.

• 38:07Chris Benson?

  But it's a good thing. But it's a good thing.

• 38:09Speaker 03

  Yeah.

• 38:10Daniel Whitenack?

  Well, it can be a good thing.

• 38:13Chris Benson?

  It's a track to make things better. So, you know, it sounds like a pretty great endeavor. I'm really, really looking forward to using it myself. And I'm pretty excited about it. So it's been a great conversation from my standpoint. I guess, having brought it this far, all the way from the Himalayas to it is code that the rest of us out here can start utilizing, what do you see as some of the biggest open problems that are still out there in this data labeling space? And how do you see it being augmented in the future, either by your own team or by contributions from outside? What, you know, what's next?

• 38:49Speaker 03

  Um, sure. So I think there are many things happening right now. A few that I want to mention is that I personally think that moving forward, at least some part of the data labeling is going to be commoditized. Just because the models are getting better and better and you'll be able to reuse those models at some point. Maybe not right now, but soon. Then there is another trend with a weak supervision that you can also use to label your data set. And so I think what's coming next is we really need to start putting a lot of thinking into quality control. Because what a lot of companies that I talked to found out is that

• 39:31Speaker 03

  you outsource your data labeling, you get back the labels, they are of very low quality. And as the results, your models are failing in the real world. That's a very common and very concern and happens actually a lot. So quality, how do we verify that the labels are of high quality? Another one is understanding and finding edge cases in your data sets and trying to understand how to label those. That's also very interesting because if you have a real world data sets that consists of millions of images, there is no way you can look into each image by hand. You need some ways to automate that and pick those items for you that needs some attention.

• 40:15Speaker 03

  So I would say, yeah, at least from my standpoint, we're right now concentrating on the quality control a lot.

• 40:21Daniel Whitenack?

  Yeah, that makes a lot of sense. It sounds like there's no shortage of things to explore there and improve upon, but it does sound like actually AI augmentation of the labeling process, especially in terms of quality control and all of those things is going to be really important. So if people are listening and they want to try out Label Studio, they want to kind of follow you on this journey and as things come out, where can they find out more about Label Studio? And maybe get started and try out a few things and also maybe what are some of the great ways that maybe people could contribute to Label Studio? Because you are getting those bug reports and and other things.

• 41:04Daniel Whitenack?

  What are some of the ways that the community can can give back as well?

• 41:08Speaker 03

  Uh, sure. So. I would say the easiest way that you can try out Label Studio right now is npm install label/studio, then labels2.io is our website, and GitHub, yeah, it's github.com/hardex/labelstudio. We have documentation there and some quick guides how you can start very quickly. It's basically a couple lines of command line and you're up and running. With the contributions, yeah, you can open up the list of issues on GitHub and with whatever you want to help us. We ideally are looking at if people are doing some sort of labeling themselves

• 41:53Speaker 03

  and we have not covered that cases yet. Instead of building their own data labeling tool, they can contribute a component to Label Studio. So those contributions are very welcome, particularly in the 3D spaces, in videos. We have some work done on time series, but there is always more to be done. So yeah, that would be perfect ways.

• 42:20Daniel Whitenack?

  Awesome. Well, thank you for taking some time during your travels to deep dive with us on data labeling and talk about a lot of these challenges and a lot of the great things you're doing, HardX and Label Studio. We'll definitely put those links in our show notes so people can find them. And I know I'll be coming back to my team to think about how we can get up a reading comprehension data labeling tool. Yeah, I was just gonna say he's giving me hope for the future here. There is hope in the midst of data labeling.

• 42:56Speaker 03

  A long way to go, a long way to go.

• 42:58Daniel Whitenack?

  Yeah, yeah, still a long ways to go, but it's encouraging to know that people are working in this space and also that there's kind of a modular approach where we can build up components. That's really exciting. So thank you so much for joining us. It was really, really great to talk to you and hope to run into you sometime at a conference or somewhere.

• 43:17Speaker 03

  Thanks very much, Emi.

• 43:18Daniel Whitenack?

  Thanks a lot.

• 43:21Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoy the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change law is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear change law because of Rollbar. Check them out at rollbar.com/changelaw. And we're hosted on Linode Cloud Servers. Head to linode.com/changelaw. Check them out. Support the show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address. Get our

• 44:05Speaker 00

  weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.