Production data labeling workflows

It’s one thing to gather some labels for your data. It’s another thing to integrate data labeling into your workflows and infrastructure in a scalable, secure, and useful way. Mark from Xelex joins us to talk through some of what he has learned after helping companies scale their data annotation efforts. We get into workflow management, labeling instructions, team dynamics, and quality assessment. This is a super practical episode! Featuring: Mark Christensen – Website https://xelex.ai, LinkedIn https://www.linkedin.com/in/mark-christensen-ceo-85795728Daniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Xelex.ai https://xelex.ai/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Data prep is so challenging. It's probably the most challenging part of a project. And it's oftentimes because of the sheer volume of data that is required. Oftentimes, really highly paid and talented data scientists are managing projects in a highly manual way, where their time and talent just isn't being utilized. And I'd say that's probably one of the biggest challenges we hear data scientists describe is that they're spending too much time manually managing project minutia. Oftentimes, it's the use of automation tools and and project management platforms that can help them to kind of refocus their energies on higher level of priorities and allow the the application, the software application of the platform

• 0:45Speaker 00

  to automate a lot of the workflow and allow other team members to manage a higher percentage of the workflow.

• 1:02Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now. If you haven't already, head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com and to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:38Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm normally joined by Chris Benson, who's a tech strategist at Lockheed Martin, but he's doing great tech strategy things and traveling as part of those things. So he won't be joining today, but I've got a really, really wonderful guest and topic to talk about today. We've, we've sort of been diving deep into a number of modeling related. Things in terms of stable diffusion and various things coming out. And I think it'd be good to kind of shift and talk about, again, we're practical AI.

• 2:20Daniel Whitenack?

  So talking about some practical data related things would be worthwhile. And I'm really pleased today to have the CEO of Zellex with me, Mark Christensen. His expertise is all in the area of data labeling and workflows around that and bespoke data processes. So welcome to the show, Mark. It's great to have you.

• 2:42Speaker 00

  Thanks, Dan. Glad to be here.

• 2:44Daniel Whitenack?

  Yeah. Well, could you give us a little bit of a background about how how you kind of got interested in this space of data labeling and kind of producing custom training data sets? And eventually kind of built a built a business around that. How did how did that happen?

• 3:04Speaker 00

  We didn't come out of a data science discipline, actually. We came out of healthcare. And we've spent the last 17 years managing healthcare data at scale, specifically in the area of dictation and transcription. So kind of an entirely different field. But the thing we had in common was managing large amounts of data at scale. In healthcare, what we would do is we would record audio. And for 17 years, we recorded audio from healthcare providers and then moved that audio through an enrichment workflow, which just essentially was transcription. So we'd have skilled medical transcriptionists in the States and around the world, who would take the audio and transcribe it into the completed healthcare note.

• 3:49Speaker 00

  And a few years ago, we met with a friend of ours who was an owner of an NLP company. And he really liked our platform. We happened to be working with him on a speech recognition project. And we really have a need for this in managing trading data for NLP workflows. And so that launched a discovery process that took about two years and we investigated the use case and determined that there was really a really neat fit. And so we modified the application for the next two years and then launched our training data services workflow called Zelle X AI. So that's how we got into it.

• 4:28Daniel Whitenack?

  That's super interesting and I know data specifically in the healthcare space, there's some very interesting restrictions and very specific processes that you have to have to make sure that you're following in that healthcare space. I'm I'm wondering if you think that that perspective on data and like the security, the compliance things around that data, did that sort of shape maybe how you think about handling data for some of these use cases? Any thoughts there?

• 5:06Speaker 00

  Yeah, man, that's a great question and you're totally right. Data security is so paramount in healthcare and my colleague at the NLP company cited that as one of the specific reasons why the workflow that we had in healthcare was a great overlay for data training. The data has to be audited, so data should have a couple of different audit trails on it. Data should be encrypted, both in transit and in rest. Data shouldn't reside on the devices of people that are involved in data labeling. So all those things were just a perfect fit and carryover between our healthcare workflow and an AI workflow. Yeah, you're right.

• 5:49Daniel Whitenack?

  Interesting, yeah. I'm wondering as you, maybe it's as you talk to this NLP colleague or as you've worked with clients around the world, working on data labeling projects. From your perspective, like, how are data scientists most often labeling their data these days and where do they encounter challenges because of how they're approaching data labeling?

• 6:15Speaker 00

  Yeah, I mean, the greatest challenge we always hear, and it's an obvious one, is about getting data that's accurate enough to improve the model, especially in specialty use cases, or, you know, let's say, new language modeling, where a clickworker approach doesn't, it doesn't hold up, it just doesn't work as well. And for that reason, for smaller projects, maybe, you know, the size of a few hundred to a few thousand data objects, a lot of our clients try to do the work in-house, just for the sake of, I'd say primarily, you know, retaining, for the sake of quality control. But for larger projects, it's just too hard to do

• 6:59Speaker 00

  everything in-house, and so it winds up being a combination of in-house team members and outside vendors doing the data labeling. For our purposes, on the approach that we took, we decided that rather than commoditize the role of the editor or the annotator, we'd invest more in training and compensating our labelers as a means of building long-term relationships. And for us, we found that's an essential part of maintaining the kind of the consistency of the data quality and making sure that the data quality remains at the accuracy levels our clients require. And that's to be able to have those relationships with annotators that we can trust and that aren't just commoditized relationships.

• 7:44Daniel Whitenack?

  Interesting, yeah. So have you encountered cases where maybe clients come to you and they say, hey, we tried to throw up like a crowdsourced task and get a bunch of labels, we invested a lot in that, and then, and then, you know, it didn't really help us that much. Do you think that those, those cases are maybe due to like unclear instructions to the labelers, or a sort of like variety in like the motivations of those data labelers? Or what do you think kind of leads to some of those, some of those quality issues from your perspective?

• 8:29Speaker 00

  Yeah, the commoditization of the annotation workforce is, I think it can be a project killer.

• 8:37Speaker 00

  And a very high percentage of projects that launch stall and never complete. And that's one of the key reasons. We've talked with companies that try that approach and they wind up iterating the data so many times to try to get an accurate set of data that they can use, that they ultimately wind up going to a more bespoke approach where the teams are more handpicked and more highly trained, even though the costs are higher, in order to finally wind up with a dataset that is useful. So yeah, I think that is one of the key problems that plague data aggregation projects.

• 9:18Speaker 00

  And that is to wind up with a clean set of data that can be done kind of on time and on budget.

• 9:25Daniel Whitenack?

  Yeah, I know, Mark, that, so like in our projects, and we've done some speech projects as well, like one, we've struggled with this also in terms of like the data quality. And I remember in one case, like really, we were saying, well, we need five labels for each, for each sample, because like the variability between labelers is such that like we need either a majority vote, or we need to analyze how much they agree one labeler to the other or something. And of course, that gets, that gets really expensive over time. Could you speak a little bit to, like, you mentioned this training, focusing on training

• 10:08Daniel Whitenack?

  and kind of bringing in this, like upskilling these data labelers. What does training annotators look, look like in your, in your projects? And what maybe have you learned about what's important as you, as you are training data labelers?

• 10:25Speaker 00

  Yeah, I recently did a paper called Improving Model Accuracy Through Better Translation, and it was really just an attempt to lay out some tips for translating source texts for natural language processing models.

• 10:39Speaker 00

  And one of the items I mentioned, and it's something that we've seen as we've worked with teams around the world doing language projects, is that it's important for the editors and those involved to understand the use case. And, you know, that might seem like it's perhaps too much information or an unnecessary amount of information to share with the editors or the annotators, but once they understand the project description, or I should say the better they understand the project, oftentimes it really does translate into higher quality data. And so I encourage companies to share that information with annotators so that they

• 11:19Speaker 00

  are more vested in the work that they're doing. And as an example of, you know, how a project description might be written up as part of the guidelines for the annotators. It might be something like, "This project involves," and this, I'll just cite briefly a paragraph out of the document, "This project involves training a software application to automatically assist call center agents with their tasks and increase their efficiency. For example, if the customer says, 'I have a warranty issue,' the agent's software application can respond by automatically opening the customer's warranty clause, reducing the time required for the agent to assist the customer. The translation project consists of a set of English language scripts that reflects

• 12:02Speaker 00

  some of the typical conversations that occur between call center agents and customers. The purpose of this project is to translate those scripts into a target language in order to add NLP-driven process automation into the call center's workflow, thereby adding new efficiencies to the agents and company." And so by giving those, that kind of insight and detail to the translators and the editors, it enables them to Have more buy into the project and have a better understanding of how their work is going to be used.

• 12:36Daniel Whitenack?

  I think that there's a lot of content about like hyped AI data science things, but in reality, what people are really wanting more content around is this sort of practical concerns of like, hey, my data labeling isn't actually working, right? Like, how can I fix that problem? And so I think there's a, from my perspective at least, there's an eagerness for this sort of conversation where people are actually, they have a lot of the other, but they don't have enough like practicality in their, in their content. So I, I think that that bodes well for the sort of conversation from my perspective.

• 13:18Speaker 00

  Okay, that's encouraging to know because we're the guys on the process side, you know, so the sexy work is being done by you and the data scientists. We're more the guys down the boiler room.

• 13:30Daniel Whitenack?

  Yeah.

• 13:30Speaker 00

  You know, we're kind of the, the operations team that makes the process happen, but doesn't necessarily know a whole lot about the data science side of it.

• 13:42Daniel Whitenack?

  Yeah, I think that in reality though, the data side is what is driving things. So yeah, I think that's good.

• 14:35Daniel Whitenack?

  Well, Mark, we've talked a little bit about the importance of, of training. Annotators, we've talked a little bit about kind of specific data concerns around healthcare and other things. I'm wondering, from your perspective, since you're really plugged into the area around like how people are managing their data workflows, how they're managing their data labeling. From your perspective, what does the current data labeling, sort of annotation and tooling landscape look like? Like, what choices do people have? And what does that landscape look like right now?

• 15:10Speaker 00

  From my perspective, the landscape seems to be rapidly changing. But I would say that off-the-shelf models are being used more often. They continue to improve, and they're used either, I'd say, as-is or with in-house tuning. The projects we see, and the projects we're getting more involved in, are kind of specialty applications where off-the-shelf models aren't accurate enough, or, you know, they don't exist. And cases might be places like medical documentation labeling, sentiment and intent projects that have a highly customer-specific language or vocabulary that can't be picked up by off-the-shelf models. And in specialty models, I'd say, training data

• 15:54Speaker 00

  is needed in cases where unique vocabularies, you know, warrant kind of highly specialized, bespoke model tuning. And an example might be gathering business intelligence from call center interactions, for example, where, you know, the client is seeking to obtain business intelligence through an NLP automation process, and they need a model to be custom tuned. And they need the model to be custom tuned to meet their business objectives. Another area would be, I guess, new language modeling. And that's exciting to me and encouraging, because we're starting to see an uptick in interest in other major world languages where models don't exist

• 16:36Speaker 00

  in a production environment. On the tooling side, I'd say, we've seen companies, both big and small, relying on a hybrid of in-house data labeling, and in-house plus clickworker-driven labeling, and fully external third-party labeling. But what we're not seeing is AI companies that have systems in place to manage those different approaches in a cohesive way. So there's a lot of manual aggregation. There's a lot of one-off coding that gets done to unify the results from those hybrid sources. So to answer your question on the tooling side, this is one area where the tooling is broadly not keeping pace with the growth of the industry.

• 17:19Daniel Whitenack?

  And I know it's like one thing, and this comes from personal experience. It's one thing to get data labeled, like gather a label. It's another thing to develop like a workflow around that that's integrated into your systems, integrated into your backend. What do you think are the challenges facing data scientists around this workflow side of things and the bespoke sort of things that they have to do to integrate data labeling into the sort of wider set of things that they're doing?

• 17:53Speaker 00

  Yeah. Data prep is so challenging. It's probably the most challenging part of a project and it's oftentimes because of the sheer volume of data that is required. And what we see, not just at small companies, but even at big companies is that highly skilled and oftentimes really highly paid and talented data scientists are managing projects in a highly manual way where their time and talent just isn't being utilized as efficiently as it could. Senior data scientists are doing things like vetting samples from annotators and doing quality scoring on annotators. And I'd say that's probably one of the biggest challenges we hear data scientists describe

• 18:33Speaker 00

  is that they're spending too much time manually managing project minutia. And oftentimes it's the use of automation tools and, and project management platforms that can help them to kind of refocus their energies on higher level of priorities and allow the, the application, the software application of the platform to automate a lot of the workflow and allow other team members to manage a higher percentage of the workflow. So I think that's, that's one of the things we're seeing.

• 19:07Daniel Whitenack?

  And along with that, how does Zellex specifically approach the, the data labeling problems that you've described? We've talked about sort of workflows. We've talked about the, the custom setups that are needed for certain tasks. We've, we've talked about a variety of things. How has that filtered down into your approach specifically and, and the approach that Zellex takes?

• 19:30Speaker 00

  Workflow platforms are all about moving off of spreadsheets and manual processes into processes that scale better. That's what we do. We're focused on the production process. So, everything from training and managing the skilled labor to meeting deliverables on time and at quality levels that clients expect. I mean, keeping projects on budget. Those are all things that training data services companies like we do and that we bring to the table. Our focus is on making complex workflows easier. And another part, this was interesting, that one of our clients said to us one time, is that they needed all of the stakeholders at the company to be able to see what was going on with the

• 20:15Speaker 00

  project. And the platform, our platform, enabled them and other platforms too. It enables stakeholders to do that. And there's all kinds of stakeholders at the production level and the commercial level for projects. Because projects on the commercial side, you know, they're typically done on the request of a client and in service to a client. And so, all kinds of different people outside of the data science team are involved. You know, the sales team, the ops team, the procurement team, the quality assurance team. And everybody needs to know what's going on. They want to see if the project's on budget. They want to see if the project's on time. They want to see if the quality thresholds that the

• 20:57Speaker 00

  client has set are being met. And so, a platform gives everybody that kind of visibility. And I really enjoy and appreciate being able to do that for a company because it does then keep all the stakeholders in the loop. At the same time, it allows the data science team to not get bogged down managing minutia manually. So, that's one of the neat things that we like to deliver. And then on the services side, the approach that is always about managing the workforce successfully. And, you know, success in data sciences and in projects like this is measured in being able to deliver a project on time and on

• 21:41Speaker 00

  budget and at the accuracy levels that have been determined or set by the client and by The service provider as being the goals of the project's objectives. An example of this, kind of of how this can backfire is when service providers like us enter on, let's say, a new language or a new project area and, you know, maybe their client has come to them and said, you know, can you do this? Or can you do a project in data labeling in this language? And of course, the knee jerk reaction is always, sure, we can do that.

• 22:19Speaker 00

  But if saying we can do that involves hiring a third party vendor in that target country or target language to do the project, and it's done in a scramble, it can really backfire. And so hiring a third party vendor in cases like that can result in kind of a black box approach where you're unable to adequately measure quality and where you're unable to adequately manage deliverables. So the projects wind up running late or projects are delivered with poor quality data. And then you're left kind of scrambling to do those corrections on the data internally or finding another source to do those corrections for you. And it's a recipe for disaster.

• 23:03Speaker 00

  So for us, kind of the way we mitigate that is when we move into a new language, for example, the initial step is to do the hires and do the training ourselves so that we have our own team and we're not dependent on a third party vendor source for that labeling effort. And that way, even though it's, it's going to take us longer and the cost might be higher, and there are cost sensitivities that are realities, but the truth is, if you're using a third party vendor and working out of a black box, chances are you're not going to be able to deliver the project on time and at cost. And so your costs and timeline are going to be affected anyway.

• 23:45Speaker 00

  And so we've opted for taking an approach that's more expensive to our clients, but that ultimately delivers projects with a higher quality and consistently higher quality data that are on time, that meet the turnaround deliverables, even though the price might be a little higher.

• 24:02Daniel Whitenack?

  Yeah, and I think that's a really good and practical advice for the whole community that's trying to do a variety of these data labeling projects is really kind of at the start of these data labeling projects, not only thinking about gathering samples, but thinking about how is your workflow going to be managed and how are your annotators going to be trained? Because thinking about that stuff upfront and taking time or spending more money on getting that in place from the start might actually save money in the longer term if you're not doing as many iterations of labeling, right? If you start and you do a bunch of labeling and then you don't get the quality that you

• 24:45Daniel Whitenack?

  need, then or you get some sort of unexpected biases or other things in your data that could cause more problems down the line. And one of the things, maybe this isn't a specific, I guess it could lead to specific quality issues, but one of the things that is hard for me as a technical introvert person who's not maybe the most people oriented person in the world is thinking about all of the team dynamics that happen on a data labeling project and setting up like maybe a disparate and kind of distributed set of labelers and vendors for a data labeling project.

• 25:28Daniel Whitenack?

  How can the problems associated with those sort of dynamics be addressed in this kind of online distributed labeling environment?

• 25:36Speaker 00

  Yeah, you're totally right. There are inherent challenges in managing an online workforce, but many of those can be mitigated through kind of a well-developed, robust workflow application. You know, things like centralized controls, giving managers total visibility to what's happening in the workflow at any given moment. You know, the status of data objects as they're moving through the workflow and And how you're doing against your timeline for deliverables. You know, those are the kind of things that software is really good at managing. As I mentioned earlier, we've seen cases even with really large companies where

• 26:19Speaker 00

  pretty complex projects were still being managed on a spreadsheet. And when you're doing that, there's almost no ability to manage the workflow effectively.

• 26:50Daniel Whitenack?

  Well, Mark, given the sort of team dynamics that can happen that we've been talking about, this sort of variety of tasks that Zellex is exploring and other people exploring in the space from like standardized machine learning tasks to more custom ones. I'm wondering what sort of like would you say about kind of proper ways to set up maybe manual and/or annotated QA type of workflows associated with your data labeling?

• 27:21Speaker 00

  Well, I can tell you a little bit about what we do. The first is to establish the ground truth version of the data object. And for all data objects as they're moving through the workflow. And once we establish the ground truth data object, then we're able to measure the distance between that and the work that the editors are doing. And that helps to generate a whole lot of different metrics for us. You know, who needs additional training, how pay might be affected, how our costs are effective if data objects are moving through the QA workflow, more for some editors than others. The second thing that we do is a multi-level QA workflow so that work gets automatically routed. And that Could be in cases where we've got new hires or maybe editors are being flagged via our auto-check

• 28:06Speaker 00

  process for certain error types. And then thirdly, we run an error script that dynamically checks against a known error list so that those items are routinely kind of recycled through the workflow to be re-edited and re-QA. So those are kind of some of the typical things we do. Judgments, of course, and multiple judgments on data objects is really important to make sure that using multiple layers of judgments is also important. And we do that through the QA workflow process as well.

• 28:36Daniel Whitenack?

  It's been extremely helpful for me to think through some of the dynamics and the workflows associated with data labeling. I think it's extremely practical and very useful. I'm wondering, as you kind of continue to be more and more involved in this space of data labeling and interacting with clients in the data science and AI space, what excites you about the future of sort of data science and AI practice? And, you know, what, maybe within that, what could easier data labeling enable in the longer term?

• 29:10Speaker 00

  Well, when you look at the number of data sets that have been developed and models that have been developed so far, it's overwhelmingly all English-based and in that regard, it's probably largely focused on the US market. And, you know, we're overwhelmingly the largest economy in the world, so that that makes sense that it would be that way. But what I'm excited about is seeing the tools and expertise that have been developed in English modeling to now be used in other major world languages and specifically in developing economies where AI can be used to help developing economies move forward. All of those nations are generating customer and employee experience data in the form of things like, you know,

• 29:55Speaker 00

  Like customer behavior data and online reviews and, you know, sentiment and intent data and all medical data, things that are in unstructured format where AI can be used in a beneficial way.

• 30:08Daniel Whitenack?

  Well, Mark, I'm really happy that you brought up this side of the impact of data and NLP across the world's languages. As our listeners will know, I'm very passionate about this topic and I'm really excited anytime we get to talk about that. It's something that excites me for the future as well. I've really appreciated you taking time out of your work with Zellex to help us parse through some of these data labeling challenges and the workflows associated with them. And yeah, really appreciate you taking time and looking forward to continuing our conversations over the coming months as I hit my own data labeling issues.

• 30:51Speaker 00

  Thanks, Dan, very much. I really enjoyed it.

• 31:02Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.