#298 Ryan Kolln: How Appen Trains the World's Most Powerful AI Models

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. How do the world's most powerful AI models get trained and trusted at scale, and what does that really take from data to deployment? In this episode, Appen CEO Ryan Kolln joins Eye on AI to unpack how rigorous human evaluation, culturally aware data, and model-based judges come together to raise real-world performance. In this episode of Eye on AI, host Craig Smith speaks with Ryan Kolln, CEO of Appen, about building evaluation systems that go beyond static benchmarks to measure usefulness, safety, and reliability in production. They explore how human raters and AI evaluators work in tandem, why localization matters across regions and domains, and how quality controls keep feedback signals trustworthy for training and post-training. Ryan explains how evaluation feeds reinforcement strategies, where rubric-driven human judgments inform reward models, and how enterprises can stand up secure workflows for sensitive use cases. He also discusses emerging needs around sovereign models, domain-specific testing, and the shift from general chat to agentic workflows that operate inside real business systems. Learn how leading teams design human-in-the-loop evaluation, when to route judgments from models back to expert reviewers, how to capture cultural nuance without losing universal guardrails, and how to build an evaluation stack that scales from early prototypes to production AI. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Novel in the L L N space is you've got a large number of models all serving a similar purpose, which is to bring a lot of context and knowledge around all of the information we have into a way that can be easily summarized and presented to humans. So I think for the first time is this real reliance on benchmarking to understand which models perform any better than others. But I think as we've seen through some of the benchmarking, even though there is a particular high score on a benchmark, that doesn't always translate into a good user experience. For most of our customers who are operating at a global scale, the evaluation

• 0:41Speaker 00

  plays a really important part into bringing that cultural nuance and context of how the world operates.

• 0:48Craig S. Smith?

  Build the future of multi-agent software with Agency. That's A G N T C Y. Now an open source Linux Foundation project, Agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on Agency, including robust identity and access management that ensures every agent is authenticated and

• 1:29Craig S. Smith?

  trusted before interacting. Agency also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. Agency is dropping code, specs, and services, no strings.

• 2:10Craig S. Smith?

  Attached. Visit agntcy.org to contribute. That's A G N T C Y dot O R G.

• 2:20Speaker 00

  Hello, my name is Ryan Collin. I'm the CEO at Appen. I've been at Appen for over seven years now, focusing on growing the business in the exciting world of AI data annotation. Prior to Appen, I have a background in engineering, specifically in data requirements for telco networks. I've done very large scale data annotation and modeling around network optimization for telco networks. But I'm very excited to be here today and talking about how Appen's supporting the development and evaluation of L L N networks.

• 2:58Craig S. Smith?

  Can you start for listeners that aren't that familiar with the space? I mean, for a long time, L L M's, well, I don't know this to be true, but it seemed like everyone was relying on performing to certain benchmarks. But quickly, people realized that that's a very narrow way to evaluate L L M's. And it can certainly be gained. You can train to meet those benchmarks or exceed those benchmarks.

• 3:37Craig S. Smith?

  Can you talk about kind of the evolution of evaluation? And as I said, I'm particularly interested in the role of human evaluators.

• 3:49Speaker 00

  Yeah, it's a very interesting market. And I think, you know, what's become, you know, novel in the L L N space is, you know, you've got a large number of models, all serving a similar purpose, which is to bring a lot of context and knowledge around all of the information we have into a way that can be easily summarized and presented to humans. So I think for the first time, is this real reliance on benchmarking to understand which models are performing better than others. But I think as we've seen through some of the benchmarking. Even though there is a, you know, particular high score on a benchmark, that doesn't always

• 4:33Speaker 00

  translate into a good user experience. Um, and I think the, the reality is, is the benchmarks,

• 4:43Speaker 00

  they're static and typically quite narrow. Um, therefore, it's really difficult to get one benchmark that even within a narrow domain that covers all of the edge cases. I think the other issue is that, um, the ways that humans use these models evolve very rapidly, the model itself evolving rapidly. Um, so while the benchmarks play a really interesting, snapshot into model performance, I don't always think that it's the best, uh, way to really get a measure of how good the user experience is for the models.

• 5:18Craig S. Smith?

  That leads us into about L L M evaluation of other L L Ms and human evaluation. Can you start with human? I mean, that, that was part of the original, uh, development of, of these L L Ms with, uh, reinforcement learning with human feedback. Uh, but it's gone beyond that, hasn't it?

• 5:40Speaker 00

  Yeah, that's right. And, um, even before the L L M, uh, model developments over the last kind of five to eight years or so, evals for large, you know, human based evals for large models has been going on for a very long time. And Appen has played a, a big role in this. Um, so there was a type of human annotation called, uh, relevance. And really what it was is providing feedback, subjective feedback into how the large models run by the very big search and social media companies were performing. And these are the core models that are related to for a given search, uh, uh, term, what are the right search results that are being shown?

• 6:26Speaker 00

  Or for a given user experience, what is the right ad that is to be shown? And human feedback into the model development lifecycle um has been operating at really large scale for a long time, but it's just a it's a small niche um within a few very large technology companies that's been um riding on that. And it's probably helpful for me to explain what an evaluation looks like um or what search relevance looks like. And it's a fairly simple but subjective task, which makes it complex. So the um at a at a really high level, for a given prompt, here are two responses. Which one do you prefer and why?

• 7:11Speaker 00

  Right? And the type of questions that we ask our our annotators and our workforce really are dependent on what the customer is trying to do. Sometimes they'll be looking for how accurate was the response? Was it formatted in a right way that is, you know, can be easily digested? Is it grammatically correct? Um and then you get into, you know, was it harmful? Was there any biases contained? So you can think about all of the different subjective elements that a model builder would want to know how well their model's performing is up for grabs in in these evaluation tasks. And then what can be really important also is typically there's a follow on question. Why did you think that way?

• 7:54Speaker 00

  Or what would have been a better way to answer that question? And it serves two purposes. One is that it's used to measure the performance of how the models uh uh are performing. But also it's used in in oftentimes in the retraining process to improve the model performance going forward. Now, what is the unique part here is um for most of our customers who are operating at a global scale, the evaluation um plays a really important part into bringing that cultural nuance and context of how the world operates. Um so I'll give an I'll give an example which helps highlight this.

• 8:38Speaker 00

  Recently, we're asked to find uh people in our network who were experts in parenting advice, right?

• 8:46Speaker 00

  And if you find someone who's an excellent, you know, parent in the US, they may give very different parenting advice to what is done in the Middle East or in India or in Africa. So there is this cultural context into the evals that is incredibly important. So the models work and are accurate and are giving great advice, um, not just based on a a Western point of view, but on a real global point of view, and it brings in that cultural nuance.

• 9:17Craig S. Smith?

  That's interesting. And and does that then train the model to recognize what culture the customer, the person interacting with the model is coming from so that it answers in a culturally relevant way? Or or is it averaged out, which seems to me may be problematic as well, because you end up with just extremely general, not very useful responses?

• 9:45Speaker 00

  Yeah, the the cultural context is really important, and the idea is that it is addressing, um, you know, the requirements with within a culture. But then there is also a a base overlay around, you know, what is the right thing to be doing at a humanity level? So it is trying to get the balance right between how do we want the models to be behaving at at a generic view across across all cultures, but then capturing that nuance within within a specific, um, uh, geographic or demographic setting.

• 10:21Craig S. Smith?

  Yeah, are you working with, uh, sovereign models in countries outside the West so that they're trained to their cultural viewpoint? I mean, that's, of course, one of the big complaints, uh, among smaller company countries that they're they're getting models that are trained to a US viewpoint.

• 10:46Speaker 00

  Yeah, it's a it's a really good example, and the rise of the sovereign models are really addressing this issue that I've spoken about where it's like it's it's, um, the cultural context isn't there.

• 10:58Speaker 00

  The there's an interesting set of developments and, you know, we work with with some of the specific, um, you know.

• 11:06Speaker 00

  Model developers in different regions. I think for our side, though, where the bulk of our customers are the large technology companies. Um, but we do have, uh, Appen is unique in the industry where we work with both the very large US L L M model builders and the Chinese model builders, um, which represent kind of the most innovative, uh, model builders on the planet. Um, and there is, and we do see some differences in there and it's, it's not always cultural. I mean, that's a big part. But we also see, um, it's derived by the distribution models. So in China, we see work,

• 11:47Speaker 00

  uh, more work related to medical and financial, um, advice. And that's driven by the super apps, um, where you go in and it will give you medical advice for a, um, to a consumer. Um, and that's a really big part of, of the ecosystem there. Where in the West, you don't typically go to your smartphone for medical advice. At this stage, you go, you go to your G P. So there is a different, um, also set of data requirements instead of evals based off how consumers use technology in the different, um, in different environments also.

• 12:24Craig S. Smith?

  So does that mean that, uh, with the human evaluators that you have uh, teams from each culture that you're working in, you, you, you're not using a a global team or a U S based team to evaluate or, or tune Chinese models? Or are you? I don't know.

• 12:45Speaker 00

  Yeah. So, um, there are two parts to what our customers in China want. So there is support for the domestic applications. And then a lot of our customers in China, technology companies have global customers, right? And, you know, there's, there's some really good examples on social media, also on e-commerce that is a very big global footprint. So we're helping both with the, the, um, domestic and the international, um, data requirements for, for the China customers.

• 13:17Craig S. Smith?

  Uh, you, you know, OpenAI and, and, um, it's the one that comes to mind, uh, but certainly in some of the other. Uh. Models, consumer facing models periodically, you'll get two answers and be asked to choose one or the other. To me, that's a a pretty flawed way of. Uh. Because uh uh a lot of times I'm busy and I just pick one just randomly because I don't want to have to read both. Uh, it it. Yeah. Is that useful that consumer facing uh uh?

• 13:59Craig S. Smith?

  Effort to to have them show a preference.

• 14:03Speaker 00

  Okay, I think it's um. You know, when when looking at the evaluation of the models and how they're working in the data that's used to retrain. It's um. There are many, many different sources that are used, all right? Um. The benefit of going with uh an approach like what Appen provides, we are, you know. Qualifying people, we know their exact demographic backgrounds, their exact interests. Um. We know that they're going to sit down and pay attention to that task and not just click option A every time. So there's a lot of um. Precision in the data that we're providing, albeit with very subjective feedback.

• 14:42Speaker 00

  Um. And I think that, you know, looking at, you know, the model builders look at things like the example that you gave about Open Eye. But also engagement rates, click rates, the way that they respond to the question. So there's an ensemble of. Data inputs that are provided to improve the performance and you know what companies like Appen provide is a really important part of that.

• 15:04Craig S. Smith?

  And and what you're providing is is a very focused. Initially, I mean, we'll talk about evaluation of other L L M's, but but uh on the human side, you're providing, as you said, uh. A focused and trained workforce. Uh, with instructions of what they need to be evaluating. How how many people do you have in your?

• 15:31Craig S. Smith?

  Database of potential evaluators, it must be massive, or maybe not. I mean, maybe that's my other question. How many people do you put on a model?

• 15:43Speaker 00

  Yeah, so we have millions of people in our database and getting access to people isn't typically our challenge.

• 15:53Speaker 00

  The, you know, it comes through finding the right people who are going to deliver the demographic requirements for what uh is required by the customer. And then the ability to ensure that they are creating data that meets the requirements of the customer. Um, and I'll give an example, like if if we had millions of people and they were only based in the US, we would have a very US centric based workforce. The benefit that happens brings is we've got people in, uh, I was just looking the other day, I think it was one country where we didn't have someone in our database on the planet. And we've been doing this for a very long time and and the the art of what we do is

• 16:39Speaker 00

  running a project where the customers were asking us to, you know, connect with people in a hundred countries in parallel. Let's speak a hundred different languages and have a hundred different, um, you know, ways of interacting with us, payments and and support, et cetera. But it's, but the customer wants a common framework around how they're going to, um, the type of questions they want the contributor to be asking, how they're going to understand whether their output is high quality or not. So the ability to translate this is single task into a hundred different um, cultures and a hundred different languages.

• 17:19Speaker 00

  And then ensure that the, you know, workers are um really attuned to the nuance of the task that's that's being asked of them.

• 17:29Speaker 00

  It's um, it's a non-trivial task and you know, something that, something that we, we do and we productize and we have a, you know, a great set of infrastructure to be able to do that.

• 17:39Craig S. Smith?

  Yeah, and that's interesting. So you have uh, human evaluators evaluating the evaluators.

• 17:45Speaker 00

  That's right. So, yeah, and, and um, quality is a really interesting topic for human uh, data, alright? So when we have someone evaluating the model of performance, we need to check that um, they're not just choosing option A every time, alright? And there are many different techniques that we've got to ensure that um, they're providing the right kind of effort and, and the right quality into that data. The other issue we face a lot, Craig, is that um, you know, there are people who um, try to be in a different country to what they are. Or try and, you know, say that I've got these skills when they don't.

• 18:28Speaker 00

  So we have a pretty robust process, highly automated, to verify if people are in the location where they are. Are they humans? Um, do they have the expertise that they want? Um, and that's, that's again, some of the things that we've built up over doing this since uh, you know, over twenty plus years. We've got a lot of expertise that we've built into our products to, to make sure that we're getting high quality data.

• 18:53Craig S. Smith?

  And typically uh, of, to evaluate a customer's model, well, how large of an evaluation team are you fielding?

• 19:04Speaker 00

  Yeah, it, it's a good question. It does vary. Um, and it varies about, you know, the, the task type, that it varies about the, you know, the global reach. Um, but it's not uncommon for us to have um, you know, thousands of people.

• 19:23Speaker 00

  On a specific model eval, working on an ongoing basis um at at any given time. So um the numbers can get very large very quickly. We also see there can be bursts of of data. Um, you know, a customer uh recently asked us to find twenty thousand people to evaluate the performance of their model. Um, and that's, that's, that's a lot of, that's a lot of humans.

• 19:52Craig S. Smith?

  Yeah, and uh um it is so Appen has this database of people that you've recruited and and put through some evaluation, both automated and human. And then when you uh wanna pull together twenty thousand, uh I imagine a lot of that's automated. But how then do the people interact with the model? Do you you just point them at the model or do they interact with the model through an app and uh website or app?

• 20:29Speaker 00

  Yeah, but a great question. So there's two parts to our technology stack. One is uh the you can think about as workforce management. And that's where we have, you know, the database of crowd workers, the ability to go and verify and, you know, and put them through different qualification steps to understand what they're going to be good at. And also to advertise the tasks to them. So we have um, you know, a uh CrowdGen is our brand from our crowd. And people go in there and they see what tasks are available to them. And if we find someone, um if we find a project that matches someone, we will notify them. There's some great earning opportunities for you. So that's, that's one side of it. Um, the other side is the data annotation platform.

• 21:15Speaker 00

  And this is exactly what you're talking about, Craig, where they come in and complete the tasks. And Appen has an exceptionally flexible um and powerful data annotation platform. And the reason why it needs to be flexible is because every task that we get from our customers is a little bit different. The question might be different, the scoring, et cetera. So we need to create very modularized um components for the our crowd workers to come in and complete the task. The other thing what's really powerful about the annotation platform is the ability to check for quality in real time. So I'll I'll give an example.

• 21:59Speaker 00

  If the task is evaluate the performance of this model and rewrite the response in a numbered bullets, so one, two, three, four, five. We get the we have a lot of AI checks that run in parallel to what the contributors are doing. And it could be as simple as was the formatting correct? Was it on topic? Yeah, was, you know, we get into specific things like spacing and and, you know, grammatical checks. So there's a there's a lot of AI that we're running in parallel to the task to make sure that the quality is where we need it to be.

• 22:38Craig S. Smith?

  And and Appen is publicly listed in Australia. When was it founded?

• 22:46Speaker 00

  Yes, I founded in ninety ninety six and the founding story is an interesting one. And we've been doing AI data work since the since the start. So there was a company in ninety six called Nuance. I think they got acquired by Microsoft at some point. They were building a speech recognition model where you could call up and place bet on horse races.

• 23:13Speaker 00

  Right? And they built that in in in the US and this was in ninety six, so it wasn't called AI back then. And they thought that Australia would be a next great market to roll this out to. And they rolled it out in Australia and it didn't work because it couldn't understand our accents. So they called up our founder, who was a professor of linguistics at the time at a university in Australia. And said, can you go and help us collect Australian accented English? That was used then to train the speech recognition model so people could place bets on horse races.

• 23:46Craig S. Smith?

  No, that's fascinating. Did did she? It's a woman, right?

• 23:50Speaker 00

  Yeah, that's right. Julie Von Willough.

• 23:52Craig S. Smith?

  Did she use Mechanical Turk or or what? Did she use an exist existing crowd working platform?

• 24:02Speaker 00

  No, I I don't think Mechanical Turk existed at this point in time. So this was pre the data annotation industry and really um you know, at the tip of the spear of the foundation. So Julie went out and found people and brought them into a recording booth and recorded them speaking Australian accented English um for the specific requirements of of nuance at the time. Um, and you know, we did some really great work early on and that snowballed into you know, really supporting um uh the development of speech recognition models. Uh we did a lot of work with

• 24:42Speaker 00

  IAPA and DARPA following September eleven around building out models that had a specific focus on um Arabic languages in all of the different dialects. Um you know, it's interesting about Arabic is that um depending on the location, it can be spoken very differently, but it's all written the same. So we app and developed a way and it was um very innovative to phonetically transcribe colloquial Arabic that was then used to, you know, build into the speech recognition models. Yeah, um so following that, we we did a lot of work with Amazon Alexa early on

• 25:24Speaker 00

  uh to support the development there. And really, the focus of app and it was it was fairly focused on speech uh models at the time. And we built out this global network of people around the world to get um language and dialect um accuracy. Um, and then that's what, because we had this big network all around the world, some very large technology companies came to us and said, hey, that would be great if we could leverage that to evaluate the performance of our global models. So that was the the evolution of app and from speech into relevance. And then from then, we've done a lot of work in image annotation.

• 26:07Speaker 00

  Continue to do a lot of work in speech and then more recently clearly for Jennifer AI.

• 26:14Craig S. Smith?

  Build the future of multi-agent software with Agency. That's A G N T C Y. Now an open source Linux Foundation project, Agency is building the Internet of Agents. A collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on Agency. Including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 26:57Craig S. Smith?

  Agency also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than seventy-five other supporting companies to build next-generation AI infrastructure together. Agency is dropping code, specs, and services, no strings attached.

• 27:36Craig S. Smith?

  Visit agntcy.org to contribute. That's A G N T C Y dot O R G. I was gonna say that on the mechanical Turk thing, that Fei-Fei Liu built the ImageNet data set that then was used to validate convolutional neural nets and set off the AI supervised learning AI craze. That spawned a whole industry of data annotators. I was close for.

• 28:10Craig S. Smith?

  A few years with a company called Label Box, very bright guys who have a platform. And I think I had Appen on the podcast at some point talking about your data, data labeling system. But is did during that supervised learning phase, not that it's over, but before transformers and generative AI took over, was was that a big part of Appen's business?

• 28:47Speaker 00

  Yeah, it certainly grew to be an important part of the business. We did, you know, work in computer vision annotation and still do. So certainly that, you know,

• 29:01Speaker 00

  the the ImageNet and the FAI and the deep learning kind of evolution was really great for image recognition models. There's also neural nets used in the speech model development, which really spawned a next wave of the data collection and annotation for the speech models also.

• 29:22Craig S. Smith?

  Let's talk about you. You've built this a network of evaluators, human evaluators. You have a system that that they work through to evaluate responses from from AI models. This is for fine tuning. Is that right? Or is it for the foundational training?

• 29:46Speaker 00

  It's mostly for in an LLM context for post-training. So there are post-training and eval. So, you know, in the LLM model development, there is pre-training, post-training. Pre-training is hoovering up, you know, all of the available.

• 30:04Speaker 00

  Copy on on the internet and and books, et cetera, highly curated. But it doesn't bring in the the human context. Post-training is making sure it's bringing in how the models that have this great knowledge wanna be able to convey that to to humans.

• 30:21Craig S. Smith?

  And is is that part of the RLHF

• 30:27Craig S. Smith?

  cycle, or is it after that?

• 30:29Speaker 00

  That's it's part of the RLHF cycle. So a lot of the evals now, there's a shift from reinforcement learning with human feedback, which was effectively providing examples around how the output should look to reinforcement learning with verifiable rewards. So this is more evaluating at a at a rubric base, so with a scoring system, how well the model is performing, which is then used to create a reward model which guides the the output. So there is an eval component that's used is a really critical part of post-training. And then there is evals which more just like how well is it performing,

• 31:13Speaker 00

  which may or may not be used to train the reward models.

• 31:17Craig S. Smith?

  I see, and that that the reward model is you're you're not talking about fine tuning. You're talking about post-training, right? Before the model is released. And the

• 31:34Craig S. Smith?

  you know, this is this is a tremendous amount of labor. I I I remember when I first found out that there were human annotators for supervised systems. It was like, oh, this is AI. But in fact, there's this army of of humans out there working on it.

• 31:59Craig S. Smith?

  You've moved into L L M

• 32:03Craig S. Smith?

  evaluation, right? Into using AI to evaluate AI. Can you talk about that?

• 32:10Speaker 00

  Yeah, sure. So

• 32:13Speaker 00

  there there is a big part of the market that's using AI to evaluate AI, right? Which is really important.

• 32:21Speaker 00

  There are some challenges that that presents, though, because you are trusting another model to effectively act as a human.

• 32:34Craig S. Smith?

  Yeah.

• 32:34Speaker 00

  And the ability that the the the level of confidence that you have in that the eval model

• 32:42Speaker 00

  really depicts what confidence you have in the outcome of those evals, right? So there is no perfect eval model at the moment. What we are seeing is that there's eval models being created that are very narrow domain specific. Right? Um, you know, one, it's cheaper, it's faster, but it's, you know, fine tuned to be very specific eval. I think that as we wind the clock forward, AI is evaluating AI is going to be a big part of the market. Um, I think that human evaluating AI is still going to remain the gold standard and be, you know, a really important part of the market going forward.

• 33:27Craig S. Smith?

  Yeah, and those app and have evaluation L L Ms that you've developed yourself or?

• 33:35Speaker 00

  Yeah, so we we have the ability to bring AI and human. And what becomes really important is an AI will evaluate the AI model and will come with a confidence index. And where it's low confidence, it can be verified by a human. Also, we use the humans to do spot checks around making sure that the model's performing right. So the ability for us to incorporate humans with the AI evals in a single workflow. I think that's that's something we're betting on and investing in and making sure. I think that's where the market's going to be headed in the long run.

• 34:12Craig S. Smith?

  And when you say the market for for evaluation, is is it a crowded market? It's certainly a large market.

• 34:20Speaker 00

  It's a large market. It is crowded and, you know, what what we typically find is that there's. Competitors who have their different strengths. So there will be some who are very focused on, you know, very uh um advanced science and, you know, the hard science of mathematics, there's some who specialize in coding. The thing that Appen brings is that broad contextual knowledge with the ability to go, you know, domain specific deep in different areas. So now we can find people um all around the globe who represent humanity,

• 34:59Speaker 00

  but then also translate that into people in Thailand who have a PhD in molecular biology. All right. Now, that pool gets a lot smaller. We can't find tens of thousands of of those folks. But it's the ability to go both uh broad and deep is is where Appen shines.

• 35:16Craig S. Smith?

  How do you figure out what to pay people? Uh, because if you're doing a PhD in molecular biology, certainly they're going to demand a higher uh pay than than somebody who's just evaluating garden variety question and answers.

• 35:34Speaker 00

  Supply, demand, economics, which which derive. But there's also, when we find someone who is delivering consistently high quality data and has a um a real interest in the work that we do and a passion for it and wants to come and complete that work, those folks are incredibly valued, both for us and for our customers. So um we do differentiate our pricing, even an individual level.

• 36:02Craig S. Smith?

  Yeah. Is this something that somebody can make a living on? Or is it sort of gig work to that uh, you know, people drop in and out of?

• 36:13Speaker 00

  Yeah, it's increasingly we're seeing people um who this is a large part of their their earning opportunity. So we have folks at Appen who, you know, twenty plus hours a week and they consistently. Doing this work and again, I'm really passionate about it and really enjoy doing the work and understand the value that they're providing uh to the AI ecosystem. So that there's uh I I think the human data annotation industry gets a bad rap sometimes and it's like pushed to the lowest common denominator. When you speak to our, you know, crowd workers, they really enjoy the work that they're doing and, you know, they're rewarded appropriately and um yeah, they're super passionate about the industry.

• 36:55Craig S. Smith?

  Yeah. Do you have you, I mean, there there are certainly uh in India and the Philippines, for example, uh a whole industry of annotation uh provided by uh business uh process outsourcing companies. Uh do you have, do you top those? Do you subcontract? Do you uh acquire those uh to to add to your uh your roster or uh how do you interact with those those people?

• 37:33Speaker 00

  Yeah, there's um, we typically have enough breadth in our database of people who are working from home to be able to deliver the needs of of our customers. Um, there are times when um a customer will have uh data security requirements where it needs to be done in a facility. And we support that model and Appen does both. Appen um is predominantly a work from home crowd based model. But we also have many facilities around the world where we can support the most secure requirements of our of our customers.

• 38:14Craig S. Smith?

  Yeah. And and what's the, do you have kind of, I mean, of course, there's a bell curve uh distribution, but who? What is the top of the curve of what kind of people uh do you have doing this kind of work? Are they? I don't know, stay at home moms? Are they uh?

• 38:37Craig S. Smith?

  You know, young people, the, you know, digital nomads, or, you know, are they uh, you know, professionals that just wanna be involved?

• 38:50Speaker 00

  Yeah, it's a, it's a, it's a great question. It's really fascinating. So the mix of, um, I'll talk about Appen previously, which is, um, pre L L N. Um, it would be a mix of, you know, some of the things that you spoke about. So stay at home parents, um, people who are in remote areas and before work from home was such a big part of the market. Um, retirees and students would say kind of the big categories. But as we get into the L L N space and the work that we're doing is increasingly domain specific. Um, I mean, we have people who have multiple PhDs who work for Appen,

• 39:32Speaker 00

  who are just very passionate about the field that they work in and wanna support the development of A I. So the market has shifted very quickly and now it's, you know, people who may have a full-time job and then they come home and can earn, you know, a really good hourly rate, providing feedback into how the models are performing in an area of domain specificity that they're really passionate about.

• 39:56Craig S. Smith?

  Can you talk about what the hourly rate is? I'm frankly thinking a few friends and relatives that might like that work.

• 40:05Speaker 00

  Yeah, so it, it, it does vary and it's, it's, um, a significant variation and it can be, you know, the, the, um, the industry as a whole that Appen in specific, in particular, you know, we pay above minimum wage every place we operate in. So there is a floor, right? Um, and it does not go below that floor, right? So I think in the past, you know, maybe this is more mechanical Turk days where it might have been, you know, you get a penny for a task and you look back after an hour and you're like, well, that wasn't worth it. That's, that's not the case anymore. So, which is, which is great. And Appen's never been in that part of the market. Um, so anywhere from kind of

• 40:48Speaker 00

  above minimum wage to hundreds of dollars an hour for a domain-specific expert.

• 40:53Craig S. Smith?

  Yeah, yeah, that's interesting. Uh.

• 40:56Craig S. Smith?

  Yeah, and so this is a growing space of. Do you think that the role of AI evaluation will crowd out the human value? When you were saying there will always be a role for for humans, but how do you see that developing? I mean, also, models are through various strategies are getting increasingly accurate.

• 41:27Speaker 00

  I see the market in two sides at the moment. So there is a B to C market, which is focused on consumer users. And a B to B market, which is focused on, you know, that's where you get into agentic AI and you know, how our enterprises will be adopting AI to improve their operations. The B to C market, I think will be dominated by a few companies that have very large distribution in digital services. So you've got Google, Meta, OpenAI in the West, handful of companies in China, and I think that's going to be the landscape there.

• 42:12Speaker 00

  And the interesting thing is that they they operate all globally. So I don't think we'll see a regional B to C

• 42:22Speaker 00

  sovereign models that actually have impact purely driven by the distribution. The other phenomenon that I see will play out is we've been as consumers, as you know, from the consumer side, trained to exchange information about ourselves for some utility in a digital service. All right. All right. And this is a very well trodden path and that information about ourselves is used in in advertising.

• 42:56Speaker 00

  Alright, so there is a very well established ecosystem in place um to create that transfer of wealth. And I don't think we're there at the L L M point yet where this has been a major part of it. You know, but you start to see what Google are doing, and I think we'll see the similar thing with Meta. And it'll be interesting, OpenAI, at what point, if they do start to bring in advertising as a way to kind of offset the the cost of the development and the inference for the models, because it's it's um.

• 43:33Speaker 00

  I I can't see a world where the subscription fees are going to offset the compute required to to run these models. Um, in in a bit of a long arc here, but I think what's going to be really important, and again, you know, something that we've experienced through the market is getting the user experience, that marginal part better, and getting the the um ad targeting that marginal part better, just has huge economic ramifications. So I expect that as AI becomes uh more pervasive than what we've had in the past with search and,

• 44:17Speaker 00

  you know, other forms of consumer um applications, that there's going to be an increasing need for the subjective feedback into how these models are performing. And that will rise because of the economics, uh you know, the order of magnitude of the economics that support that environment.

• 44:36Craig S. Smith?

  It so you have a view of across many, many uh different models. Are you generally impressed with model performance? Or are you kind of privately appalled and thank God we have human evaluators?

• 44:53Speaker 00

  It's a little bit of both. There are areas where they work incredibly well. And there are areas where uh, you know, I don't think they. Uh, living up to the hype. So, you know, I'm excited about our role and, you know, helping and the way that the industry is progressing. And I think it's the, the pace of innovation is faster, you know, much, much, if you compare it to the, you know, you, you brought up FeiFei Lee and ImageNet, that was incredibly fast paced. This is, I think, ten times faster. So we're, we're just moving at a pace that we haven't seen before. So it's, um, it's, it's hard to make an assessment at a point in time because I'll be wrong in two months from now.

• 45:35Craig S. Smith?

  And I've been asking people this, what do you think has to happen? Right now, uh, it's, it feels that we've hit kind of a plateau on, on uh, L L M development. I mean, there's incremental improvement on this model or that. Um, but there isn't the kind of breakthrough that came with the transformer introduction of the transformers, transformer algorithm. Do you, it, to me, the, the, the next step is, um, solving, uh, catastrophic forgetting so that you can have a model learn continuously without erasing

• 46:20Craig S. Smith?

  uh, its, its earlier training uh, weights is, uh, what do you think is the next horizon that, that the, the industry is, has got to get through?

• 46:35Speaker 00

  Look, I think, um, when we, when I cast our eye back over time, right? Um, there are three things that drive model performance. It's some type of shift in architecture. It's scaling compute and scaling data, right? And, you know, I draw this curve where it's like the, it, the architecture gives a step change, like what we saw with transformers, like what we saw with, you know, convolutional neural networks, you have a step change and then this gradual. Improvement over time, alright, and that improvement over time is driven by less about architecture breakthrough that's refining the architecture,

• 47:18Speaker 00

  but it's just throwing a hell of a lot of data at it and a hell of a lot of compute. So I think we're at this stage now, and you see it with the, you know, subsequent releases of, you know, OpenAI's models. The difference between, you know, uh, GPT two, the chat GPT two was enormous, right? And that was driven largely by human feedback and then three to four and four to five less so, alright? Um. The breakthroughs that are needed to make this adopt at scale. Um, it will be interesting whether there is an architecture breakthrough that creates this, or it's just going to be the long slog of data and compute.

• 48:03Speaker 00

  And I think when you look at the investments that are being made in the industry at the moment, compute is clearly something that people are betting on, so it's, um. And it's a hard one because the, the model architecture breakthrough, it's really difficult to predict, almost impossible to predict. At what point will someone come with a, a new architecture? It's like, um, trying to predict, um, you know,

• 48:34Speaker 00

  innovation, it's, it's, it's, it's a nonlinear path. Um, which makes it really exciting, right? Because there will be a point in time where someone comes up with a, uh, either a slightly different architecture or a radically different architecture that has a huge breakthrough. Um, but I think we're in this period of, of time at the moment where it's grinding out the improvements through scaling compute and scaling data.

• 48:58Craig S. Smith?

  Is there anything I haven't asked that, that you, you, you, you'd like to talk about?

• 49:04Speaker 00

  I, I think the, um, you know, I spoke a bit about to B to C side of the market. I think the B to B side is a really interesting one also. So, you know, as we think about evals and data curation and training.

• 49:20Speaker 00

  The B two B market is quickly evolving as a quite separate stream. Where, you know, the training data for agentic systems requires, um, you know, what's called reinforcement. Uh, gyms where people can go in and create a simulated environment with lots of different software, synthetic data. Synthetic documents. Um, you know, the example is like, if you want to build a agent that is really good at creating. Financial documents in the oil and gas industry. You need to create an environment that it's going to be able to go and train on something that looks like that. So we're seeing increasing demand for these, um, highly demographic, uh, sorry.

• 50:04Speaker 00

  Highly industry and functional specific training environments. And then I think the, you know, the next phase of that is making sure that the evals of that data is really accurate. So there's a, is a, um, part of the annotation market, which is can you find me a functional and domain expert? So a finance person with oil and gas industry to go and evaluate how these models are performing. So, um, it's a, it's a really interesting market. I think the B to C side, um, there's a really clear path for massive. Human evals at scale to bring that global subjectivity and context. In the enterprise space and the B two B side, highly fragmented.

• 50:50Speaker 00

  You know, it's small and narrow applications that are going to be built out. So it's going to be super interesting how both of these sides evolve, but I see them kind of quickly diverging.