Why Humans Are Still Powering AI [Sponsored]

Ever wonder where AI models actually get their "intelligence"? We reveal the dirty secret of Silicon Valley: behind every impressive AI system are thousands of real humans providing crucial data, feedback, and expertise. Guest: Phelim Bradley, CEO and Co-founder of ProlificPhelim Bradley runs Prolific, a platform that connects AI companies with verified human experts who help train and evaluate their models. Think of it as a sophisticated marketplace matching the right human expertise to the right AI task - whether that's doctors evaluating medical chatbots or coders reviewing AI-generated software. Prolific: https://prolific.com/?utm_source=mlsthttps://uk.linkedin.com/in/phelim-bradley-84300826The discussion dives into:**The human data pipeline**: How AI companies rely on human intelligence to train, refine, and validate their models - something rarely discussed openly**Quality over quantity**: Why paying humans well and treating them as partners (not commodities) produces better AI training data**The matching challenge**: How Prolific solves the complex problem of finding the right expert for each specific task, similar to matching Uber drivers to riders but with deep expertise requirements**Future of work**: What it means when human expertise becomes an on-demand service, and why this might actually create more opportunities rather than fewer**Geopolitical implications**: Why the centralization of AI development in US tech companies should concern Europe and the UK

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tim Scarfe?

  There's a dirty secret, isn't there, in Silicon Valley and in the tech world? That there is, and I don't think people realise the extent to this, there is an absolutely huge importance on human data and human expertise and human understanding. And that is completely glossed over.

• 0:15Speaker 01

  Yeah, I mean fundamentally, artificial intelligence is founded in human intelligence. And I think in the stack of data, algorithms and compute, I think the human data element is often the least spoken about, maybe the least glamorous. People want to imagine that there's a simple kind of input-output equation, but ultimately there's a messy layer in the stack of human beings who are providing their data to, you know, either label data, provide ORHF, post-training data, and then ultimately the evaluation and the assessment of the model performance. All ultimately is kind of, has an element of human data in it.

• 1:00Tim Scarfe?

  I can't tell the difference between a doctor and someone who pretends to be a doctor. The only way that you can actually tell the difference is if you have this deep abstract understanding and you know that they're breaking the rules.

• 1:10Speaker 01

  It's increasingly clear that frontier models as a platform are going to be centralised and controlled by a relatively small number of players, which at the moment is almost exclusively these US tech companies. So I think there is a bit of a wake-up call.

• 1:25Tim Scarfe?

  What the future will hold is basically a marketplace of intelligence. So in the past, we had a marketplace of oil, for example, or electricity. Intelligence is going to be the new traded thing.

• 1:39Speaker 01

  I'm the co-founder and CEO of Prolific. And Prolific is a human data infrastructure company. So we make it easy for people developing frontier AI models and running research to get access to trustworthy, high-quality participants for high-quality online data collection. Prior to the ChatGPT moment, the primary modes of data collection, the people were fairly fungible, right? So you're optimising for cost and scale, maybe offshore, lower-cost labour, which I think created this dynamic of human data for AI being a bit of a dirty secret.

• 2:16Tim Scarfe?

  Let's talk about your core technology. Now, you've solved an interesting problem that I've tried to solve in the past. So I started a company called Merge and it was a code review platform and it was exactly the same thing. I realised that code review has to be done by humans and people talk about automation and the software engineering lifecycle. It's mostly bullshit. It's actually orchestration. You need humans involved in every single step, most importantly, code review. We had a skill matrix and we could learn their skill and we could, you know, a pull request would come in from one of our customers and we would dynamically assign it to an expert in that field and they would, you know, because you don't want a superficial rubber stamping pull request. You need to have someone actually who understands the code and goes into it. And this, I think, is one of the biggest problems in business in general, which is that, you know, we have all of this expertise out there, we have these problems

• 3:00Tim Scarfe?

  over here. How do we match them together? How have you done that?

• 3:03Speaker 01

  First, yeah, I appreciate that you understand the challenges and the complexity of dealing with with human data and the orchestration and the routing of tasks of the right human. I think often people want or expect this data to be as simple as calling an API and getting a kind of a CID, CID style response where it's fully automated, easy, clean and simple. But the reality is that humans are messy and dealing with humans, especially at the scale that we're dealing with them where we have hundreds of thousands of active participants or raiders

• 3:44Speaker 01

  on the platform is challenging. And I guess fundamentally, the value that we add is the deep kind of verification, vetting of these participants or building up a deep profile of the nuances of each of our participants and the behaviour that they provide in the data collection tasks so that we are able to route the most appropriate task or project to the right humans. I think increasingly, understanding how to incentivise these participants and make sure that we're incentivising the right behaviour.

• 4:26Speaker 01

  Kind of a win, win, win dynamic across the the three different relationships on the platform. So the data collector, us and the participants and not treating this as a strict supply chain where we're trying to commoditize or aggregate the participants and their their data and get it for the the lowest lowest cost. Ultimately, we we believe that the highest data quality is produced by people who are properly incentivized, like understand the impact of their their work and are going kind of beyond just the financial financial incentives.

• 4:56Tim Scarfe?

  I mean, obviously, we could use the the Uber analogy, but it doesn't quite work because we're talking about very deep expertise here. We're talking about very, very specific things. So you need to find people and verify that they have the expertise. You need to check that they're not gaming the systems. I'm not sure. You must have some kind of operational analytics where, you know, like this is what a normal behavior profile would look like. And you need to incentivize them. And certainly, when you employ people in the real world, you incentivize them in terms of things like autonomy and cultural fit and lots of like human like factors. How do you how do you do all of that?

• 5:28Speaker 01

  So the onboarding of of the participants, everyone is kind of ID verified. Be sure they are who they say they are. They are where they are in in the world. Secondly, there's a feedback loop from the from the researcher. So this is analyzing and assessing the kind of QA of of the data, feeding that back into the the model and ultimately using that information to rank participants. So you're not just like selecting for an audience, but we're able to preferentially provide the participants who are going to provide the highest data quality for that task context. And then I would say the third is the network analysis. So looking at this participant as our participants as a network, understanding the interconnections between those and finding,

• 6:12Speaker 01

  you know, pockets of this behaving participants or or people who are maybe trying to game the game the system and then using that information to kind of like filter filter the pool or or de-rank those participants so that they are providing less data over time or ultimately removed from the from the platform if if if required. And then the other other thing is, I think, is the incentive piece, which I think is like very, very interesting kind of game theory around where.

• 6:37Speaker 01

  From kind of behavioural research, you mentioned Danny Canaman and Co. We know that when the opportunity arises, people tend to cheat a bit. And especially when you have single shot relationships. So there's the classic game theory, share or steal. I don't know if you're familiar with that.

• 6:55Tim Scarfe?

  No.

• 6:56Speaker 01

  That experiment.

• 6:57Tim Scarfe?

  It's a bit like the prisoner's dilemma.

• 6:59Speaker 01

  Yeah, prisoner's dilemma, that's what I was thinking about. So there's the game theory idea of prisoner's dilemma, where ultimately, if you see that as a single shot relationship, a single point in time, the incentive is for both participants to steal. The things that change that into a dynamic where both participants are incentivized to share, or in our context, kind of not cheat or not game the system, is if you treat it as a relationship and you have multiple touchpoints over a long period of time. You have high communication between the two sides of the platform. And ultimately, you drive towards this kind of like win, win, win dynamic,

• 7:40Speaker 01

  rather than a single shot experiment or data collection.

• 7:45Tim Scarfe?

  You might be measuring how long they spend to do tasks. So how do you sort of bring the human component into it?

• 7:52Speaker 01

  That's a great question. So you're probably familiar with the analogy of the mechanical Turk.

• 7:59Tim Scarfe?

  Yes, tell the audience about that.

• 8:00Speaker 01

  We had a chap who was, I think it was a robot playing chess was a classic example. And it was perceived from the audience as this was a fully autonomous process. And then behind the scenes, ultimately, it was a human controlling the robot. I think the analogy of the mechanical Turk is super interesting because people want this process to be to be simple and for you to be able to like fully abstract the humans behind an API and for you to be able to call human intelligence on demand via an API. And that is the value that we want to provide ultimately. And we try to abstract away as much messiness and as much complexity as we can.

• 8:44Speaker 01

  And I think how we bring the human element back in. Is by trying to get out of the way as a middleman. And our philosophy is to try to build a direct connection between the people collecting the data and the participants providing the data. So they're able to communicate, provide feedback, ultimately kind of understand the impact that their work is having on the on the data collectors, whether that's research or model development, et cetera. So getting this feedback, mutual feedback across the platform, this peer to peer messaging, we think is crucial to the trust and making sure that there's a sufficient amount of empathy

• 9:26Speaker 01

  for the people who are providing this, this super valuable data.

• 9:31Tim Scarfe?

  Now, I've got a lot of experience about work and I, it's good and bad. In a way, I don't like it because as we were just saying, there's this huge epistemic history, even if someone is a creative professional, they've been doing it for years. I still find that quite often they're just uncalibrated and the onus is on me to specify what I want to an insane level of detail. And the onus of doing that is often greater than just the cost. I might as well just do it myself. So how, how do you guys overcome the specification problem? And do the tasks that you do on prolific, do they tend to be quite close ended? What I mean by that is, is you have specific outputs or sometimes are they quite ambiguous and open ended in terms of the output?

• 10:11Speaker 01

  Yeah, that's, that's a great question. I think it ties back to obsession with data quality. And I think it's two, two aspects to data quality is obviously the profiling, the quality of the audience. So how, what, what expertise and specialism training that the participants have. And then also the quality of the task, task design, or even the specification of the people you're looking for, right? We'll often get requests for PhDs in biology. We're like, okay, there's, there's quite a lot of nuance within, within biology. You're looking for genetics, expertise, bioinformatics, healthcare, et cetera, et cetera. So, so supporting the, the researchers in specifying.

• 10:57Speaker 01

  The audience with sufficient detail. We have a mix of tasks, so the platform is fairly use case agnostic. So some of it is fairly self-contained. Game theory dynamic of multiple touchpoints with the same participants tends to lead to a relationship between the researcher and the and the participant, which leads to better data quality. We have many, many projects that are long running where the first part of the project is training or providing that context to the that audience, so that they're able to learn over time what what good data quality means for this project. And that can be very, very long running.

• 11:39Speaker 01

  So sometimes weeks or months of longitudinal or multi-step data collection.

• 11:45Tim Scarfe?

  Very cool. So essentially you built this platform which gives you human expertise on demand. What does this mean for the future of work?

• 11:53Speaker 01

  I think, yeah, so we've optimized the platform for breadth of audience choice. Although you can earn a great side hustle on prolific, don't necessarily want to optimize for like very, very professionalized raiders or research participants. We want to tap into real world users. So let's say, for example, you're looking at healthcare workers to evaluate your medical chatbot. We want to tap into folks who are actively working in the field and not people who've left the field and are now professional annotators.

• 12:34Speaker 01

  We're trying to reflect the real world and real world users as much as possible. So we absolutely see our platform as augmentation to work necessarily than a than a replacement, though increasingly this work of human data for AI is being professionalized.

• 12:52Tim Scarfe?

  What I like about it is it increases the market efficiency. You know, like the market is all about, we have these economic tasks that are valuable and we have these folks over here with skills.

• 13:02Speaker 01

  Yeah, absolutely. That is something we're explicitly thinking about is how do we? Train folks with the skills that are useful for these frontier model providers. So for example, like, how do we take general participants and turn them into a high taste evaluator? So it's not necessarily just domain expertise or domain knowledge, which is valuable, but also kind of general audience who are skilled in overcoming the kind of typical biases of preference valuation.

• 13:34Tim Scarfe?

  You know, like Uber, for example, there is a critical mass, maybe even dating websites is another example. Like, you have to bootstrap it. So Uber wouldn't work if there was just no density of cars in my area. So I don't know, is it the case that you had to bootstrap it and it got easier? Or was it easier at the beginning? Because even when you have loads of participants, you might have problems where there's just not enough work to go around. Because it's very stratified, you know, for example, if you're doing some demographic research and I think you said, you know, 6% of people are of a major. So you'd want to like get 6% of people, you know, so you might have this sparsity problem. But by the same token, when you actually have this scale, like, do you find that it kind of works better?

• 14:12Speaker 01

  Yeah, we have the chicken and egg problem of all marketplaces. And I think you can think of this, again, analogous to the Uber cities problem, right? So in the early, I think the analogy of a city in the Uber context for us would be a segment of the audience or people with a particular skill set where we might have bootstrapped up to, from an atomic network to a kind of a scaled network, say, for example, for a US general audience population or a UK audience population. But then for each new expert or kind of segment that's in demand,

• 14:54Speaker 01

  we need to go through that kind of atomic network, scaling that network up to a point where we're able to provide kind of incident data. You know, we optimize for data collection kind of in the space of hours rather than days or weeks. So ultimately, we look at that kind of market liquidity as a user segmentation problem, where we might have scaled networks for particular audiences. And then it's like thinking about what's the next marginal user who can kind of incrementally add to that network the most value? We can drive the growth of the network in that way.

• 15:31Tim Scarfe?

  The matching algorithm itself, could you tell me about that?

• 15:34Speaker 01

  Yeah, secret sauce.

• 15:37Tim Scarfe?

  Yeah, I'd be so fascinated to know because I'm thinking, you know, maybe there's an analogy to Google search and they have like a page rank algorithm. And that is an example of this kind of social graph type metadata, because you know, you put a hyperlink to another page, you know, if you actually like that page. And then, of course, you can build up a ranking from that. But there's like intrinsic content type metadata, which, you know, obviously you guys have come up with a way to like mix all this together.

• 16:00Speaker 01

  Yeah, exactly. I think the other analogy is like to, I think it's called a two towers algorithm. So TikTok, Instagram reels, where you have the context of the user who's filtering for the for the content. And then you have all of the choice of content. I think the analogy here is the you have the task context and then you have the the human context and you want to be able to rank the rank the humans so that you're getting the kind of optimal people floating to the floating to the top. And in a similar way that when you open up YouTube or Instagram reels or TikTok, you get the content that's most relevant for for you.

• 16:41Tim Scarfe?

  Who controls these AI platforms? Controls quite a lot.

• 16:45Speaker 01

  The internet is a fully decentralized platform, not particularly owned by anyone. It's increasingly clear that AI infrastructure and frontier models as a platform are going to be centralized and controlled by a relatively small number of players and predominantly US players at the moment. Maybe China's playing a role. Unfortunately, not a very, very significant role being played in the UK and Europe as a whole right now. I think I think we're lucky that many of the folks who work with these global labs, even if the capital is coming from the from the US are international, they have a global perspective.

• 17:25Speaker 01

  And we find when working with our customers in the frontier labs have extremely positive intent. They want these models to be globally useful and reflect the input from a wide variety of opinions and subjectivity. So I think. There is obviously a risk if these models do become super intelligent and there is massive labour impact, that cost is going to be felt internationally and will be felt by us here in the UK and Europe. And the value of that efficiency is going to flow to the owners of the platform, which at the moment is almost exclusively these US tech companies.

• 18:10Speaker 01

  So I think there is a bit of a wake up call for the for the UK and Europe, even though this we're maybe maybe late to play a more significant role in the lifecycle of these models, whether it's like owning more of the training, having more locally produced models, data centres, energy abundance in order to power these these energy hungry models. Definitely think there's there's room for more UK EU dynamism and accelerationism in this space. We are still hiring software engineers aggressively, including junior software engineers. Some people have a philosophy of junior engineers and software engineers won't be won't be hired anymore because a senior engineer plus AI agents will be 10

• 18:51Speaker 01

  times more efficient. I think that neglects the fact that these models are extremely powerful teachers and coaches and junior engineers much more rapidly become as competent as senior engineers with this co-pilot training to be better better software developers. You know, when I when I started prolific, one of the motivations for starting prolific was to learn about web development and building a product. And this was a relatively sort of learning process than it would be would be now. And I would have I would kill for a near super intelligent co-pilot in order to build more product faster. And I think there's a very elastic demand for many of these things like software that models are going to improve our efficiency to build.

• 19:35Tim Scarfe?

  There is something special about local situated human expertise. And what I see is that this could make the pie bigger because assuming that these models will be limited in how they understand different domains, we might in the future need to have an operational loop on top of language models, just so that people can actually verify the, you know, it might be consequential health advice or something like that. And the user doesn't know whether it's whether it's correct or not. So we could have a prolific type plugin system where a user could press a button and say, is this legit?

• 20:07Speaker 01

  This is definitely where we see direction of travel also, like AI, human interaction, or like agent, human interaction, analogous maybe to like hu- human computer interaction. I think that's like the next, next phase of, of research. Yeah, you could, you could imagine, for example, a, a deep research style agent going off and, and doing a long-running task. And one of the steps along that workflow is to get a review from a, a human expert that's routed to the, the most appropriate person for, for that, for that task. So again, I think something that we're looking at at, at a prolific is how do we build these systems where agents and humans can collaborate and where we can move beyond this kind of relatively simplistic, preference-based evaluation in order to build

• 20:51Speaker 01

  out systems which better simulate the ultimate objective that we're trying to aim for, right? We, we, we know that models are, when they have a, a goal, because of Goodhart's Law, they're very, very good at optimizing for that, for that goal. So therefore choosing the right goals is increasingly important and developing the, the tools to effectively simulate those, those goals and get as close to the, the true objective as possible, whatever I think you mean by, by that objective. But I think ultimately it's real-world performance for real-world users in like real-world contexts, ultimately the thing that we're trying to simulate with all of these evaluations and, and benchmarks.

• 21:32Tim Scarfe?

  We are still building a marketplace of intelligence by creating this link between all of these human situated experts and the system which matches and, and learns. And maybe there'd be a middle way. Maybe we can, we can create some automation against that. But fundamentally, I believe that we're going to need human expertise more than ever. I, I, I think the pie has gotten bigger because so many people are just, they're, they're getting their appetites whetted. They're, they're generating videos, they're writing code, they're building applications, they're doing all these things they couldn't do before. And then they, they, they hit a brick wall because they realize they don't actually understand it deeply enough. So they need to bring the experts in. All of these experts are in, in, in more gainful employment, in my opinion, than they ever were before. And this could create a virtuous cycle. I mean, I, I think that would, that's a positive outlook of the future.

• 22:18Tim Scarfe?

  What do you think?

• 22:19Speaker 01

  I, yeah, I tend to agree. I think this like reminds me of the dynamic between synthetic data versus human data. And people often like kind of frame that as a either or proposition. Whereas I'd say we're very bullish both on synthetic data and human data and ultimately augmenting human data and human expertise is expensive and models are able to make that that cheaper and maybe more effective. But when you reduce the the cost of something, you you tend to increase the demand to to kind of compensate for for that effect. Often comes up with them these AI models as as as infrastructure. But I think you'll end up with a with a kind of a similar effect, even if we're accelerating human data with,

• 23:02Speaker 01

  you know, in the in the in the context of of post-training or model evaluation with LLM as a judge or synthetic data because of the explosive demand for these models. In the first place, even if the proportion of human data decreases over time, the actual scale and importance of that that data is is likely to to increase. The Financial Times, the New York Times, YouTube, Reddit, etc. They get a recurring payment for for their data and it's it's licensed. And I think you could also imagine that human expertise being licensed in a in a similar similar way where you get an ongoing passive incentive for for you providing data to to improve the improve the model, maybe analogous to like a Spotify, right, where you have the proceeds of all of your subscriptions going

• 23:45Speaker 01

  out to all of the people who are providing music. You could imagine a similar approach to incentivizing human data in the future as well, where where we get this kind of more ongoing incentive to to continually improve either like a very personalized model, as you mentioned, right, kind of a digital twin of your expertise. I think that's maybe the the more obvious example, or even like to improve the infrastructure centralized models.

• 24:08Tim Scarfe?

  Yeah, it's almost a bit like you you buy a solar panel and then you can give you can give energy back to the grid.

• 24:13Speaker 01

  Yeah.

• 24:13Tim Scarfe?

  Yeah, similarly to that. Awesome.