#325 Phelim Brady: Why AI's Future Depends on Human Judgement

AI often looks fully automated. But behind the scenes, a huge amount of human judgment is shaping how these systems actually work. In this episode, Craig Smith speaks with Phelim Bradley, co-founder and CEO of Prolific, a platform that connects millions of real people with researchers and AI labs to evaluate and improve AI systems. They explore the hidden human layer behind modern AI, why traditional benchmarks are becoming less reliable, and why AI companies increasingly rely on real human feedback to measure model performance in the real world. Phelim also explains how demographic differences influence how models are evaluated, why human judgment remains critical even as AI improves, and how the collaboration between humans and AI will shape the next phase of development. This conversation reveals the human backbone behind today's AI systems. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Preview and Intro (02:45) Founding Prolific And Early Pain Points (06:30) From Mechanical Turk To Representativeness (09:55) Academic Research And AI Use Cases Split (13:40) Vetting Real Participants And Fighting Fraud (17:45) Scale, Community Growth, And Talent Mix (22:00) High-Complexity Projects Over Commoditised Labeling (26:40) Measuring Model Persuasion With Live Conversations (30:20) Demographic-Aware Model Preference Benchmarks (34:10) The Rise Of Human Evaluation Over Benchmarks (38:00) Enterprise Model Choice And Continuous Evaluation (42:00) Why Humans Won't Disappear From The Loop

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Craig S. Smith?

  I have in past interviews with people working with human resources for AI is that it's kind of the dirty little secret of AI, that it's built with humans or that a lot of the other, these armies of of human evaluators or or labelers out there.

• 0:22Speaker 01

  Or computational biology, but while there, realize that there was core pain points in running high quality, academically rigorous human subject experiments online.

• 0:38Speaker 01

  My name is Phelim Bradley. I'm the co-founder and CEO of Prolific. And Prolific is a human data platform. So we connect both the infrastructure methodology and high quality global pool of humans and connect them with data collectors for purposes of research and human data for AI, in particular, post-training and evaluation use cases. Started Prolific while I was doing a PhD at the University of Oxford. I actually did my PhD in bioinformatics or computational biology. But while there, realize that there was core pain points in running high quality,

• 1:22Speaker 01

  academically rigorous human subject experiments online. So platforms that were popular at the time had problems of poor quality data and verification of the humans who were taking part in research. The tooling and the infrastructure was extremely poor user experience for both sides of the platform. Again, impacting both the ease of use and the quality of the resulting resulting data. And ultimately, it was a created a kind of a an unhealthy platform and dynamic. We set out to build Prolific in order to solve that problem. And originally it was a side project while I was completing my PhD.

• 2:06Speaker 01

  And grew the company to the reasonable traction before completing the my my PhD.

• 2:13Speaker 01

  And the story since then really has been rediscovering that same core pain point across a range of different industries. How do I recruit high quality data online from a, an audience of highly verified, trustworthy participants? This is pain point across research applications, so user research, polling, academic research, which was kind of our origin story. And now more recently also human data in the AI development lifecycle. And I think we, we try to apply some of this methodological and academic rigor of the behavioral sciences to some of the problems in, in evaluation of generative AI applications.

• 2:55Craig S. Smith?

  Yeah. And this is an interesting area. And I know you, some of your other interviewers have pointed this out as I have in past interviews with people working with human resources for AI is that it's kind of the dirty little secret of AI that it's built with humans or that a lot of the other, these armies of human evaluators or labelers out there. My experience or my knowledge of this goes back to interviewing Fei Li, who I just had on the program.

• 3:35Craig S. Smith?

  But the first time I interviewed her was about ImageNet, which was this massive database, really the first image database for training supervised learning programs. And it was her data set that allowed Jeff Hinton to validate deep learning in 2012 with Alex Nutt and, you know, kick off the deep learning revolution. And she turned to Mechanical Turk to do the labeling of her images. At that point, I hadn't heard of Mechanical Turk, but it's kind of funny because it's

• 4:13Craig S. Smith?

  a reference to a 17th or 18th century, I guess, 18th century German automaton mannequin that played chess. And famous people came and played chess with it. I think Ben Franklin played chess to it with it and it would win and people were just amazed. What they didn't know was there was. Small statured chess master curled up in the cabinet underneath the chessboard, who was working the mannequin with levers and things. So it's kind of like that, like you have AI, but under the table, there, there are all these humans working at it.

• 4:57Craig S. Smith?

  So that was Mechanical Turk. And then I worked for a long time and had on the podcast, guys who had a platform called Label Box. Are you familiar with them?

• 5:07Speaker 01

  Yes.

• 5:08Craig S. Smith?

  They're, they're.

• 5:08Speaker 01

  Label Box.

• 5:10Craig S. Smith?

  Yeah, they're a labeling platform and they sounded, their thesis sounded similar to yours, that there isn't a good platform. They built this platform and you can plug in third-party, you know, BPO teams into it, or you can have your own labelers, but it's a platform that they don't manage the human resources, or they didn't at that time. And then, as you mentioned, I spoke recently to Appen, that fields teams, or rather coordinates teams to do reinforcement learning with human feedback on foundation models. So where do you fit in that tradition?

• 5:53Craig S. Smith?

  Besides being the chess master underneath the chessboard?

• 5:59Speaker 01

  That's a great, a great question, a great, I think, framing of, of, of the space. I think we fit most in the tradition of the Mechanical Turk. So trying to abstract away the complexity of dealing with the messiness of real humans in the real world and provide labs and data collectors, the kind of high-quality data that they, they want. I think in some ways you could describe prolific as what could and should have become. And the, I think, a key kind of shift in the market, which we've invested heavily in,

• 6:40Speaker 01

  I think, in contrast to a platform like Mechanical Turk is, Mechanical Turk was great at applications like ImageNet that you mentioned, where, you know, labelling cats in images, is this a hot dog style questions any human would do? So it's a fairly commoditized task. And as a result, every human on the other end who's doing that labelling is somewhat fungible or replaceable with any other person. That's changed quite a lot since the days of ImageNet. And now the audience of the people who are annotating your data or providing your post-training data, providing your RLHF data really matter, their background, their expertise.

• 7:26Speaker 01

  And I think coming from behavioural research context, the representation or representativeness or the generalizability of the of the audience, that I'd say is like where we have focused to date in providing this human intelligence layer, but also maximizing the breadth and depth of the audience that you're able to access through a platform with an aspiration that the the best and depth of humanity is ultimately reflected and encoded in these these AI models.

• 7:56Craig S. Smith?

  Yeah. How much of your business is working with AI models? And how much is? Because I know you provide manpower or research manpower, I should say, for academic researchers. How much of it is working with AI models?

• 8:15Speaker 01

  Roughly fifty fifty in terms of the business, in terms of the investment and manpower we put behind the platform. Although I think crucially there, we focus on areas where there is strong synergy between the the two sides of the platform. So the audience choice and the representativeness and the ability to tap into real world participants in order to understand real world behaviour is a shared requirement and kind of pain point across both industries as well as. The quality of the data collection tooling and the infrastructure that we provide. You know, I think there's two kind of core dimensions into data quality. It's the quality of the contributors or the participants who are providing that data.

• 8:56Speaker 01

  And then quality of the infrastructure and the methodology that you use in order to analyze and abstract that data. So those two two components are core investments that we think that provides synergy across the markets that we operate in. And then obviously there's going to be differences in terms of like how we serve those customers. Whether they're large enterprise and need deep integration into their on-prem tools. Or they're a PhD student who needs, you know, easy, flexible, self-serve, self-serve access. And I think crucially, like fundamentally, the frontier of AI capabilities and AI research is AI models is a research problem.

• 9:38Speaker 01

  So there's a strong overlap between academic research, universities and the talent who's driving forward the capability of these labs. So our purpose as a company really is to accelerate the frontier of human centered or transformative research and AI. And there's a lot of kind of shared investment that supports both of those challenges.

• 10:00Craig S. Smith?

  Yeah, and a part of that investment and what you're selling is a vetted workforce, right? I mean, this isn't like Mechanical Turk, which has very little vetting, as I understand. I've never used it, but it's kind of self-serve. You sign up and pick a task and execute it. And consequently, the tasks are very simple. But you're further up the food chain dealing with more nuanced tasks. So you vet your workforce. How do you vet them?

• 10:38Speaker 01

  First, I maybe slightly correct the term workforce in that our participants are folks who participate in prolific in a flexible manner. And as a result, we're able to reflect kind of real world users, either consumer samples or people who have full time jobs in other professions and are able to participate in prolific in a ad hoc manner. So slightly different from the contract to solve workforce of some of the other platforms that you mentioned. And the verification and the betting side of things is one of our core investments and core IP. We have many layers of protection from the fairly obvious things in terms of doing identity verification of all of the participants. We know who they

• 11:23Speaker 01

  are. Not only do we know who they are when they sign up, but we check this on a periodic basis. So that we know that person is still the same person. Deep profile information to make it easy to route the right tasks at the right person based on either kind of independently verified data points or qualification style gates that we put folks through in order to run them through an exam, as well as lots of behavioral analysis and understanding like how these folks are actually interacting within the data collection or experimental workflows. But I think the latter is increasingly crucial in the world of

• 12:03Speaker 01

  agentic fraud. So people being able, agents or AI systems being able to replicate the behavior of humans very accurately is kind of a new threat to online data collection and primary data collection more generally, and one that we feel like we're right at the cutting edge of being able to defend against while still providing the scale of audience choice and global participation that we're able to power.

• 12:29Craig S. Smith?

  Yeah. And so the who, what would be, first of all, how many participants do you have on the platform? Not people using the workers, but the people providing the human talent. How many of them are there?

• 12:48Speaker 01

  Yeah, so we have a couple of million people in the, on the platform in general with typically several hundred thousand active in any given, any given month.

• 12:57Craig S. Smith?

  Wow, that is, that's impressive. And I mean, obviously you're not going through LinkedIn and reaching out to individuals if you've got, you know, a million people. How are you, how are you reaching them?

• 13:12Speaker 01

  Yeah, it's a combination of factors. The core growth of the platform comes through word of mouth growth and referrals. So because we offer a positive experience and meaningful work and meaningful pay, this is something that people tend to share online, share on social media, share with their family and friends. That's a core growth engine for us. Secondly, complement that with some incentivized referrals. So where we have gaps in our audience that our customers want to tap into, let's say we're looking for, you know, a PhD in a particular aspect of biology.

• 13:51Speaker 01

  We typically will have a small number in our pool and we can use that to bootstrap a larger sample through a folks network. And then third, we do community and event based marketing in order to supplement the first two channels. But I'd say our core growth and our core engine on, on both sides of the platform is the trust and integrity, high, high quality experience that we're able to offer, leading people to share this amongst their colleagues or their, their friends.

• 14:21Craig S. Smith?

  Yeah. And if I hear about it, I mean, you mentioned PhD students, but you know, you certainly don't have a million PhD students on the platform. What's the range of sort of talent level? And can you give a breakdown? Like we've got, you know, 80% are, you know, college educated, doing.

• 14:45Craig S. Smith?

  Simple tasks and twenty percent are PhD students providing more difficult responses.

• 14:57Speaker 01

  I don't have the precise numbers to the hand, but I would say we we generally optimize the overall platform to reflect the dimensions that exist within real world populations. So try to reflect the demographic breakdown that exists within a representative US sample or representative UK sample, et cetera. And then in terms of the AI work specifically, I'd say it's maybe even like roughly a third between general audience sampling. So this is kind of consumer based testing where the representativeness and the generalizability of the audience is key, but no particular specialism or skills are required.

• 15:39Speaker 01

  And then we have tasker based work where you're we either qualify or train people within the crowd to be high taste evaluators or people who have some skills in general AI valuation and and data labeling, which requires a kind of a different level of attention and understanding of how these models work. And then the final third would be expert level workflows where the the subject matter expertise that folks bring from their prior experience is the critical dimension of whether they are appropriate for the for the project.

• 16:12Craig S. Smith?

  Yeah. And so if I wanted to make some money on your platform, what would I do? What's the onboarding process?

• 16:23Speaker 01

  Yeah, so the onboarding process from a contributor or participant view is pretty pretty straight and straightforward. So it's a kind of a five step process in terms of filling in some basic information, giving us a a short interview on your background and experiences and where you think you might have unique skills or experience that would be valuable either to researchers or to the AI labs. And then some verification of both your your backgrounds, this KYC style of check, know your know your customer, as well as a shortcut by behavioral assessment to validate that you're engaged, attentive and trustworthy.

• 17:03Craig S. Smith?

  Yeah, and then I'm on. And then is it a list of tasks that I can choose from? Or do you reach out when you get a query from a customer? That wants a certain profile person working on their project? That you pull together that cohort and then send that cohort an offer? I mean, Sellserv or? Yeah.

• 17:30Speaker 01

  Yeah, it's Sellserv between kind of one-off projects or tasks, which you'll be notified for on your project, on your platform, on your dashboard. Or longer running projects where you need to agree to a certain level of commitment upfront. Maybe these are going to last for several weeks or months and require a little bit more time. Involvement where perhaps there's going to be an initial assessment. And then if you get past that initial assessment, you're able to participate in the longer running, longer running project. Yeah, it's a kind of a double opt-in marketplace. So every task is, you know, is advertised to you. You're able to assess whether or not that's something you're interested in at the reward that is is being offered.

• 18:11Speaker 01

  And you trust that the, you understand how your data is being being used. If you're interested, you can then accept. If not, you reject the place and that opens up then to another participant on the platform to choose if they want to take part.

• 18:26Craig S. Smith?

  Yeah. What are the range of of projects that people use the platform for? I mean, again, you know, you've got PhD students that are doing research projects. What kinds of, can you give us an example of that? And then on the low end, I guess the low end would be data labeling or? And what kinds of projects do you, can you give us an example there?

• 18:56Speaker 01

  Yes. So one of the beauties of the of the platform, I think, why platforms like Mechanical Earth became popular in the first place also is the flexibility of use case. So we've run nearly half a million projects over the last year with a vast range of different requirements and outcomes resulting in in thousands of publications as well as like many capability improvements to models that don't get as well publicized or cited. But maybe to give you a few recent examples, we actually don't do that much work on the, as you've described, the low end of the data labeling. As I mentioned at the start of the the conversation, we

• 19:38Speaker 01

  focus on high quality projects where the the audience and who is behind your data is a critical component. And most of the more commoditized data labeling use cases don't have that requirement, maybe a better fit for a a platform that optimizes for lower cost of labor. Some types of projects we optimize for or maybe have a slightly higher complexity bar and a higher bar of methodological rigor. So we've run a project recently which was investigating the persuasive capabilities of data of the art models, which was run by the AI Security Institute in the in the UK. And this is really this, I think it's a nice example as a combination of the behavioral research skills that we bring and the capabilities

• 20:21Speaker 01

  we bring in model evaluation, sort of understanding from a safety perspective, how critically persuasive can these can these AI models be in the real world with a representative selection of an audience. Another example is a project that we've run ourselves called Humane, which is a a user preference model evaluation benchmark. So there are popular platforms like LLM Arena or Chatbot Arena, where in short, two state of the art models are pitted against each other and users need to select which model they prefer and why. And ultimately, the preference of these model, these participants.

• 21:04Speaker 01

  Are analyzed and this results in a leaderboard of which models are preferred by by users. The, one of the challenges with chatbot arena is there is no control on the audience or the people who provides these judgments or preferences, which I think is analogous to running a political poll without controlling for the audience, audience selection. So we've done some projects which take the same idea, battle model A against model B, but add some methodological rigor in terms of the demographic representativeness of the audience behind the leaderboard. All of the models are double, double blinded, which essentially then allows

• 21:46Speaker 01

  you to ask the question of all of these models, is there a difference between the preferences of a US population, a UK population, between left-leaning participants or right-leaning participants? Is age a factor in how people rank the performance of these models? In which context do people prefer model A or model B? That adds a lot more nuance to what otherwise is a fairly simplistic leaderboard. And the short version of the results of that project is that the ranking of models does change based on the demographics and the audience behind the models, which I think is going to be an increasingly important factor as these models are rolled out in the real world in different geographic locales, people with

• 22:30Speaker 01

  different culture are using them for different use cases. Understanding which model is best for which context is going to be a more interesting question perhaps than which model is objectively state of the art on an overall basis.

• 22:43Craig S. Smith?

  Yeah, particularly as sovereign AI develops, I mean, these different regions or countries are developing models that are trained on local data, so they're more culturally attuned. But I'm interested in the persuasiveness project. How did that work? I mean, can you describe that? I mean, you have people in your database and they talk to a model or they read outputs from the model or what's happening there?

• 23:22Speaker 01

  Yes, there were many manufacturers and it was a, an experimental workflow. So essentially people were split into randomized conditions. And as a result, there's kind of a, an A D style test of whether participants, if they were exposed to a back and forth conversation with one of, I think about 20 models. And these models were instructed to persuade the participants to agree with one of the political stances, using different rhetorical strategies. So the, the experimental methodology was fairly, fairly sophisticated. And then the participants were measured on whether they agreed with the issue before they were exposed

• 24:06Speaker 01

  to the model and the conversation and after. And then as a result, you're able to measure the, the difference in how, how different models and different, different strategies informed whether, informed how much these participants changed their minds on the topics that were, were at hand.

• 24:26Craig S. Smith?

  Yeah, but the interaction between the participant and the model, is it like, you know, have a conversation with this model for 30 seconds and, you know, about some prompt, some topic? Or is it that there's a prompt and the model responds? And the participant reads the response? Or is the participant actively engaging in a conversation with the model?

• 24:58Speaker 01

  It was, I believe it was the latter, so an active engagement, multi-turn, multi-turn conversation. I'm not sure of the, I don't have the exact constraints to mine. I think it was perhaps a minimum number of interactions that were required in order to

• 25:12Craig S. Smith?

  Yeah.

• 25:12Speaker 01

  accept a data point. But yeah, it was certainly more sophisticated than a just read the output of a, of a model. It was a, a kind of a live, interactive style of, of experiment.

• 25:22Craig S. Smith?

  But that's a really interesting news case. So you need a certain level of political awareness. I mean, how did you pull together those? That cohort?

• 25:37Speaker 01

  Yeah, in this case, this I think goes back to the representation and the generalizability of the of the audience. So trying to get a sample of of the real world so that the the outcomes of that experiment are more likely to generalize to real world scenarios and to a real world context was a crucial criteria of of that that kind of project. And we have the tooling built into the platform, which allows you to do census matched sampling, which means that the resulting participants you you get broadly match the population on like key criteria such as age, ethnicity, political affiliation, affiliation, things of things of that nature.

• 26:20Craig S. Smith?

  Yeah, the I mean, you said you don't do much sort of labeling anymore. Is that right?

• 26:26Speaker 01

  That's what we never did a huge amount of, let's say, the the simple kind of commoditized style of labeling. One could call the post-training evaluation data sets a form of data labeling.

• 26:37Craig S. Smith?

  Yeah, yeah. Actually, that's what I was getting at. I mean, you know, in the early days of supervised learning, the focus was on labeling, particularly for computer vision. And then just in the last year, as these models have matured and become more widely used and as there are more models competing, evaluation is sort of come to the fore. How long have you been doing model evaluation?

• 27:09Speaker 01

  Yeah, we've been working in that area, I would say, for two to three, three years. So the, I think.

• 27:19Speaker 01

  Post the ChatGPT boom, the shift of user-driven model evaluation has shifted towards, as I said, kind of more complex audience requirements, more expert participants and a requirement for more kind of scientific rigor. And that's been a natural fit for our for our platform capabilities, which was the original wedge within this space. We've obviously invested pretty heavily in in additional capabilities and frameworks and the ability to kind of convert human judgment into useful signal for these model developers.

• 27:55Craig S. Smith?

  Yeah, yeah. And that's, yeah, as the models, I mean, evaluation on benchmarks has gotten sort of a, people have soured on that to a certain extent because you can, there are all sorts of problems. You can train your model to the benchmark or, you know, the benchmark somehow ends up in the training data and all, all sorts of things. So people don't trust, you know, a model winning the math Olympiad anymore. That, that at one time was a big deal. Now they're more focused on this human evaluation. Does that, is the market like blowing up for you guys, that end of the work?

• 28:40Speaker 01

  Yeah, I think that is a key driver of our expansion in the market. This shift away from like the academic benchmarks or exam-based benchmarks, which the frontier labs use to kind of advertise their relative performance. There's such a strong incentive, as you said, to, to either, to unintentionally gain these benchmarks and leaderboards, so their usefulness is decreasing as they become saturated and essentially the models are able to all pass with, with flying colors. But then it's becoming harder to judge the, the real world evaluation. And I think that's where platforms like ours come to their own in, in trying to simulate

• 29:25Speaker 01

  the real world environments in which these models and agents are interacting with real people, trying to achieve real problems. And then evaluating that performance is ultimately the, the real, the real goal that's ultimately going to drive kind of the, the economic impact of, that these models have in, in practice.

• 29:43Craig S. Smith?

  Yeah, you guys, you know, are a part of an industry. You say you have, I've forgotten the number, did I heard million, was it? A million or multiple millions or something on the platform. On the evaluation side, if you were to look globally, do you have a sense of how many people are involved in evaluation through platforms like yours?

• 30:11Speaker 01

  I don't have a strong estimate of the global number of people involved in this type of work. Though I would say it is, it's increasing and also the, the types of people who are likely to be involved in it over the next couple of years are likely to shift considerably as the labs kind of shift their focus towards knowledge work and understanding the capabilities that these models have in kind of all of the areas of the economically productive labor. Folks within all of those industries are going to have some involvement in evaluating and providing feedback to the tools that they use day in, day out in order to start to improve their performance, not only on these academic benchmarks, but actually how they're performing

• 30:56Speaker 01

  in real world conditions, on real world, real world problems.

• 30:59Craig S. Smith?

  Yeah, I mean, there's evaluation of the frontier models, that's one thing, but then there are all these applications built on top. Is it companies building applications that come to prolific to evaluate their applications? Or are there enterprises that are adopting solutions who want to evaluate?

• 31:24Speaker 01

  Increasingly both. I would say that the labs are leading the way as they have the talent in-house to understand the methodology and the frameworks that are required to build out benchmarks and the majority of labs will have in-house teams and in-house evaluations that are accustomed to their requirements and are not necessarily always one-to-one mapping to the, to the public, public benchmarks. But as enterprises start to build on top of this infrastructure, build agents. For their specific use case or their specific context, they increasingly also need this evaluation, not only of of their own tools, but also because there is such a wide choice now of which open source or frontier model

• 32:05Speaker 01

  to use, even evaluating which is which base model to use in the first instance is as useful piece of data and is not necessarily obvious upfront. But then as they build their application, in particular, understanding, I think, safety and reliability and trustworthiness in real world conditions is often they have a higher bar for that kind of robustness in specific applications than perhaps the infrastructure layer is going to have.

• 32:29Craig S. Smith?

  Yeah, yeah. So enterprises, I mean, if I'm an enterprise and I'm building an application and I have a choice of, you know, eight different models to hit for my inference, you know, on that through an API. I, you know, I've seen sort of casual rubrics where, you know, Claude is good for this, ChatGPT is good for that, and, you know, Gemini is good for something else. But I mean, who knows on your specific use case. So are you having enterprises come to you and say, we need to figure out which

• 33:12Craig S. Smith?

  model works best in our application and do A B tests, as you say, or A B C B tests or something?

• 33:21Speaker 01

  Yeah, I mean, I think that's you've precisely laid out one of the challenges that we want to tap tackle, which is many of this evaluation is based on vibes and based on intuition that domain experts have. We want to bring a layer of objectivity and rigor to this evaluation, which I think is going to be particularly important in enterprise applications and particularly in in regular regulated or sensitive domains like healthcare, finance, law, things of things of that nature. And then on the second part of your question, that choice of base model is just the first step, right? I think there's many more steps in then in terms of testing for safety and having a continuous flywheel of feedback, both from automated.

• 34:03Speaker 01

  Evaluations, we've talked mostly about human judgment evaluations, but there's also a whole strand of technical and automated evaluations, which is a complement to the to the human judgment. And then also understanding like the delta between the performance that they, an outcome that they're driving towards in their, for their application and like, and then the gap between that and their perhaps fine tuned model or customized model. And then critically, the, the tooling and the products to help them bridge that gap between the capability that they're after and the kind of baseline capability that they get with the, the existing model.

• 34:39Craig S. Smith?

  The, you mentioned agents in the, in the very beginning and, and in the context of ensuring that you're, you have a human on the other end doing the evaluation and not an agent. But the agents are being employed in evaluation and they're getting better. Is there a reason why your industry wouldn't eventually fade away as agents play that role?

• 35:07Speaker 01

  Just to be clear, we are, you know, active users of agents and LLM as a judge and similar tools in order to augment human performance and make sure that we're using the relatively expensive component of the human judgment in an intentional kind of cost, cost effective way. I think there's a, a few reasons why I think at minimum, it's going to be a long time before the human component dies out. Firstly, I think the human judgment is where the alpha is. So if you're looking to push the capability beyond what is already available, almost by definition, you are not able to build a, an automated evaluator that is, is

• 35:48Speaker 01

  able to assess that gap in capability. So you need human judgment, human judgment there. Of course, in cases like chess, the models have gotten to a stage where the, the human judgment is no longer kind of net, a net benefit to improving the models. But I think chess and similar verifiable domains are a fairly small set of the.

• 36:11Speaker 01

  Yeah. Useful tasks and wherever there is kind of ambiguity or subjective opinion required, human judgment is going to be in the in the development lifecycle for a long, long time, as well as, I think, crucially, the trust and safety component, where I think at minimum, having an escalation path where the model confidence is is low or where you want to cherry pick and just make sure that the the models and evaluators are performing as expected. So certainly, I think the human judgment and the humans' role in this lifecycle will change, I think, but the relative value that it provides is going to increase. That makes sense.

• 36:50Craig S. Smith?

  Yeah, absolutely. Yeah, and I didn't ask that question as a challenge to your viability of just imagining the future, but that's that is one of the really interesting things about AI. I mean, these these large models is they encode human knowledge off of text and there's a lot of ambiguity and a lot of misinformation included in the text. And then we'll ultimately need humans to guide the models. And it sounds, you know, you didn't come back with a number, but it sounds like there must be millions of people around the

• 37:31Craig S. Smith?

  world engaged in this. Do you think that's an overstatement? Millions?

• 37:37Speaker 01

  I don't think that's an overstatement for sure. And also, I think the many more people are going to be involved at minimum in a passive way as more and more of these models are a rollout in as as products. And I think just maybe touch on your other other point, we've worked in with academics in the field of human computer interaction for for a while. I think that field is is shifting maybe towards like human AI interaction or human agent interaction. So I think there's a lot of work to be done in in actually understanding like how to get the best of both humans and AI systems. And how do you make the collaboration and the engagement between them better than the sum of some of its parts as where this is not necessarily a kind of obvious how it's going to work a priori. And I think

• 38:20Speaker 01

  will require quite a bit of intentional effort in order to understand how to develop systems that work well for for humans and make sure these agents and systems are human centered and are supportive of folks' work.

• 38:36Craig S. Smith?

  How much AI do you use in a prolific, I mean, either in, you know, managing the platform or sourcing participants or yeah, I mean, how is there AI in your platform?

• 38:51Speaker 01

  Yes, and an increasing amount. I think it's an interesting fact that AI is quite a convergent force in in technology in the sense that increasingly, I think products are evolving to the magic AI boxes where you type in your request and you expect the AI to do its magic and return its return its result. I think the very early version of the we've rolled out is on the data collector side, the audience requirement tool. So you just express who you're looking for in natural language and we go away and we use the data that we have on our participants in order to find the right people rather than you needing to really think about the and or requirements of how you select for for those people.

• 39:34Speaker 01

  And similarly on the contributor side, instead of asking them a battery of questions to understand who they are, we're able to offer them a path where they're able to interact in natural language through audio and video. And then we're able to use that richer qualitative data in order to extract what's important to the other side of the platform and improve that matching. And I think that abstraction will increase where the the magic box will you'll ask it instead of find me these people, it will be okay, and design me the data collection tool and please go away and actually execute on the the work and then ultimately maybe automate the full workflows. So the researchers or the data collectors can stay at the outer loop of the the project and designing the the hypotheses and the

• 40:17Speaker 01

  the outcomes that you want. But increasingly, the the agents and the AI tools that we build allow them to automate a lot of the the details.

• 40:26Craig S. Smith?

  Yeah. And you're doing that now or that's on the roadmap?

• 40:31Speaker 01

  This is the roadmap. So the two examples I gave at the start of the define the right audience through natural language and the understanding of the contributor's background through AI interaction. Both of those are live products. And I think early examples of more to come.

• 40:47Craig S. Smith?

  Yeah. How does the, how do the customers, whether it's an individual PhD student or a foundation frontier model company, how do they pay? Is this a subscription or pay as you go?

• 41:05Speaker 01

  Yeah, it's a fully usage based billing cycle. So we will recommend the recommended pay to the contributors and then we take a portion of that payment, but it's fully, fully usage based. So you can go from tens or hundreds of dollars through to tens of millions. And it's purely just the amount of data that you collect and the amount of time that you ask from the contributors.

• 41:29Craig S. Smith?

  And how much of your business is kind of retail in that sense that, you know, people are coming on the platform for a short project and then they're off? And how much is it big enterprises that have longitudinal studies going on that they need you guys for?

• 41:54Speaker 01

  Yeah, we span the spectrum of budgets and wallet sizes and use cases. I'd say the vast majority of users have recurring use cases. If you're in the business of collecting data or running research, typically you have many, many projects, or even if you move companies or universities, bring your preferred tools with you. So yeah, we see the majority of customers come back on a repeated basis.

• 42:20Craig S. Smith?

  Yeah. You mentioned polling also is, do people use you to run surveys?

• 42:28Speaker 01

  Yes. Again, this is a use case that is supported. And again, I think something that we're increasingly interested in as the complexities of running high quality polls is becoming more and more challenging for a variety of reasons, harder to reach people, harder to incentivize people. So I think there's an interesting role that we can, we can play there.

• 42:52Craig S. Smith?

  Yeah. And on the compensation side, each project is the contributors are paid according to the project. So there isn't a standard hourly fee or task fee that they're earning. And is it, does it, is it enough that people, it's, they can supplement their income certainly, but can people earn a living income as a contributor?

• 43:22Speaker 01

  We don't optimize for folks who want to earn a living on the, on the platform. So primarily, it's a side hustle or supplementary, supplementary income. And yes, all of the projects or the pay is based on some combination of the length, complexity of the project and complexity of the requirements on the, on the audience, on the audience side. But all of that is made transparent to both sides of the platform. People have that, yeah, openness, ability to kind of, it's a, it's not a black box in terms of us going away and not showing how the sausage is, is made. The data collectors and the participants are able to interact with each other and see both sides of the platform as well.

• 44:00Craig S. Smith?

  Yeah. And where, and where do you see prolific going? Is, I mean, as I said, there's, I would guess that there's growing demand as more and more companies release these models. But then there's also, it seems to me there would be opportunities, for example, on the polling, where not only do you source the respondents, but you could run analytics on top of those responses. I mean, if you're, if the data is flowing through your pipeline. So what's, where do you see prolific going?

• 44:40Speaker 01

  Yeah, totally. I think it's exciting use cases across a range of applications. And ultimately, our ambition is to become a full stack human data platform. So provide a global, high-quality participant pool where you're able to access both the breadth and depth of humanity, the tooling, whether that's through partnerships or through our own tools. To tap into those humans for a wide range of different applications. And then supporting the the methodology and the the tooling really to advance the frontiers of of research in AI. And yeah, mu- much more work to be done on all of those strands.

• 45:19Speaker 01

  And yeah, many exciting opportunities, both in the developing transformative or human centered AI. And then also really understanding the how this is changing and influencing human behavior. And really being the platform that provides this authentic, high integrity, human data in the age of lots of AI generated data as well.

• 45:42Craig S. Smith?

  Yeah, yeah, and I know I'm up past the hour. If you have a time for another question or two.

• 45:50Speaker 01

  Unfortunately, I probably need to drop.

• 45:53Craig S. Smith?

  Oh, you.

• 45:54Speaker 01

  Yeah.

• 45:54Craig S. Smith?

  And I have one more question.

• 45:56Speaker 01

  Absolutely.

• 45:57Craig S. Smith?

  Yeah, there there is last year or in the past year, there's been this boom in humanoid robots. But the critical block bottleneck for humanoid robots is data collection. You know, because with L L M's, you've got the internet full of text. But humanoid robots, you need, you know, data from humans manipulating objects or, you know, identifying things and scenes. Are you doing any work on data collection for humanoids?

• 46:30Speaker 01

  Uh, yes, it's not a the the core of the work is done on the platform to date. But is, I would say, an area of open discovery. In particular on, so you mentioned you had a Fifi Lin on the podcast recently. And she probably talked quite a lot about world models being a

• 46:48Craig S. Smith?

  Or a, yeah.

• 46:48Speaker 01

  Kind of a required required environment in order to accelerate the the training of embodied AI. So we've actually done some some very interesting work on integrating prolific into virtual environments. So participants can take part in data collections in in in VR. So I think we've gone from kind of text, images, video. I think world models and virtual reality are an interesting modality that we'll definitely explore in the in the coming coming years.

• 47:15Craig S. Smith?

  Okay.