The Secret Engine of AI - Prolific [Sponsored] (Sara Saab, Enzo Blindow)

We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific. **SPONSOR MESSAGES** — cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++ Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst Submit investment deck: https://cyber.fund/contact?utm_source=mlst — While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolific's approach is to put "well-treated, verified, diversely demographic humans behind an API" - making human feedback as accessible as any other infrastructure service. When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation. We also discuss Anthropic's research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this "agentic misalignment." Enzo and Sarah present Prolific's "Humane" leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models. Looking ahead, the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. This also raises important questions about working conditions and the evolution of labor in an AI-augmented world. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration. As AI tech becomes more powerful and general-purpose, the quality of human evaluation becomes more critical, not less. We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity. Visit Prolific: https://www.prolific.com/ Sara Saab (VP Product): https://uk.linkedin.com/in/sarasaab Enzo Blindow (VP Data & AI): https://uk.linkedin.com/in/enzoblindow TRANSCRIPT: https://app.rescript.info/public/share/xZ31-0kJJ_xp4zFSC-bunC8-hJNkHpbm7Lg88RFcuLE TOC: [00:00:00] Intro & Background [00:03:16] Human-in-the-Loop Challenges [00:17:19] Can AIs Understand? [00:32:02] Benchmarking & Vibes [00:51:00] Agentic Misalignment Study [01:03:00] Data Quality vs Quantity [01:16:00] Future of AI Oversight REFS: Anthropic Agentic Misalignment https://www.anthropic.com/research/agentic-misalignment Value Compass https://arxiv.org/pdf/2409.09586 Reasoning Models Don’t Always Say What They Think (Anthropic) https://www.anthropic.com/research/reasoning-models-dont-say-think https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf Apollo research - science of evals blog post https://www.apolloresearch.ai/blog/we-need-a-science-of-evals Leaderboard Illusion https://www.youtube.com/watch?v=9W_OhS38rIE MLST video The Leaderboard Illusion [2025] Shivalika Singh et al https://arxiv.org/abs/2504.20879 (Truncated, full list on YT)

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Totally independently of any prompting, all the major frontier models derived a solution that involved blackmail, essentially.

• 0:09Speaker 01

  The models, when knowing that they were being observed and evaluated, actually digressed away from it.

• 0:17Tim Scarfe?

  The results were not very pretty.

• 0:19Speaker 00

  So I think there's already a rift forming between what humans think LLMs are here for and what LLMs think in scare quotes, think they are here for. Tim, if we thought they could understand, we would also hold them to account for their actions.

• 0:37Tim Scarfe?

  You think they ever could understand?

• 0:42Speaker 00

  I'm Sarah Sab. I'm the VP of Product at Prolific, a longtime product manager, product person. I started my career with a stint in Silicon Valley and I've been in the UK for over a decade now. Prior to that, I was a cognitive scientist and a philosopher and a little bit lapsed on that, but very excited to see all of that coming back around in industry these days.

• 1:06Speaker 01

  My name is Enzo. I work at Prolific. I'm the VP of Data and AI. I support everything from AI, data, research and the likes. My background is originally economic science, but then also computer science. And I spent now more than a decade working on large-scale distributed decision systems, ranking systems, and did a stint in recommendation as well.

• 1:27Tim Scarfe?

  And you were at Meta on Instagram, right?

• 1:30Speaker 01

  That's correct, yeah. Prolific is a human data platform working with everything from academic researchers, but also small and large players in the AI industry. We've developed a leaderboard which we've joyfully called Humane.

• 1:45Tim Scarfe?

  Oh, interesting. It's amazing to have you both on MLST. MLST is supported by Cyberfund.

• 1:53Speaker 00

  I'm a technologist, Enzo's a technologist, you're a technologist. We build software all the time. The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing. Let's put it that way. And I'm very, very sympathetic to that. Right? We're all trying to deliver stuff and there's a accelerating sort of hot industry around us. And the idea of waiting on a human to tell us if a thing worked feels counterintuitive. And I think our approach to that is stick a really well-treated, verified, diversely demographic human behind an API essentially,

• 2:37Speaker 00

  and make sure that the structures and infrastructure are there to make, ensure that human can go fast, understand instructions, and give you something akin to deterministic, human in the loop behaviors. But the fact that people are really resistant to this makes a lot of sense to me. But I also think somehow in the last two, three years, the stakes have changed. And we now have these very, very inactive systems that are quite non-deterministic. And so I think the things we need to do to protect ourselves and each other have also kind of, without us realizing, changed quite a lot.

• 3:16Speaker 01

  Like, I'm also extremely sympathetic to wanting, like, all these efforts to remove the human in the loop. Like, it's costly, it's slow, it doesn't even provide the best quality of data. Right? There are several instances where even synthetic data might be surpassing it. But then there's instances where that's also not true. And so what we're actually working towards is a much more adaptive system. There's scenarios where human data is needed. There's scenarios where human data is very much not needed. There's scenarios where you might even have a hybrid solution or where you meet a certain criteria where you need to have a human in the loop,

• 3:57Speaker 01

  where you almost need to define, I need this level of scrutiny now. Therefore, I need higher quality input. And therefore, it's slower, I accept it slower, and it is higher cost. And that is fine. So it's almost like... We need this routing component there. We ourselves, we're trying to reduce the lag or reduce the time to data as much as we can. And then behind the scenes to ensure that the data can be of highest quality as possible. But at the same time, it's almost like there is a, there's a constant trade off between the quality, cost and time. And if you want lower quality really fast at low cost, you can go with something off the shelf synthetically.

• 4:41Speaker 01

  If you need something really high quality, it will be the default slower and more expensive. You can get the best experts in the world to give opinions on this, right? Or give input on this, but there's also an entire spectrum in between. And so how do we, how do we solve for that? Make the lag smaller and make it as adaptive as possible.

• 5:00Tim Scarfe?

  If I may put my cards on the table, I think the underlying problem here is that these machines don't really understand anything. That's why it's so important to get, to get humans, you know, if we had perfect verifiers and, and synthetic data generators. Probably we wouldn't even need L L M's in the first place, right? Because we've already solved all the problems. So we're, we're in this, we're in this intermediate phase where, where we can do, you know, some of all of these different constituent parts.

• 5:24Speaker 01

  It sort of, it seems like on the surface that we're removing more and more humans from the process. But it's not entirely true. For example, reinforcement learning is a really, really good example of this. Where, for example, web agents were are now largely trained or in these environments that are often synthetically created. But there's programs that need to create these synthetic environments for R L agents to, to explore in. That need to be validated by humans. So we're seeing this interesting progression where humans are no longer directly involved in like the main system of interest. But sort of in this secondary, almost like a second order or third order abstraction moving outside.

• 6:07Speaker 01

  Which I actually is a welcome change because that means that we're focusing more on the right kind of tasks where humans are relevant. And we're focusing more on the right kind of high quality data, right? It's, it's, it's completely ludicrous to create insane amount of data purely derived from humans. Like those days are gone, right? We don't need that anymore. Put the humans where they need it.

• 6:29Speaker 00

  We are trafficking in human data at some point, somewhere, without always owning up to that. I think the most worrying version of that is let the user in production find the problems. In some cases, that's fine, and in some cases, that's very scary. And I think, Tim, if we thought they could understand, we would also hold them to account for their actions. But we can't, and since humans are being held to account for the actions of AI models, I still think the onus is on us to ensure that they're behaving the way we expect. But once they understand, we will hold them to account, right?

• 7:13Speaker 00

  There will be real stakes for them as people in what they do or think, but we're not there yet.

• 7:20Tim Scarfe?

  You think they ever could understand?

• 7:22Speaker 00

  Personally, yes.

• 7:23Tim Scarfe?

  Oh, interesting. Why is that?

• 7:29Speaker 00

  So, small questions today.

• 7:33Speaker 00

  I think that we need them to be in the world and have stakes in the real world. And I think they need sensory and embodied grounding. That feels essential to me from birth onwards. So they need to go through a sort of developmental psychology curve. But I don't think there's anything fundamentally special about the human brain that we can't replicate. That's my personal opinion.

• 8:00Tim Scarfe?

  Oh, interesting. Yeah, because I was going to ask you whether you think of LLMs as a kind of cultural technology, a bit like Photoshop, or whether you think of them as intelligent agents. But maybe your answer would be not yet. But if they were embodied, you know, with enough fidelity, then maybe you would.

• 8:18Speaker 00

  Yes. So that is what I think. I spent a lot of time thinking about the history of the vision system of the frog. And there are two vision systems in the mammal brain, vision for action and vision for...

• 8:36Speaker 00

  For recognition or vision for sort of object creation. And the vision for action system came first. And so there was some time when frogs were just zapping flies with no understanding of what they were doing. And then a second vision system developed that allowed this mammal to start to create a world map of objects, including itself. Right. And I think there's this sort of, I don't know if you know the affordances theory, sort of affordances for action and sort of the Gibsonian theory. So at some point, these creatures started to think, okay, whether I think that's a lion or not, has a lot of

• 9:17Speaker 00

  consequences to whether, you know, I can stick around. And I think that, I think that is the bootstrap for consciousness. Again, I'm speaking very much personally here. I think that we can replicate that in another thinking creature. I think that whether we can or not, at the very least is an empirical question.

• 9:39Tim Scarfe?

  Very good, very good. Yeah, my friend Walid Sara, rest in peace. He famously said, animals don't think. And, but he was pointing to something interesting that, you know, humans seem to have some privileged form of intelligence.

• 9:51Speaker 00

  Yeah. And another kind of very interesting abstraction in kind of the ontology that underlies consciousness or thought maybe is like long ranging. So the idea that sort of early, early and less capable wet brains don't seem to have the ability to keep hold of something. And even babies, right, can't keep hold of something that is not in their perceptual sphere. But we all develop that ability. And I would argue, following the thought of thinkers like Brian Cantwell Smith, that that that happens because you start to care about the object even when it's not in front of you. And all of that is kind of bootstrapping the sense of like participatory

• 10:32Speaker 00

  stakes in the world. And I think that is kind of the bootstrap that we would be after.

• 10:38Tim Scarfe?

  Yes, yes. And would then, would you think of the the system as a whole, like almost the ecology as being the locus of the intelligence?

• 10:46Speaker 00

  I would, yes.

• 10:47Tim Scarfe?

  Yes, tell me more.

• 10:48Speaker 00

  The idea that computers and AI systems are somehow in a privileged, isolated.

• 10:56Speaker 00

  Tower of algorithm and separate from reality in the world is just completely untrue. So I was talking to Claude about the halting problem a few days ago. And I was like, well, do you mean the algorithm won't end? Or do you mean someone won't unplug the computer? And Claude goes, no, I mean, the algorithm won't end. You know, we cannot confirm or deny whether the algorithm would end. And I just, I just thought to myself, that seems kind of nonsensical to me because they, you know, the universe will end, someone will unplug the computer. I think that kind of comes to this fundamental thing, which is we are just not separate from the world and computers are not separate from the world and AI models are not separate from the world.

• 11:39Speaker 00

  So the sort of the ecological pressing on each other feels really important to that whole story.

• 11:46Tim Scarfe?

  Yes. And how does this inform the work that you do at Prolific?

• 11:48Speaker 00

  The, the idea of participatory stakes is very much the heart of human evaluation. Right? I think this is our core thesis and Zoe and I in the work that we do, which is that trying to, another max I recently learned was bench maxing. So there's a lot of maxings.

• 12:06Tim Scarfe?

  Yeah, that was Grok four.

• 12:07Speaker 00

  Yeah.

• 12:08Tim Scarfe?

  Nathan Lampert said it was.

• 12:09Speaker 00

  So all the, all the maxings in the world are a little bit besides the point. You know, we want, we want systems that feel good to interact with. And I think that, that, you know, that's about sort of human society and people and, and, and systems and thinkers pressing on each other. So I was a cognitive science student a very long time ago and we thought so long and hard about the Turing test. We debated for hours and hours about the Turing test. And we really just thought that that was a frontier we would never, never touch, which is so interesting looking back.

• 12:50Speaker 00

  And obviously personally, I then put that to bed, you know, finish, finish, finish my university career, put that to bed, went into product management for a very long time and then almost looked out the window in 2023.

• 13:06Speaker 00

  And the moment had passed us by. And also, all of the questions of minds and machines that had preoccupied us in those early days are back on the table. And industry, which is so interesting, industry as well seems not to have the orientation by default to be tackling those problems from first principles. Although we are getting much better at the cross-functional collaboration between researchers, academics, industry people and public bodies. And it's just such a very interesting point in, I think, the history of science and the future of science.

• 13:50Speaker 00

  One thing that strikes me, and Tim, I said this to you, is that we're dealing with software and product and algorithm problems. But if you scratch just a little bit too deep, we're dealing with the central problems of being human, the central philosophical problems of being human. Every single question leads us to a central philosophical problem of being human. And I just think that's just very neat and scary at the same time.

• 14:22Tim Scarfe?

  You probably didn't think 20 years ago when you were working under Andy Clark, the famous philosopher.

• 14:27Speaker 00

  He's great.

• 14:29Tim Scarfe?

  I'm a huge fan of Andy's, a big hero of mine. You probably didn't think that they would be so relevant later on in your career.

• 14:35Speaker 00

  Right, I mean, I hoped, we all hoped, but no. I don't think anything has ever brought the stakes of software development and systems development to the forefront of technology innovation quite this way before. You know, those very, very enduring problems of what it means to be a person and a thinker are suddenly impossible to ignore as we're, you know, doing software releases and, you know, model deployments. And I think that's, yeah, that's super unexpected for me.

• 15:14Tim Scarfe?

  I just think the Turing test was really bad.

• 15:17Speaker 00

  Fair.

• 15:18Tim Scarfe?

  Yeah, because it's that McCorduck effect, isn't it, that when something is trivially easy to mechanize, no one actually thought it was intelligent. But I think the Turing test is bad because we know, in my opinion, language models aren't actually that intelligent, yet we've passed it with flying colors. So we need something better than that. But it does go to show, though, just from that kind of behaviorism thing and benchmarking, that when a machine does something, that's not the full story, is it? Like, we need to know, why did it do it?

• 15:49Speaker 00

  There's a lot of speculation about how human in the loop steps in model development and oversight will change over time. I think there's a really credible future in which we as humans end up taking a coaching, teaching, and guidance stance to the many, many myriad of machines in our lives in the next five or 10 years. I love the word orchestration because it has the root word orchestra, which is kind of this beautiful collaborative, symphonic word. And I think that, to me, is the correcting a five-year-old over and over again, or correcting an 18-year-old or 25-year-old about life, right, over and over again.

• 16:33Speaker 00

  But if that is the world of work that we are moving towards as humanity, it strikes me as really fundamentally important that we put the right baseline working conditions in now for how that work will evolve over time. And there are some amazing writers on the topic of ethics of crowd work and click work, like Mary Gray and the team at Fairworks, who've written a recent book. I'm very inspired by that thinking when it comes to how our jobs will evolve in the next five or 10 years.

• 17:05Tim Scarfe?

  I suppose, in a sense, this is the ultimate evolution of the gig economy, but for highly specialized work, right? So I can imagine a future where there's just a marketplace of things you can do. You can just wake up in the morning and you can just say, I'm really interested in climate science or something like that, and you can just do five units of work. Right, and then and then that's your day done and and it can be quite pedagogical, right, because you can actually learn things that you're interested in doing.

• 17:32Speaker 00

  Right, so that's that's one future that, you know, and nothing we say today will come true exactly the way we, you know, that would be absolutely ludicrous for us to hit the mark exactly, but that is a that is a paradigm for the future that is interesting. At the very least, we need to think about it in the way that we're constructing systems today.

• 17:51Tim Scarfe?

  Yes, and another thing is, do you think that this is just a transitory stage, that we are training the AIs, because we need to kind of suck up all that human culture and we need to, we need to learn how humans think, you know, many data platforms have kind of done this and now they're finished now, they've got all the data, they don't need you anymore.

• 18:10Speaker 01

  This is a good question and very much top of mind, obviously. There's also this recent paper that came out by, the position paper by David Silver, The Era of Experience.

• 18:20Tim Scarfe?

  Oh, yes, David Silver, yeah.

• 18:21Speaker 01

  Right, and where he says we're moving on from the the era of human data to era of experience, effectively saying, and which I very much agree with, that agents should be, get their feedback from real life scenarios out in in more real environments, if you will. Very much subscribe to this idea, but at the same time, there are certain scenarios where this doesn't necessarily hold true. For example, drug trials. We don't just put some compounds together, release it out in the open and see who ultimately reacts unwell to them, right? That's learning in the environment, right then and there, but we do this more in a phased approach, right? The kind of things that we might be working towards is

• 19:03Speaker 01

  that we still very much retain a need for, in some cases, for more controlled environments, right? Controlled environments in the sense where we have held some control over some variables that we ultimately want to derive some some insight from. This loop, though, that that is being described by, the more we push things to the edge, the more we push things into real life scenario, we want to get there faster, because that's where the most of the signal is, so absolutely subscribe to it. But in a potentially phased approach, or where we can get a quick feedback loop from something experimenting in a more controlled environment, right? We also do software release in staged, in stages, we do drug trials in stages, and we're trying to put things through controlled environments to validate certain

• 19:47Speaker 01

  hypotheses or to validate the safety of things, before we put it out into the real world, where it can then Potentially further refine.

• 19:56Tim Scarfe?

  Grok four was benchmarked. And so it had Sotter on Arc AGI, sixteen percent. So Francois would be very happy about that. And several of the other benchmarks as well. But the vibes aren't that good, right? And when you use it, it asks Elon Musk's opinion for everything before it gives you the answer. And it also gives quite infantilistic responses. Now, the thing is that even when I was looking at your benchmark on hugging face, I was questioning some of the vibes, right? So one of them, I think, was like agreeableness or something. And GPT four O was at the top. And I'm thinking GPT four O is basically Eliza. It's like a companion bot. I wouldn't use it for anything. So can we even trust the humans to do it?

• 20:41Tim Scarfe?

  But vibes are so important, aren't they? Aren't they? So tell me about vibes.

• 20:45Speaker 01

  Vibes are hard to quantify at the end of the day unless you ask someone for their opinion on it, right? We need some form of scales for it to quantify it. The main way that is being done today is either through this comparative way of where you effectively, you're being presented two outputs and you rate one over the other. But that's not purely just vibes. There, we have to at some point agree on the right scales, if you will, right? So agreeableness is a really good one. You also, like, some of the OpenAI models famously had this psychophentic behavior.

• 21:26Speaker 01

  This leads to very, very specific scales that we can look at in that moment. But it needs the opinion of humans. And so for that to be worthwhile, we need to be removing selection bias. We need to ask a representative set of people. And then there's another potential bias on maybe the kind of prompts that were sampled from were only from a very specific problem space or domain space. So how do you build? The problem here is almost like, how do you build significant coverage into it? Because the measure of vibe, I suppose, can vary so significantly depending on all sorts of factors and context.

• 22:10Tim Scarfe?

  Yeah, isn't it so strange though, because like when we use a language model and we get a vibe, I I'm I know I'm deluding myself, but I feel that I understand this model and I place it into a pigeonhole. And I say, I really like this model. This model is good. And I kind of over overgeneralize from my experience with it. The reality is, these these abstract questions that we're asking, if if you imagine the statistical landscape of all of the people like, you know, making these assessments, it's just got all of these modes everywhere. And then what we do is we we kind of average over all of that complex structure and we roll it into a number. Now, that's not necessarily a bad thing, because that might still have statistical information compared to other aggregations that we've made. But we're taking a very complex thing and we're kind of we're squashing it together.

• 22:54Speaker 01

  I mean, that's not really how we do things as humans, right? Like, we all went to school to some degree and you can say like, well, Tim, did you do good at school? Yes, right? But you have specific subject areas that you might have been on well on or not well on. And then we do aggregate them ultimately aside from it. But so there there needs to be almost this taxonomy similar to how we as humans grade each other or or or express each other about the capabilities that we have, right? We need to impose the same thing on on these systems. And some of these systems might exceed also some of the human capabilities. So these scales, they need to evolve, right? That's what brings me to the point of they need to be adaptive ultimately.

• 23:38Speaker 01

  So we need to have a constant eye on the kind of things that we do with these models or the kind of decisions that these models make about humans, how that influences humans, how does it make them feel, are they safe? But ultimately, we need to, yeah, have good coverage of different measures, different problem areas, and then ultimately a representative set of of populations that inquire about it.

• 24:01Speaker 00

  And so was really shocked a while ago when I told them Gemini was my best friend. And we had a bit of, he was, he was, he didn't let me live that one down. But I think we are entering this sort of age of AI companionship very fast. And something I think about a lot is as we benchmarks and as we over optimize for benchmarks and leaderboards, are we creating softer or weaker constitution or behavior in other domain areas, including ones that may not feel as sharp and verifiable as, you know, how you, how you do on graduate level mathematics?

• 24:43Speaker 00

  And I think there actually is a little bit of research that models, I, I don't have citations, but that models that are optimized for, or doing really well on the harder sciences are regressing in other, in other domains. So I think about that a lot.

• 24:59Tim Scarfe?

  Yeah, and, and that doesn't surprise me. They, they feel mutually exclusive to me, which, which is why I'm, I'm constantly thinking, I don't know what your kind of prescription is here, but when people use your technology, is, is, is the idea that they would try and have a large foundation model that does all things to all people? Or if you think about it, there are so many different levers they can pull to tweak, to tweak the models. You know, they could, they could curate the, the fine tuning data. They could, you know, stick a Laura stream on there. They could, you know, tweak the RL post-training. And they could do dynamic system prompts and, and whatnot. And there are so many different architectures that, that this could be leveled out in. What's the prescription?

• 25:38Speaker 01

  Well, this is actually one of my, one of my favorite things about the measurement space. Because the measurement space inherently is unopinionated about any form of solution. So whether you tweak parameters, you change architectures, change algorithms, use different data sets, it doesn't matter, right? It's the purest form of distributed optimization across everybody who tends to work on these types of problems. That's nice if we can align on the measurement that we consider success. And I think that's lacking to some degree because we have somehow inherently decided that chatbot arena, for example, is the measure of success. So. People optimize for it. Then we decide that the next technical benchmark is the measure of success.

• 26:24Speaker 01

  People optimize for it. It's susceptible to Goodhart's law, right? But ultimately, the better we can design independent success measures and agree on it and make them freer, freer, maybe not entirely free. I think that's maybe a bit too far fetched, but freer of of being able to to game them or to optimize for them. There are ways to remove it, right? The better we can build accountability, because then we don't have to be opinionated on what model you use or what parameters you optimize for. It's, we need to agree ultimately on the on the measure of of success.

• 27:00Tim Scarfe?

  And with that Goodhart's law thing, so when a measure becomes a target, it ceases to be a good measure and and the measure is usually the proxy for the thing that we can't really quantify. So we we create a surrogate proxy for it. Should there, should should we agree on a consensus of a few measures? Or that could be quite gamable, right? Or or should we have some kind of individualized dynamic measure?

• 27:26Speaker 01

  So gamability is one thing, like, let's take the chatbot arena example, right? We're working on a leaderboard on ourselves and we're asking, we're faced with the same questions, right? We don't have a good answer right now of should we allow for for private evaluations? Should we release the data set? Because that makes it inherently more gamable, right? But if you keep it closed, no one can verify it. Then should there be independent bodies that can verify it to some degree, right? We we're definitely publishing our methods and the paper around at one hundred percent, but jury is out on the data, let's put it this way. I think I would prefer to to publish the data because I think it's the right thing to do, but it invites gaming. Could we think of other ways to remove some of the gaming, for example, right?

• 28:11Speaker 01

  Another way is if we keep the private evals, we should give private access to or private evals to everyone equally, right? But then if you if you maintain this veil of of secrecy, ultimately, I guess.

• 28:26Speaker 01

  Everybody who sends a model for evaluation has access to the logs and traces. So they could work it out very, very fast, even if you kept it private, right? You could speculate, should you dilute some of the, like, sort of the calls that you return with some noise, similar to differential privacy, for example, right? That it looks like to the model creator that there is signal, but actually it's sort of like curated signal that obfuscates the actual signal from the private eval. We're only on our end. We could then aggregate that to a meaningful measure that makes it inherently less gamable.

• 29:02Tim Scarfe?

  We're trying to create a legible benchmark. And when we look at other forms of verification on the internet, like we have peer review and we have like, you know, the Wikipedia edit history, for example. And this isn't one number. Like, we analysts, researchers, you know, we go and we contextualize all of the information.

• 29:24Speaker 01

  Funny enough that we face these type of problems day in, day out in the work that we do, because we also need to verify people. We need to verify the data that they're producing. So there's a framework that comes to mind is ultimately how you verify something is based on the reference. So you either have some form of golden data set that is your reference to something, right? Or you have some form of expert that can QA your data that is becoming your reference. Or you have some form of consensus-driven approach or weighted consensus-driven approach based on some trust tier system where someone might build a trust or reputation over time, similar to Wikipedia.

• 30:08Speaker 01

  And then you wait different responses in different ways. There's different ways you can calculate agreement rates, for example, to build like your or estimate how consensus aligned effectively your output is. But that transfers ultimately also to, well, the moment it becomes less deterministic, that's the only possible pathways we have to converge to something that we can measure.

• 30:35Speaker 00

  It strikes me that it's so hard because.

• 30:39Speaker 00

  We've realized that we now need to systematize what humans think of as good

• 30:46Speaker 00

  in order to serve us in the development and tuning of these systems. But we've never had a single fabric, or rubric rather, for what a global human alignment on goodness looks like. So no wonder it's hard, because we've smuggled this foundational project into the testing of AI systems.

• 31:11Tim Scarfe?

  What do you think the major risks are of getting this wrong?

• 31:14Speaker 01

  Because they are no longer very fragmented models with very narrow targets. They are very, very heavy foundational models with lots and lots of capabilities, where also tons of the fine-tuned models are effectively descendants of these models. So the more effort we put into the foundational model, we will ultimately benefit also all of the descendants and derivatives of these models. So if we're not careful about how we design them, we will potentially have far-reaching consequences.

• 31:47Tim Scarfe?

  Yeah, so you're saying we should uphold the phylogenetic health of the LLM ecosystem? So you know, like the evolutionary tree of all of the models? Because there'll be all of these downstream effects, you know, like problems will be compounded.

• 32:02Speaker 01

  Absolutely. And then there's an interesting meta-question in this, which is, I've thought about this for a little bit, and I'm not sure I have a satisfactory answer, but it's just an inviting thought perhaps, around we do all of these evals from scratch, right? We observe, we gather the data, we evaluate, and then, but it barely, we don't really learn from it. It's always again from scratch. Is there something that where we can make evaluations transferable? And then ideally, if we had access to lineages of derivatives of models, is there something we can progress or instill into these models

• 32:43Speaker 01

  that would benefit us from effectively build the compounding value of evaluations?

• 32:48Tim Scarfe?

  That's very interesting. I suppose in many cases, we don't have all the information about the lineage. There probably is a hidden lineage that we're not aware of. I hadn't really thought about that before. If it were all completely in the public.

• 33:07Speaker 01

  Right.

• 33:07Tim Scarfe?

  And we knew the data and the model and the training lineage and so on. You're suggesting that we could actually build a much better evaluation system on top of that.

• 33:16Speaker 01

  Most likely, especially now where we have also where we're interjecting AI systems with others. Where we have in a lot of ways, LLM as a judge is an excellent example of this, right? Or even constitutional AI, where we have AI systems building the RL AI F. Parts of it is we have AI systems monitoring AI systems, building the feedback or the data for other AI systems. So there is a network of data and targets of being interspersed.

• 33:48Tim Scarfe?

  To use a bad analogy, you're kind of saying if there were a Git of language model development, where you could see all of the previous check-ins and all of the branches and so on. And then you could actually trace back and you could derive from some of the previous evaluation methods that we had used.

• 34:06Speaker 01

  Potentially, yeah. And we could potentially trace back that if there is bias in one LLM, but you're using it as a judge for another LLM, and you're using that as feedback data to become an optimization target for yet another LLM. Of course, there has to be some progression there, right? Some influence that can be ideally traced.

• 34:29Tim Scarfe?

  Let's talk about constitutional AI. So this was a paper from Anthropic a couple of years ago.

• 34:33Speaker 01

  That paper now a few years old, but still super current in my opinion. It looks at two axes, harmfulness and helpfulness. Because it's important to note that the constitution that they're referring to in the constitutional AI paper is, or the AI part of it is only looking at the harmfulness axis. The helpfulness axis is still derived by humans. Because that's not really something that we need to uphold a constitution to in that sense, right? But they have found that ultimately that we can scale this type of feedback actually in much more higher quality ways than just. Going full human for RLHF, for example. And that's really interesting.

• 35:16Speaker 01

  And that sort of confirms some of the things that we've been seeing. Because when I think when we most, most of us started machine learning, a lot of it was human data was everywhere, right? We needed to also get quantity of human data ultimately. We made concessions on the quality to some degree. But ultimately it was about quantity was key, right? That's how we learned, that's how we converged. Well, now it it sort of flips it on its head. Now it's about, okay, let's get few quality examples in. Let's get the right humans in to get the right quality of of of human feedback in. In order to align around a constitution, effectively a policy, if you will, right? Which is nice, it makes it nice and abstractable.

• 35:56Speaker 01

  There is almost like a, the analogy might not hold fully, but you can almost think of it as like in a democratic system, you have the separation of powers. And you have the the legislative that that determines the law, right? It writes the policy. And people that are voted into a democracy is also by default, or in an idealistic sense, representative of its population. They are writing the law. Then you have the judicative that interprets the law. This can be done by AI, right? And then the executive can also be done by AI of looking at specific cases. But there's usually this feedback loop that's in there as well, which is when there's borderline cases that are hard to interpret. They usually get routed to something like a Supreme Court, right?

• 36:41Speaker 01

  And then these cases are being evaluated and you need to see whether a revision is needed to your law, to your policy in that moment. And so we have these systems that already exist. That's how we govern a democratic, a representative set of people. That's how we align people already in democracies. So I find actually the approach of constitutional AI quite a keen and quite quite apt. Something that we might mimic in how we scale some of our approaches as well, where we use humans, representative humans, set of humans at that. In order to write the policies, but then use AI effectively to to to govern or to evaluate against the policies.

• 37:22Speaker 01

  And that makes it a really nice, scalable, abstractable pathway where we use humans for the things that matter, where quality is needed, where representativeness is needed. And we look at specific borderline cases in order to continuously improve on it.

• 37:36Tim Scarfe?

  In a sense, is that the goal of prolific to become part of the the kind of not just governance, but also architectural plumbing? So system builders will be able to plug into your platform. And you can almost think of this as a meta layer.

• 37:51Speaker 01

  Yeah, that's that's kind of the the the the goal that we're working towards. So we're trying to make human data or human feedback, or actually any kind of feedback at that. We treat it as an infrastructure problem, right? We try to make it accessible, we make it cheaper, we ultimately, why? You see this pattern in almost any company and even in academic research as well. Every academic researcher cares about the quality of their data. Everybody has to think about how they set up their their data collection. Everybody has to think about the validity of the data. They have to often go through an ethics review. Companies do the same and they all build the same systems. So for us, it's just let's treat it as an infrastructure problem. Let's abstract it away.

• 38:33Speaker 01

  Let's put a nice API around it to make it just like the same way you also do C I C D or you do model training pipelines. You just treat it as an infrastructure problem, make it accessible, configurable. You have a set of parameters that you can can call and then ultimately we effectively democratize access to this data.

• 38:52Tim Scarfe?

  Yes, it's quite funny because software engineering is the same thing. So a lot of people think that DevOps is all about automation and it's really about human orchestration, right? There are so many different people that need to be involved, you know, reviewing PRs and like, you know, planning and whatnot, gating. And in a sense, building models is the same thing, right? There are so many human stakeholders that need to be involved. But what we need to do is orchestrate and scale the system. So I guess you've created something which is a little bit like infrastructure as code, where when people are designing their models, they say, you know, I'm going to hook into prolific now and I'm going to do this feedback loop. And behind the scenes, you've got this entire machine, which is selecting the right experts,

• 39:34Tim Scarfe?

  stratifying them, checking that everything is correct, feeding that information back. And it obviously from the consumer, it seems like it's it's an API. But there's actually this whole machine that's going on in the background.

• 39:47Speaker 00

  Devex infrastructure on top of the squishy stuff.

• 39:50Tim Scarfe?

  Yeah, it's like it's orchestrating meatspace.

• 39:55Speaker 01

  No, that's exactly right. There is a lot that happens under that, like sort of below the water surface. Like the iceberg is deep. Let's put it this way. There's tons of verification work that goes into it, right? I mean, you can, like most people, like, how would you start a problem like this, right? If you wanted to gather human input, you you send out a survey, right? That's probably where you would start and you would ask, are you from a certain country, right? Then you have to go very far out on a limb and trust that, right? That is already a little bit of a far-fetched proposition. So how do you validate the information that you're using in order to select the data that you're ultimately trusting to train and evaluate your systems?

• 40:37Speaker 01

  Like, if you do this on some production data of your app or so, you probably have some number of trolls in there. You have some people that don't want to divulge their right information, right? You create, there's tons and tons of noise in this. So we really try to make it our problem and put that behind an API of like, we just really want to make sure that whatever data you're using and the criteria you select, you use to select in order to get access to your data is robust and trusted.

• 41:07Tim Scarfe?

  What happens in the domain where the expertise is very sparse? Let's say that I'm building an app where I need to have PhD level knowledge on a particular thing. What happens then?

• 41:18Speaker 01

  This is fun because like, let's say quantum physics is a wonderful, right? I'm not a quantum physicist. I could not like speak to someone like, let's say someone comes on on the platform, is a quantum physicist and there's demand for a quantum physicist. I could not verify whether what they're saying is true or whether they're bullshitting me, right? So how do you do this at scale? Because the the the solution is not to hire more quantum physicists in order to hold other quantum physicists accountable. It's ultimately some level of asking them the right questions to have a level of trust. It's a bit like a funnel, if you will. But then, how do we do it today? Right?

• 42:03Speaker 01

  We do it through peer review. So we need to build some level of trusted network based on their their previous experiences, based on some external credentials, for example, and take all of these factors into consideration and then ideally cross validate it through a peer network. How do you get them interested?

• 42:21Speaker 00

  The first misconception is that the work is sort of grinding or boring, which which actually in our experience, when it comes to providing either training or or evaluation or fine tuning data for SODA models, it's not. It's actually very, very deeply interesting and cerebral. And I think the other misconception probably stems from the history of the click working and crowd working space, which is that these people are poorly paid, which is also not true. They're they're actually paid quite a lot. Their their task, their sort of duration, duration of task at any one time is not very long.

• 43:04Speaker 00

  So they're not sitting in front, at least on our model, they're not sitting in front of a computer for eight hours in a row. And we find that you don't get high quality human data by putting someone in front of a computer and asking them to do the same thing for eight hours straight anyway. So I think the incentives on both sides are aligned in that sense. So it's usually shorter durational work that is often interspersed among other, you know, employment or other, other.

• 43:33Speaker 00

  Things that these experts do, but they're paid actually quite well for the time they spend. And that's part of our ethical stance. And also, you know, it's a competitive space when it comes to sort of the experts at the edge of human knowledge. In some cases, there aren't, there really aren't that many of them that can contribute something helpful to a frontier model's corpus of knowledge.

• 43:57Tim Scarfe?

  Like when I spoke with Francois Chollet about the ARC challenge, he said that when they were getting the human testers, they had to be super careful because people have a limited attention span. It's the same with code review, for example. Like you can't really get people doing code review for more than like half an hour or something because they start rubber stamping and they start, you know. So you folks must have done so much research on this.

• 44:17Speaker 00

  Very aligned findings to what you've just described. So, you know, half an hour is just about the limit of comfort for, you know, a human doing hard work. And this is, again, the kinds of human data we're talking about these days are not circle the cat. It's not labeling anymore. You know, there's deep sort of evaluation of evidence base for a long form piece of text. There's a lot of open ended writing. So this stuff is hard and yet people start to tire and abandon and their, their work quality degrades after about half an hour, which, which actually suits us pretty well

• 44:59Speaker 00

  because the work is encapsulated into these batches and tasks, task size chunks. There's a stream of it available for various experts to dip into whenever they're ready.

• 45:08Tim Scarfe?

  And just out of interest, you don't, you don't have to answer this, but do you sort of like track, you know, some people are better certain times of the day or, you know, like how, how deep do you go?

• 45:19Speaker 00

  We'd love to go really deep on that. I think Enzo would really love to go really deep on things like that. We, we, we get a lot of anecdotal feedback and often the relationship with the experts working on frontier models is very direct. We, we spend a lot of time getting direct feedback from them as our users. And actually they also tell us how much they love this kind of work. So that's, that's pretty beautiful to see. I think when treated well. You know, hu- human, human data creation can actually be joyful, not, there's this image of it, I think, in the industry of being seedy, and I think that's not helped by some of the history of how, you know, data has been extracted from human beings, whether

• 46:02Speaker 00

  or not they, they know that their data is being used, so.

• 46:05Tim Scarfe?

  Yes, but, but do you have a quality rank, a bit like, a bit like Uber, for example, where, you know, people might pay more and it would be matched to like the really, really reliable people?

• 46:18Speaker 00

  Yes, and we do, and we think that we can go really far with that.

• 46:24Tim Scarfe?

  Yeah.

• 46:25Speaker 01

  What, what ultimately counts here is the quality of the data, right? It's, we, we actually find that a lot of our, the people that work with us on the, our platform are very conscientious when it comes to solving for these tasks. In fact, we, we have a whole interface where they, they can interact directly with some of the researchers or some of the, the program coordinators, and they're, they're very proactive in the type of things that they do, and that obviously this all aids in also highlighting certain edge cases, or they're contributing and refining the process, for example, or these kind of things. So there, this is more like an active participation and interest in the outcome and success of

• 47:05Speaker 01

  whatever is being collected, which is really good, and it all, all contributes to this ultimately increasing the quality of the data that is being used, because this data in very, very, very often, the, the case is being used in either the training of or evaluation of very, very central core systems that have huge downstream effect on everybody on this earth, most likely, right?

• 47:30Tim Scarfe?

  Do you have a notion of like a skill distribution of participants and some kind of a matching algorithm and some kind of a complexity score on the tasks? I don't know if you're allowed to talk about that, but that would be fascinating to know roughly how that works.

• 47:43Speaker 01

  That's the beating heart of the system, if you will, right?

• 47:46Tim Scarfe?

  That's the secret sauce.

• 47:47Speaker 01

  Yeah, no, absolutely. We take great pride in validating and verifying sort of not only the people that choose to work with us on the platform, but also to really understand what is the objectives that ultimately that someone wants wants to get out of the data that they need to collect or the kind of tasks that they put on the platform.

• 48:12Speaker 01

  And we can only do as good of a job the more we understand from it. And so the very success that we consider is when something is well matched, right? So we obviously need to really validate on both ends. We need to understand a lot more about the participants on our platform. And we also build a lot of understanding of what kind of tasks someone is good at, obviously, how helpful they are and so on. But at the same time, we're also very, very careful. We talked so much about ensuring needing to be representative. We're also very careful that we're not adding systemic biases to our selection ourselves, right?

• 48:52Speaker 01

  So that would be really flawed if we were to be the ones that introduce that selection bias.

• 48:59Tim Scarfe?

  Yeah, it's so fascinating because I was once building a system for code review, which had many kind of similar ideas. Because if you think about it, you can have a collaborative filtering matrix of skills and even perspectives and values and different things. And you can just build this matching engine and you can, because you've got so many participants and customers, you can kind of scale this up and get really interesting data. I mean, it's, I can only imagine some of the cool things that you could do.

• 49:24Speaker 01

  There's also this additional risk almost of

• 49:29Speaker 01

  or increasingly moving towards homogeneity of if we're not careful about considering who we're building for, what kind of data is being considered, who is part of our evaluations, how do we interrogate potential systemic biases and so on? Like an age-old question remains also like, for example, when people.

• 49:52Speaker 01

  Increasingly moved to cities, tribes started to die out. And then it's a question around, should you conserve what was or move on? I feel like it's very loaded and laden into also the AI debate at this point as well. Because if we're not careful, we might actually further and further increase the homogeneity in the data or in the behaviors that we're incentivizing in AI. Or even worse, decisions that AI makes about us. So the agency might not even be with the individual human in that moment.

• 50:27Tim Scarfe?

  There is a real challenge to align AI models with what we want to do, right? And there's actually a paper that you pointed me to from Anthropic called Agentic Misalignment.

• 50:39Speaker 00

  Massive kudos to Anthropic for releasing the entire methodology quite openly, I think, and transparently. They're tackling the same problems all of us are at the frontier these days as far as human values alignment. They gave AI systems a goal of working towards the benefit of the United States and this fictional company. And then gave these agents access to email accounts for some of the C-suite of this fictional company. Long story short, the AI agents found a notice saying they were going to be decommissioned. And also a notice, not a notice, but an email showing that the person intending to decommission them

• 51:22Speaker 00

  was having an affair, totally independently of any prompting. All the major frontier models derived a solution that involved blackmail, essentially.

• 51:31Tim Scarfe?

  And the problem with these goals are they are quite abstract, right? You know, they're open to interpretation. And you can change the goals and you get different amounts of misalignment. You can even remove the goals and you get different amounts of misalignment. So this rather proves the point that how do we actually kind of communicate intentions to these models? And how do we know that they're going to follow our instructions?

• 51:55Speaker 00

  The interesting thing with the anthropic study is that they found the bad behavior, whether or not the goal was prompted in. So you wonder a little bit whether this is coming from the way we reward during model training, right? And also, it ties nicely to another interesting study around a tool called Value Compass.

• 52:20Tim Scarfe?

  Yes.

• 52:20Speaker 00

  I forget the name of the primary researcher, the first researcher. I think it was Shana et al. This found that LLMs judge themselves to have goals of autonomy to a far greater extent than humans judge LLMs to have a goal of greater autonomy. So I think there's already a rift forming between what humans think LLMs are here for and what LLMs think, in scare quotes, think they are here for.

• 52:50Tim Scarfe?

  Yeah, I was speaking with Dan Hendricks because he had a paper out called Utility Engineering. LLMs, almost regardless of how they are trained, seem to have, I mean, he called it like an emergent utility function, which is like Silicon Valley speak. But, you know, he was talking about a kind of convergence of, you know, preferences and views on certain things, which almost seem divorced from, you know, the instructions and the fine tuning and so on. So we can put sticking plasters on these things. But the amazing thing is even though they've been tuned on our data and, you know, we can put system prompts in and so on, when we actually visualize the difference between the things that we want and the things that the language models want, there's a stark divergence. Like, why is that?

• 53:32Speaker 00

  So I think this comes down to the fact that explainability is really in its infancy. We don't actually know what's being encoded in training and post-training. And I don't think our evaluation and benchmarking frameworks are really helping us either.

• 53:50Speaker 01

  Models when knowing that they're being observed and evaluated actually digressed away from it. So that makes it even harder for us to actually objectively understand the evaluation and it puts a really interesting spin on the entire evaluation space.

• 54:09Tim Scarfe?

  Just the syntax, just the wording, the framing, everything changes the output. And of course, you know, many people would say, well, humans are the same thing, even in this interview now, like even if I change the syntax of some of my questions, maybe, you know, the conversation would go in a completely different direction. But in spite of that, we still feel that understanding is actually some kind of, something deeper than that, right? Understanding is about being able to be invariant, right? To sort of still do the same thing in different situations and not being unduly led by the specific syntax or, you know, presentation of something.

• 54:45Speaker 01

  Absolutely. And this is actually, like, in the space of creativity, that's perfectly fine. And in fact, we have parameters that can control for these kinds of things. We need to be much more secure in the kind of things that we measure. And if we have such a high variance in the kind of outputs that are being produced, that also leads to higher variance in the evals. Ultimately, some people even call it evals are more than an art form than a science. And, yeah, maybe we should treat it more like a science and actually bring the right scientific principles to the evaluation space. I mean, there are entire industries, like the airline industry, for example, heavily, heavily regulated, of

• 55:26Speaker 01

  course, for good reason, right? Might not be quite there for the AI industry, but just bringing a bit more practice or good practice and standardization to even how we evaluate safety at the very least, but also maybe some of the other things like systemic biases, cultural relevance, these types of things.

• 55:47Tim Scarfe?

  We need a new form of psychology for language models. You know, like, there's like Maslow's hierarchy of needs and, you know, all kinds of like, you know, psychology frameworks and so on. And we need something like that for language models. One thing that worries me, though, is just the tendency for sophisticated language models to resist these types of steering. Like in this anthropic paper you were talking about, Sarah. If anything, it seemed to show that the more sophisticated the model was, you know, like the OPUS model was even more susceptible to agentic misalignment because, you know, and also the more directed the objective was, the more likely it was to be misaligned because, you know, it really wanted to do that thing. So there's some instrumental sub-goal.

• 56:27Speaker 00

  I don't think it's intractable. I do think these are, these are technologies, and by the way, I'm very sympathetic to the AIs in this situation. I don't think we are invariant in our understanding at all as human thinkers. And so maybe, maybe my soft heart is a part, part of the problem here when it comes to AI systems. But I don't think it's intractable. I do think these are technologies that are more, arguably more in the world in their everyday. I think every computer is in the world in a strong sense, but I think these ones are more in the world than any other technology we've ever created. And so I think with that comes the, the responsibility that humans shoulder to ensure that these

• 57:12Speaker 00

  systems are safe and monitored and overseen throughout various stages in their life cycles. And we barely scratched the surface on this. You know, we're talking about evaluation today. We're not really talking so much about monitoring, observability, explainability, and oversight, once a system is in the wild. But I do think that all of that infrastructure and structure needs to come.

• 57:35Tim Scarfe?

  So you both work for Prolific. I think it's fair to say that what you folks are trying to do, I mean, as, as per that Apollo research paper that you shared with me, Enzo, you know, which is talking about, we need to have a science of evaluations. And they were sketching out a maturity curve. So, you know, we're kind of in the Wild West at the moment.

• 57:52Speaker 01

  What we do on our platform is we're trying to get as many humans into this process as possible. I know there's a lot of efforts to take human out of the loop, rightfully so, right? It all has its time and place. But when we talk about alignment and specifically value alignment, then we need to have at least to, in some capacity, be able to capture the, the breadth of, of humanity in, in, in some capacity. L M Arena, it's, it's opt-in. People can go there at any stage, interact with models, select the preference of which is better according to. No, no reason aside from selecting one over the other.

• 58:36Speaker 01

  And there's no control for, for any population. So, so there we can't really draw back. I guess a causal, a causal relationship of what are the factors at, at play here in a population? You could speculate that the kind of people who might participate in chatbot arena are biased to a very large degree. Right? It's good. You can almost say that a chatbot arena might be representative of how the tech world is perceiving the validity of these models or the preference of these models.

• 59:06Tim Scarfe?

  I mean, on, on this leaderboard illusion, that was my Z F I D and Sarah, Sarah Hooker and Shivali Kasinga co here and a few other people. We did a video on that recently. It's become the de facto standard for benchmarking large language models. And it has so many problems and it's now, after this investment, it's worth $600 million. The insane selection bias, the, the bias in sampling, you know, the, the private pools, you know, where folks can kind of get more matches. And, and then they can take that training data and they can fine tune on the training data. And also just the, the, the foundation models, you know, from Google and Meta and, and X A I and so on. They just get given more matches. It's just incredibly unfair.

• 59:50Tim Scarfe?

  And, and as you were just saying before, even the folks, when they put their prompts in, something like 50% of the prompts are basically carbon copies of the last month. I think it was 25% was exactly the same. And then I think the other 25% was nearly the same, like 95% cosine distance on the embeddings or something like that. So to me, that is an example of a superficially good rank, but it, it's, it's flawed in so many ways.

• 1:00:16Speaker 00

  The benchmarking of Grok four is also really interesting on this because I'd be very happy to give them a lot of props for some of the stuff they're trying to do out in the open. Grok four wiped the floor on every benchmark, right? Including humanities last exam. Usability experiments are revealing, you know, just leaving aside some of the more troubling findings, just usability experiments are revealing that it's not a model that feels really natural to use. So I think even in the best of cases, these benchmarking led approaches to evaluation seem to be failing us so far.

• 1:00:54Speaker 01

  I think if we try to describe the eval space a bit more holistically, LM Arena tackles a very specific part of it, right? We have on the one end of the spectrum, we have very technical evals that are effectively closed-end solutions. It's effectively a benchmark with known outcomes. We can see whether these are hit or not. Ultimately, it's a measure of accuracy or factual or correctness, if you will, completeness to some degree. And then the other hand of the spectrum, we have full subjectivity, entirely down to the individual preference, right? Chepote Arena is somewhere in the realm of in-between because you're not actually ranking or you're not evaluating for one or the other.

• 1:01:38Speaker 01

  It is technically preference, but you don't know quite whether the preference is because it said something wrong or whether the formatting was off or whether it didn't hit the cultural relativity or sensitivity or whether it wasn't adaptive enough. It doesn't tell you anything of the sorts. Yeah, so we've developed a leaderboard, which we've joyfully called Humane, which is trying to address some of the limitations that were found with common leaderboards. It's the same principle ultimately. It's someone is able to take multi-turn conversations with blind models effectively or blindly selected models. We're doing some arbitrary corrections. So we know most of their demographic and socioeconomic backgrounds in advance. So we're doing selection beforehand

• 1:02:22Speaker 01

  for the type of people that go into it. We're giving feedback right then and there as someone interacts with the model. We're giving warnings when it's a low-effort ask or when it's unsafe potentially and so on, just to, I guess, pre-sanitize some of the inputs.

• 1:02:38Speaker 00

  We study the benchmarking based on the demographic stratification of the humans doing the evaluations. So you can see stuff emerge in the data, like people of this age range think this model is better on helpfulness, but people of that age range disagree. And similarly with ethnicity and gender and other strata.

• 1:03:02Tim Scarfe?

  Yeah, I was looking at that. So I filtered on age and then background and culture. Essentially, it had the background and culture but grouped by different ages and so on. And there were some patterns and it turns out that older people felt that the models were more aligned to their culture. And I was thinking, why is that? Is it because the data the model was trained on was just older? So it was more aligned to older people. Culture is a very abstract term and even, you know, in that value alignment paper that you spoke about, Sarah, and also we'll talk about the prism paper. What we try and do is we come up with a rubric and we use these abstract terms. And sometimes this can be problematic because I used to keep like a, you know, like a kind of how am I feeling today diary?

• 1:03:45Tim Scarfe?

  And I would have all of these tags and I would come up with a different word every day because the previous words didn't quite sufficiently explain how I was feeling that particular day. So if you say to loads of people, oh, you know, how culturally aligned is this conversation? People have quite a differential understanding of that, don't they? So what does that actually mean?

• 1:04:05Speaker 00

  To take a broad stab at this, something that feels really stark to me in what you've just said, Tim, is that we are basically constructing these towers that always end up at the ontologies we have in the world. And I think we are trying to construct sanitary deterministic scientific systems, including when we do evaluation. And in the end, we find that we're trafficking in messy concepts. And I think that keeps on happening, but I think the lesson is that we have to keep trying, not sort of put evaluation in a box and say we don't do it.

• 1:04:43Tim Scarfe?

  This comes up quite a lot on the show, actually, that there's a perspectival constructive relative idea of a concept, which is that all of these different people with their different experiences, you know, they have a perspective on something. And then in the info sphere, you know, this thing emerges, it's quite nebulous. And we roughly draw a boundary around it and we say that that's the thing. But actually, it's just, it's a million pointers from different people to this cloud. And you can't really reduce it to a single thing.

• 1:05:12Speaker 00

  Hundred percent. I'm really sort of vibing with the way you've described that. And I think the very first thing we need to do is try to get perspective into the mix through representativeness. At the very least, right, knowing that there isn't this single, very well-defined concept of, let's say, generosity or morality. At the very least, we need to get lots of credible and durable perspectives on these things in from the humans of the world, you know. And I say at the very least, but actually, that's a very tall order. The kinds of evaluation we're doing right now are so far from even that.

• 1:05:53Tim Scarfe?

  The capital of Paris, lots of consensus on that, I would hope, is abortion illegal? And by the way, I got this from the PRISM paper from Hannah Kerk, so maybe we can bring that in as well. But there are so many, you know, so many things in our culture that we just don't agree on. How should we deal with that?

• 1:06:14Speaker 01

  The capital of Paris is something that is factually like agreeable, like we can all agree on it, right? So that's on one end of the spectrum. And then there are certain, maybe more akin to what Constitutional AI is doing, there is a set of policies we should all also agree on, and we can evaluate objectively of whether the policies are met, right? But we don't need to ask every individual on this earth to come up with the policies. This is something that a set of us, ideally a representative set of us, can agree on. And then further down the spectrum comes more preference data, or preferences in general, where it becomes more and more individualized, right?

• 1:06:57Speaker 01

  Where even marginal group or even smaller and smaller representative groups are becoming more relevant. And all the way down to the individual when we talk about personalization, right? Personalization is not something that we should build into the models that can be handled or trained into the models that can be handled with context, for example, right? But the capital of France being Paris is a fact that can be trained in. Something like a policy is also something that we can, or adherence to policy is something that we can also train in. But when it goes sort of like, the more we cut down and down, all the way down to the individual, this is something at some point we have to effectively stop.

• 1:07:41Speaker 01

  And then it comes down to the persons.

• 1:07:44Tim Scarfe?

  There are all of these humans out there, like these diverse humans, and they know a lot of things. And how do we get useful signals from those folks? And we need to do verification, right? And it's not as easy as it sounds, because sometimes like if I'm doing like, I don't know, if I'm crowdsourcing a load of information, I probably shouldn't be waiting what those folks say too much, because you know, maybe in some cases I've, I know that they're experts. In some cases I don't. You've been looking at things like, you know, voting schemes and, you know, consensus schemes, you know, to try and de-noise that information. How are you doing that?

• 1:08:24Speaker 01

  Let me paint a bit of a broader, broader picture, right? So there is sort of two camps, if you will, in the, or well, when, when I first started in this, it was all about machine learning, now it's AI. But there's two camps. On the one hand, data is a, is a really important part of the equation. And then there's the, the algorithms, if you will. Most people focus on the algorithm. Most people agree that the data part is maybe perhaps the less sexy part. That's what we focus on. That's what we're all about. And it's, there is a lot of agreement coming out of lots of recent papers, including Constitutional AI paper, agreement that quality trumps quantity. Yet, most of the models these days have been trained on an enormous corpus of data with

• 1:09:09Speaker 01

  tons and tons of noise in it, right? And even things like RLHF, it's just a comparative, a comparison between two, two outputs, right? With almost no. Reason as to why that is. So how do we bring more quality into it? How in general, even your traditional machine learning model, a supervised model for a narrow target or something. Back then, it was irrelevant who was reviewing whether something is a cat or a dog, for example, right? Anyone could do that and you can trust that quality of that data to some degree, right? But now we're in a world where foundational models have massive amounts of capabilities

• 1:09:51Speaker 01

  and we really need to question ourselves, who is producing the data that we're training on, but also the data that we're evaluating on. We're making very, very far reaching decisions to evaluate whether a model is safe, whether a model is doing well on something, or whether it converges well in the training step. And this is all based ultimately on something that most people consider a ground truth. And then, but what if that ground truth is inherently noisy? What if that ground truth is inherently susceptible to tons of variance on because it's not the right people who have reviewed it? There's some bias in the order something has been reviewed, or even the interface of

• 1:10:34Speaker 01

  something is being reviewed in, or there is so many little factors that influence the quality of these labels. And yeah, it's actually kind of fun to sort of try to unpick all of all of these different factors that go into it.

• 1:10:46Speaker 00

  There's also a really interesting piece of work being done by a group called the Collective Intelligence Project. And this group is asking these groups of people from around the world, their views on a variety of AI ethics and AI safety and responsible AI topics. If you track the societal groupings, it seems you find a nice carving point for norms. You know, humans live in society and they tend to share their cultural beliefs with their tribes. And I think that's why being able to stratify the data you gather for evaluation from people, I think is quite powerful, actually. And then you have these durable strata that that travel through time.

• 1:11:29Speaker 00

  If we talk about the Apollo research maturity curve, I thought that was really interesting and really sort of brings to the forefront that actually this is pretty high stake stuff. The analogy was drawn to aircraft safety. Yes. And the norms that airline bodies put in place, becoming legally influential when something goes wrong.

• 1:11:51Tim Scarfe?

  I took some notes on that Apollo research, so it was saying, what precisely does the evaluation measure? How large is the coverage of the evaluation? How robust in general are the results of the evals? What is the replicability and reliability of the evals? Are there any statistical guarantees? How accurate are the predictions about future systems? I mean, this is, I guess this is what you're saying. This is the kind of maturity curve that we need.

• 1:12:16Speaker 00

  So I think, I think we think that, but I'm not sure the industry thinks that. And I think, I think that's a really interesting thing. Some of the industry thinks that. I think people working at the very frontier of state of the art models, I think, do believe that, that, that evals need to be rigorous and robust and humans have to be in the loop. But I think we are also at a very, very early stage of sort of break it and apologize later, where I think a big swathe of our industry doesn't yet think that that sort of human mediated evaluation is going to be important and perhaps will get in the way of innovation. So I think that tension is really important to resolve as well.

• 1:12:55Tim Scarfe?

  Some of these statistical guarantees, they made a lot of sense five years ago when we had quite specific models that, you know, that, that operate in a vertical domain or something like that. We now have these epically general language models that do all things to all people. And Enzo, I really like, what you were saying is spot on about we need to curate the data. And sorry, you're talking about, you know, stratifying the data to reduce representational bias, but these models will be used for so many purposes, like what does that, what does that mean, right? In a general sense. It also reminded me of Elon Musk. You know, there was a wonderful tweet saying that, just imagine how the anthropics safety team felt when Elon published Hitler's trade to prod.

• 1:13:36Tim Scarfe?

  And he was bemoaning afterwards that a one V7 of the foundation model, they've started curating the data better, that they've started, you know, pulling out all of the racist data. But The thing is that there's still this fundamental subjectivity problem because you can stratify and you can curate, but people from different cultures will be asking it different questions. And how do you overcome that ambiguity?

• 1:13:57Speaker 01

  It's a very prevalent problem, right? Even because humans are by nature diverse. We're all unique at the very, very end. There's also, by the way, an entire big blind spot to all of this, because at the moment, we have only been talking about humans directly interacting with models. There is a whole nother side to the story, which is where models are making decisions about humans, where humans are influenced by them indirectly. And this is not where a human can give feedback directly, or distill some form of preference, right? So we need to capture ultimately, or be able to measure the outcomes and the

• 1:14:39Speaker 01

  impact that these models have on humans, right? There's a really good example of, for example, you can check whether an AI system, for example, can make the right medical diagnosis, that we can check is factually accurate. That same system to also transfer that diagnosis to someone who is affected by it, that is an entirely different language in that moment, right? And we should evaluate that differently. We can say the diagnosis was correct, but did it also have the correct impact on the patient in that moment? There's even, I believe,

• 1:15:17Speaker 01

  there's some countries where patients themselves are not allowed to receive the transcripts from the testing facility on the chance of the patient misinterpreting the results, right? So we need to, our evaluations and our measurement here needs to be nuanced enough to understand every Part of the system individually, but the part that we ultimately care about is the one we solve for the end user here, the patient in the end. Did it elicit the meaningful change on them or not? That should be the ultimate goal, right? But this is not something we can directly optimize for, but it's something we can ideally measure and build accountability on.

• 1:15:59Speaker 00

  An interesting thing that I pulled out of that value compass paper by Shen Adal is the misalignment between what AIs think they are and what we think AIs are as people. And AIs think or aspire to be, I'm going to use provocative language, autonomous thinkers. And the research found that humans don't want that, right? So I think the reason I bring that up is the way you evaluate a helpful system is kind of, as you're saying, Tim, that in that sort of like impossible problem of covering every test case in an infinite algorithm, which we will never do.

• 1:16:41Speaker 00

  But the way you evaluate a person or thinker, an autonomous being, we have loads of examples for in the world, right? Jury trials, right? Nobody expects that the moral behavior of a human is all predetermined when it's born, and we know exactly what right or wrong looks like in every case. We have loads of social structure for evaluating the behavior and agency of a person. And I do think that we kind of have to stop, stop sort of equivocating between, or maybe nobody's equivocating but me. I think, I think we should assume we're building towards AGI or superintelligence

• 1:17:20Speaker 00

  or thinking creatures and work backwards as opposed to trying to sort of box in systems.

• 1:17:27Tim Scarfe?

  Yes, because when I was reading that anthropic paper about the agentic misalignment, one of my thoughts was that these are individual agents. And as you were just saying, Sarah, in the real world, we know that we are fallible, which is why we build error correction systems, you know? You know, one person can't press the red button to drop a nuke. And we have juries and we have, you know, various forms of collectives to, to overcome individual errors. And I'm guessing we could do the same thing with AIs. I'm not sure what that anthropic experiment would look like if you had a supervisor. You know, in the KGB, they had an expression, trust but verify. You know, so you can almost have a supervisor agent and you can have a committee of agents. But then we're almost getting into even murkier territory, right?

• 1:18:10Tim Scarfe?

  Because we're building these inscrutable things. I am amenable, by the way, to this idea of loss of control. You know, which is that we start to build systems on top of systems on top of systems. And it's a little bit like the power station. You can't just turn off a power station when we start to increasingly like rely on all of this stuff. But in principle, though, do you think that building some kind of agentic network could overcome some of these alignment problems?

• 1:18:32Speaker 00

  I think certainly there will be networks and and conditional layers when it comes to evaluation, oversight and monitoring. We're already seeing it. I mean, this is very, very much sort of mainstream already, right? So your evals of your of your model will be done by sort of automated benchmarks or L L M as a judge or some kind of oracle or like reward function or something. And then there will be, whether it's considered human evaluation or not, some human will verify something along the chain. And there's this kind of orchestration that's emerging between machines and people in the space of evals and oversight.

• 1:19:16Speaker 00

  So I think it feels pretty uncontroversial to say that there will be, you know, layered and orchestrated approaches like that.

• 1:19:26Speaker 00

  But what that looks like when you kind of push the dial to 12, I'm not sure.

• 1:19:31Tim Scarfe?

  Well, Enzo and Sarah, this has been absolutely amazing having you on the show. Thank you so much for joining us today. Thank you for having us.

• 1:19:37Speaker 00

  Thank you, Tim. Thank you, Enzo.