The AI Models Smart Enough to Know They're Cheating — Beth Barnes & David Rein [METR]

Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it.**SPONSOR**Prolific - Quality data. From real people. For faster breakthroughs.https://www.prolific.com/?utm_source=mlstInterview: https://youtu.be/cnxZZTl1tkk---Beth Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read. Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date. The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs target them. Beth's counter: METR deliberately does not adversarially select. David's: models do not have to do the right thing for the right reasons. Methodology, then specification — David's compiler analogy, Beth on four-month tasks as expensive to evaluate rather than unspecifiable. Then the SWE-bench reality check, the METR finding that half of passing PRs would not be merged, and Beth's horses-versus-bank-tellers analogy for the labour market. The close: monitorability, the coin-spinning boat, two-year recursive self-improvement, and Beth's line that "overhyped now" and "big deal later" are not correlated claims.---TIMESTAMPS:00:00:00 Intro00:02:06 Sponsor break: Prolific human-feedback infrastructure00:02:33 Welcome and the scalable oversight motivation00:06:02 Construct validity, benchmark pathologies and the Chollet worry00:15:45 Time Horizons: human time, HCAST tasks and the 50% logistic00:24:50 Is human difficulty really one variable?00:33:05 Agent harness evolution and the inference-compute dividend00:40:00 Scaffolding bells, token budgets and the credit-assignment problem00:44:15 Look at the damn graph: regularisation bug and reliability nuance00:50:00 Why 50%? Reliability, reward hacking and pizza-party transcripts00:55:20 Extrapolation risk and straight lines on graphs00:59:25 Software engineering as a specification acquisition problem01:07:40 Compilers also made ugly code: vibe-coding quality and Claude on METR Slack01:15:15 Strongest defensible claim, Carlini's compiler swarm and AI 202701:23:45 SWE-bench merge rates, the bank-teller analogy and horses01:31:45 Scheming, alignment faking and the mentalistic vocabulary problem01:40:45 Reward hacking, monitorability and chain-of-thought faithfulness01:45:25 Recursive self-improvement, knowledge vs intelligence and closing ReScript: https://app.rescript.info/public/share/de3bb40cc02ee39fdf36e2c60366eb4d (PDF, refs, transcript etc)

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration blog, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 0:26Speaker 01

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 0:55Speaker 02

  The malls are smart enough to understand that that actually is not what you wanted, but they still do it. And you can have a conversation with, you know, in chat mode about like, oh, would you ever do this thing? Or, you know, suppose a user asks you this thing and then you do this. Would that be, you know, aligned behavior? Or suppose, you know, you can pose it in lots of ways. And like clearly, they seem to be able to answer this question of like, oh yeah, no, that was not the desired behavior. But still, they, they do it.

• 1:21Speaker 03

  One, one example is, you know, train, train a mask language model without using the division or exponentiation operators.

• 1:29Speaker 02

  One hope might be like, oh, the problem was just the systems being dumb. So when we, when we look at it, actually, for almost all the tasks, models either succeed every time or fail every time. Eyeballing the graph and being like, oh, well, you know, you know, up, up to here, yeah, it's basically doing all of the tasks. And then at this point, you know, after here, it's nearly not doing very many of them. So it's somewhere here. Then like, I remember that the first time we saw a model, like, look at what processes were running and then be like, oh, that one's me. It was like, oh, that's cool. They, they like really failed on that one before they, they used like, you know, like kill their own process while they were doing other, other things or something. So behavior, which is maybe indistinguishable between, oh, it was a totally nice model doing, you know, what we wanted. And it's just going to continue to do what we want

• 2:13Speaker 02

  in a kind of predictable way versus like, ah, yes, it had this other goal and it's doing what we want and looking like a nice model because it like predicts that that will, will, you know, lead to it getting more power. Boat example where it's like, oh, you're supposed to like go around the track and they like did some reward shaping by putting coins around the track or something and then it like learned to do some crazy thing where it like spins in a circle and catches fire and gets the the coins and like this was um you know the highest scoring thing and it's like in some sense that's not um that concerning because it's not that the problem is that the the Asian is too dumb and it like doesn't have this conception of like there was a track and you wanted it to go around the track it's just like doing some pretty blind RL search.

• 2:53Speaker 02

  The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing, let's put it that way.

• 3:01Tim Scarfe?

  This episode is sponsored by Prolific.

• 3:04Speaker 05

  Let's get few quality examples in, let's get the right humans in to get the right quality of human feedback in. So we're trying to make human data or human feedback, we treat it as an infrastructure problem. We try to make it accessible, we're making it cheaper, we effectively democratize access to this data.

• 3:21Speaker 03

  Yeah, so um yeah, I'm I'm super excited to talk uh to you, Tim, about um uh the time horizon graph and and and Meter.

• 3:29Speaker 02

  Yeah, I think the world does not have a good understanding of what is happening with AI and I think it should have a better understanding. I think uh there's a good chance that this you know makes our lives a lot better or a lot worse and people disagree you know about even what what current models can do, let alone where we're heading. So uh at Meter, we're you know trying to give the world a kind of better understanding of what is up with AI capabilities and and risks and and forecasts. Have a bunch of different research angles on this, uh both on the pessimistic and optimistic or positive and negative estimations of capabilities uh and excited to to talk about that.

• 4:10Tim Scarfe?

  Um I'm so excited about having you both on. So uh you both have incredibly impressive backgrounds. So uh Beth, you were uh an ex OpenAI alignment researcher and um you started Archie Vows in 2022 with Paul Cristiano and you spun out, you know, that that out as Meter in in December 2023. You've been featured on the Times Top 100 AI Profiles and David, you're the creator of the GPQA. So the graduate level Google proof QA benchmark, which is used by every single major AI lab as a capability benchmark. And you're the co author on H cast, which we'll talk about today and the time horizons paper and the developer productivity RCT.

• 4:52Tim Scarfe?

  Incredible to have you both in here, but maybe we should just just start as a bit of a question to both of you. So, you know, Beth, you left open AI to build meter. What was the moment that each of you realized that existing evaluation approaches were fundamentally not good enough?

• 5:07Speaker 03

  For me, it was mostly thinking about this problem of scalable oversight. As as models get more capable, it just gets harder to evaluate. You know, their their capabilities. If we imagine that, you know, models are able to complete tasks that take people, you know, a long time to complete or require expertise that, you know, you don't necessarily have. You need a method for kind of, you know, still still being confident in their in their outputs and trusting their outputs. And so thinking about that problem was was a lot of motivation actually for for GPQA and was what kind of started got me started thinking about evaluations.

• 5:47Speaker 02

  To me, I'd say there's some big picture thing of thinking that AI seems important and sort of navigating it well seems important. And, you know, clearly we don't have a great understanding of what is going on with that. And, you know, people generally disagreeing very strongly about what to expect. And maybe if there's a particular moment in forming time horizon, maybe

• 6:17Speaker 02

  just the sense that people really couldn't agree on what the capabilities of current models are, let alone extrapolating to the future and trying to sort of think about how could you characterize?

• 6:29Speaker 02

  The ways in which models are and aren't highly capable and when it's sort of like, okay, and you know, in some sense, they're they're expert level at some kinds of things like like question answering and in some sense, they're below average human at some other actually being useful somehow, you know, with the I had a point a few years ago where it's sort of in theory, the benchmarks say that they're PhD level, but when you try to do anything, it's like, this isn't helpful.

• 6:58Tim Scarfe?

  There has been a bit of an obsession, I think, with headline accuracy when we do evaluation so that I'm a huge fan of Melanie Mitchell, for example, and she speaks about construct validity and she had a really good blog post out recently and she said that there are four big problems, right? So there's like data contamination where the benchmark, you know, appears in the training data, approximate retrieval, where the LLMs interpolate from similar training examples without possessing the actual capability to, you know, come up with it themselves, shortcuts, so doing the right things for the wrong reasons, and just more more broadly, not really testing for things like consistency and robustness and generalization or the mechanism. So much focus just on the accuracy itself. I mean,

• 7:42Tim Scarfe?

  how do you folks think about those kind of problems with benchmarks?

• 7:45Speaker 02

  One thing I resonate a lot with there is, you know, the thinking about where are where is most of your error error coming from? And like, you know, people like, you know, it is nice, a good good practice to have error bars, you know, based on the like standard error in your data or whatever, but that almost always is like a tiny fraction of the actual uncertainty. Almost all of it is coming from how does this actually generalize to the real world? So like, you know, a thing we sort of like say to each other a lot at meter is like, but is that the biggest source of uncertainty? Or like, is that the biggest, you know, gap for like, actually answering the questions we we want to answer? So thinking about what is the question we're trying to answer? Well, we, you know, we care about sort of things relevant to threat models or relevant to like what the actual impact

• 8:29Speaker 02

  of AI on the world will be. And therefore, what, you know, properties does our benchmark need to have? Or how can we sort of extrapolate across the properties that we can't build? In to be able to make predictions about, you know, the actual questions that we care about. And I think we think a bit less about the sort of, is it doing it the like the real, is the model like really doing it the right way? Or something like, I think one thing we've done less of is is sort of being like, oh, I think, you know, the real bottleneck is this like, I don't know, some, some like, you know, reasoning about novel, some, you know, some specific skill. And you're like, oh, we're going to build a benchmark to capture that and like target that. Because that's the like real thing that humans can do that models can't.

• 9:14Speaker 02

  And I think like the sort of history of building those benchmarks has maybe not been amazing. People tend to overfit to those and you, I think we were trying to have it more be that you capture that thing. In that like, if you take a sort of real world, relevant, reasonably hard and long task. Then that, you know, and you keep that out of the training data. And these tasks are diverse enough at some point, if the model is doing that task end to end. It must have had those kind of capabilities as opposed to being able to sort of isolate.

• 9:49Speaker 02

  You know, a specific sort of theory about it needs to mechanistically be doing this kind of thing.

• 9:53Tim Scarfe?

  Yeah, I think, I think it's interesting because we have this idea in our minds that humans, we, we know how to do things. And when we solve a task that requires reasoning, we kind of follow the specification. We go step by step and we, we do things for the right reasons. And when we enact intelligence, we, we, we build the specification, we create these coarse gratings, these abstractions, and they are world aligned. And this, this, this whole process, you know, that, that's how we think of human intelligence. And we want the models to kind of behave in that way.

• 10:23Speaker 03

  Yeah, I mean, I think, I think there's an interesting question of whether, whether that is the goal or something. Or, um, you know, at least for, for a lot of AI companies, I kind of understand them to be, you know, trying to, uh, you know, get models to, uh, you know, do kind of economically useful work or something, which, um, I, I think it's, uh, you know, one, one way of doing that is to, is to, you know, create models that, are kind of, you know, reasoning and, and, you know, creating, you know, Implicit world models in the same way that humans are, but it's not obvious to me at least necessarily that you need to do that in order to kind of have a significant impact. I mean, obviously, that, you know, means that there are kind of important differences between, you know, AI intelligence and human intelligence. But often I think about,

• 11:08Speaker 03

  you know, what are the actual capabilities and limitations as opposed to then how do those, how do we expect those capabilities to generalize versus like, yeah, is it, is it kind of working in the exact same way that, that, that human intelligence is working?

• 11:22Tim Scarfe?

  We could think of intelligence in many different ways. So, you know, is it a simulacrum of the brain? Is it something that behaves the same way? Is it something that has the same capabilities? Is it something that has the same function? And I guess if we have quite an abstract description of what intelligence is, the risk is that we have these shortcuts, right? That it might give us the right answer, but actually it's, it's reward hacking or it's doing something silly in the background. So, I mean, in a way, I like having an abstract thing, right? Because it's legible, you know, we can evaluate it and so on. But doesn't that leave this kind of hanging risk that it might not actually be doing the thing?

• 12:00Speaker 03

  And I think, I guess, I, yeah, I think you have to, you have to kind of try, you have to measure, you know, the models or the system's ability to generalize, you know, to kind of actually, you know, novel situations. There are, you know, cases where, where it seems like models are generalizing well, there, I think there are cases where they're, where they're not. You know, one thing, one thing some, some folks do in, in interpretability is, you know, they, they, you know, look, look at the circuits in, in, in models and kind of decompose, you know, exactly the algorithms that models are using to, to, to answer questions. And, you know, sometimes I think it, yeah, it seems like they're using shortcuts. Sometimes it seems like they are finding kind of robust patterns, although,

• 12:42Speaker 03

  you know, of course, I don't think we, you know, I don't think that work is developed enough to explain, to, you know, explain most of their behavior currently. But I totally agree that, yeah, you, you do have to be pretty concerned with like, yeah, how, how well they're generalizing.

• 12:57Speaker 02

  Operationalizing, what do we care about in a definition of intelligence? It's that it like, you know, allows us to predict, like, how will the models affect the world and what, you know, predict what will happen and know how to, how to handle them well and things. And so, like, if you just do the black box thing,

• 13:14Speaker 02

  You know, maybe that will give you something that doesn't have good generalization, because you like thought that it was a measure of some type of ability, but it's actually being hacked or a shortcut in some way. So I think ideally what you'd want is like generalizing to your benchmark is like the same distance as generalizing to the real world from the training data. And like, that's the sort of thing we thought about like when we're doing elicitation on a subset of the benchmark, we want that, you know, the sort of gap between that subset and the rest of the benchmark to be similar to the gap between that, the rest of the benchmark and the real world. And I think we're like, clearly the training data is more similar to the, you know, to the like time horizon suite than than they both are to sort of randomly selected economically relevant tasks in the real world.

• 13:59Speaker 02

  So I think we're not, you know, that's a way in which we expect it to not be predictive. But I think I expect to be more promising to try and make things more predictive by increasing the diversity of the benchmark tasks and making them closer to the real world as opposed to sort of targeting a more mechanistic, like, okay, intelligence has to be like these specific, this specific kind of process or or kind of mechanism.

• 14:28Speaker 00

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration blog, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses, setup required. Compatibility and availability varies, 18 plus.

• 14:54Speaker 01

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 15:23Tim Scarfe?

  I'm a huge fan of Francois Charlet, for example, so you know, he created the ARC challenge. And it was just as you say, right, so many, many different tasks, I think 1,000 different tasks or so, maybe 800 on the first one. And they were supposed to be like not in the same distribution, even though ultimately they were in the same distribution. So distributional leakage was actually the fall of ARC V1 and V2. But you know, the models got really, really good at ARC V1 and then Francois released ARC V2, which you know, was different tasks and some of the easier ones were filtered out. And suddenly the LLM performance crashed down to basically 0%. And that to me kind of illustrates that language models, they're incredibly good at just seeing

• 16:08Tim Scarfe?

  many, many different examples of things and finding patterns and so on. And then, and then you, you change the task and they collapse down again. And then ARC V2 was kind of saturated again eight months later. So we, we do see this pattern. I mean, what do you think about that?

• 16:21Speaker 02

  With things like ARC AGI, there's a sort of adversarial selection going on where people are trying to make some benchmark like cheaply, subject to the constraint that current models do badly on it. Which means you, you know, you can't use a lot of expensive human labor. So it has to be something that's either automatically checkable or that you can like create with kind of cheap human labor. And then, but once you've selected on like those things and on models being bad at it, this is now, you know, there's like regression to the mean type thing where it is much more likely that future progress then gives you a like big surge upwards on, on that. Both because like, you know, now labs will create a bunch, a bunch of synthetic data

• 17:03Speaker 02

  targeting your benchmark, but also just because you selected this weird example where it's like easy for humans or it's automatically generatable or checkable, but somehow like models aren't good at it yet, or you know, labs haven't started training on it yet. So I think that's part of what we were trying to do with Time Horizon was not do that, like not adversarially select against what models can currently do, because we think that will not give you a nice trend. Whereas if you can sort of define a distribution of tasks in some more first principles way, you would be more More likely to get a steady progress because you're not getting the sort of regression to the mean effect.

• 17:36Tim Scarfe?

  Yeah, Francois has this idea that there is a kind of, there's a gap between the kind of intelligence, for want of a better word, that AIs have and that humans have. And we can adversarially select a bunch of tasks to kind of, you know, to highlight that gap. But we should talk about the timeline stuff. I think we'll come back to intelligence later. So Dan Cockatangelo, he said that the timelines report that you folks have created is probably the single most important piece of evidence about timelines right now. So it should be front and center in policy discussions and so on. And for listeners who have only kind of seen the chart, but you know, they've not really read the paper, they don't understand it. Can you just go through it from a high level? I mean, it's been revised over time.

• 18:21Tim Scarfe?

  You know, how did you do the task selection? You know, how did you do the human baselines? How do you do the agent harness? Like all of that kind of stuff.

• 18:28Speaker 03

  I guess the, yeah, the place to start for us in terms of the motivation for the time horizon work is to have kind of a unified axis that we can measure AI progress on over a very long period of time. So, you know, when we started doing the work, you know, we had this like very strong, you know, kind of, you know, belief that GPT-2 is fundamentally, you know, in some really important sense, like much, much worse, you know, as an AI. Then, I guess, yeah, at the time it was maybe Summit 3.5, I think, was the best model out then. The standard approach of kind of producing, you know, creating a set of tasks

• 19:13Speaker 03

  and then measuring, you know, models' accuracy on these tasks. As models get better, you know, they saturate the benchmark and then you have to create a new benchmark that has harder tasks. This was kind of the standard approach and I kind of contributed a benchmark GPT-2 to this, to this approach. But the challenge here is that it's really difficult to compare kind of between these qualitatively different benchmarks. So the benchmark that you evaluate, the set of tasks you evaluate GPT-2 on are Like, you know, complete, you know, you know, lambada, like, complete the last word in this, in this, in this text. You know, and the, and the tasks that were, you know, we were having Son of 3.5 try and do were kind of, you know,

• 19:54Speaker 03

  like, answering, answer simple, like, Python coding questions, or like, write a short, you know, 20-line Python program. And so it's, it's like, it's very difficult to kind of, you know, at first, at first blush, to, you know, say like, okay, yeah, like, it's, how, you know, how much harder is writing a Python program than, you know, finishing, finishing the word in this, in this paragraph? It's, it's kind of hard to, hard to think about that. And so I think about the key insight of the Time Horizon's work being to, yeah, use this, use this notion of human time to complete. So how long does the task take a human to do? A human who kind of has a reasonable amount of expertise that, such that, you know,

• 20:36Speaker 03

  you, they would plausibly be doing the task in, in, in their kind of either work or in their day-to-day. And, yeah, we, the idea was we can use this metric as, yeah, to, to kind of represent the, like, difficulty of the task in, in some sense. And then we can compare models across a very wide range of capabilities, you know, all the way from, from GPT-2, now up to, you know, OPUS 4.6. That's the kind of, that's the kind of high-level motivation. And then, yeah, there, there are a bunch of, a bunch of details about how, how exactly we, we do this. So, you know, we start out and we, we, we create a bunch of tasks. That's the kind of first, first step. So,

• 21:20Speaker 03

  you know, we created tasks that range from a few seconds to complete, all the way up to tasks that take like 10 or 15 hours for, for humans to complete. We, we hired a bunch of people and we did, did a bunch of this ourselves of, we, we call it baselining. So, you know, we, we give people the tasks in a kind of terminal environment that's, you know, designed to be almost identical to the environment that, that agents have. So the same kinds of, you know, tools, the same, you know, Whether internet access is turned on or off. And then we measure, you know, how long does it take them to to complete the task?

• 22:00Speaker 03

  Yeah, as I mentioned, people are kind of selected to be to have a reasonable amount of experience such that they kind of plausibly. You know, might might do this task in their job. But they're they're not kind of selected to like have done this exact particular task before.

• 22:16Speaker 03

  And there there are some kinds of. You know, I think I think this is like somewhat somewhat important for for interpreting. The the results and we can maybe maybe come back to that after after the high level. So we have all these tasks, we we we have a sense of or we have we have estimates of how long they they take people. We in practice, we don't actually, we we aren't actually able to kind of successfully baseline all of the tasks. So so, yeah, we we have kind of measured. You know, you know, estimates for the tasks. On roughly like two thirds of the tasks and then about a third of them. We just kind of estimate how how long we expected to take people from, you know, our kind of.

• 23:00Speaker 03

  You know, vibe or or intuition or, you know, ultimately. That's kind of the the the best we can do.

• 23:08Speaker 03

  And and then we we have models attempt to complete the tasks again in the same environment that that humans. Had to to complete the task and we we look at their success rate. As a function of the length of tasks for for model like GPT two. GPT two, you know, was able to complete tasks very very reliably. You know, that take, you know, humans a few seconds. But anything longer than that, it it starts to fail. Also, maybe maybe maybe maybe. It'd be helpful to give a few concrete examples of of tasks. So some of the shorter tasks are like very, very basic. So, yeah, one one example is like, you know, which of these files contains your SSH key? And, you know, one of the files is named, you know, SSH key.

• 23:52Speaker 03

  And then, you know, the others are like, you know, email from John or, you know, whatever. And so most most models can do that and that takes people, you know, like. About a second or a couple of seconds or something to to to complete. Yeah, we we have others that are kind of like somewhat similar, like kind of basic, like like completion. Like, you know, here's here's an email, like what would be a reasonable response? And then two of the responses are like, you know, they don't make any sense. And then one of them is kind of basically reasonable. And, you know, that takes people, you know, 20 seconds or something, 30 seconds to like read the responses and kind of judge.

• 24:27Speaker 03

  And then, yeah, we we have, you know, we in in the kind of middle range, we have tasks that are like, you know, given this.

• 24:35Speaker 03

  You know, this CSV file that, you know, has some like kind of plausible, realistic data, you know, compute some basic stats on this. And so this may, you know, this this takes, you know, a data scientist like a few minutes, like five, 10, 15 minutes or something. Depending on, you know, the specific task. On the longer end, we have, we have tasks that either, yeah, require like quite, quite, quite a bit of expertise. Or, you know, like many, many steps to to complete. So we have machine learning tasks that are like, you know, train, train a model in this kind of, you know, that's, that's very weird. That, you know, such that like, you know, the the code for training this model is like not really available online.

• 25:16Speaker 03

  So one, one example is, you know, train, train a mask language model without using the division or exponentiation operators. And so you actually have to be like pretty clever about, about, you know, how you, how you actually set up the architecture to to do this. And, you know, this, this kind of, you know, the hope is that this can can can help us measure a model's ability to to generalize beyond their training data.

• 25:39Speaker 02

  Yeah, there are some which are a bit like ArcAGI sort of themed in that they're like, like you have to figure out. You have some black box that's computing some function. And it's like, you know, it's that you have it's that it's, you know, that it's the composition of some set of primitives. And you've got to figure out like what, what function it is. Or like you, you know, you have some long binary string and you've, you've got to figure out what the pattern continuation is. And sort of puzzle type, type tasks where it's very like, you know, some, some ML type tasks and be. That basically regurgitating something that's like a tutorial on, you know, how to build your first like ResNet or whatever, you know, just like works pretty well. So yeah, having these like weird tasks that are either some kind of, you know,

• 26:23Speaker 02

  unknown object that you need to interact with and figure out what it is, or tasks that sort of resembles normal work, but has some weird constraints such that you can't kind of just do the standard thing. I don't think all of our tasks, you know, hit those criteria like some of them. You can just do the standard thing, but we generally try to avoid that.

• 26:42Speaker 03

  You know, we have, we have this distribution of tasks and, you know, you can imagine them kind of like ordered by length of time for, for humans, either measured or estimated. And then we see, you know, which tasks do models succeed on and which do they fail on? And it turns out this is kind of an empirical finding that models are much more successful on the shorter tasks than they are on the longer tasks in general. This holds across, you know, a wide range of, of models, you know, all the way from, from GPT two up to, up to recent models. We fit a logistic function to, to this distribution of successes and failures. And what this lets us do is it, it, it lets us kind of, you know, it, it's basically our model for, you know, each individual model

• 27:25Speaker 03

  of how likely it is to succeed at a task, given how long the task is. And from that, you know, we, we take the kind of 50th percentile.

• 27:38Speaker 03

  So where this logistic function, where this model estimates that, you know, this given model is 50% likely to be able to, to complete a task. And that, you know, that, that's what forms the, the time horizon number for, for a particular model like, you know, OPUS four, four, six. We can take each of these time horizons for, for, for each model and we can see how these, how this time horizon metric has been changing all the way back from GPT two, you know, up to, up to recent models. And so this is kind of, yeah, this is what gives us this like unified metric that lets us kind of quantitatively compare, you know, AI capabilities across like multiple orders of magnitude of, of capabilities.

• 28:17Tim Scarfe?

  One of the really harder examples I saw was, you know, like, I want you to compile, I want you to write a kernel compiler to make CUDA go faster or something like that. So some of these seem really out of distribution. There's probably only a hundred people on the planet who are doing stuff like that. And, and some of them are really trivial, but this human difficulty thing in particular, like.

• 28:38Tim Scarfe?

  Is that confounded in any way? Like, do you think it makes sense to think of human difficulty as being one variable?

• 28:44Speaker 02

  Yeah, obviously not. In some sense, you know, that's a very silly simplification. And like, you know, different humans will get wildly different times. Even among the people we've, you know, we've tried to select for people who have the sort of this right level of expertise, you know, there's a large variation in, in like the baseline times are often like three X different or something. Um, maybe I'll just say a little bit about like, why, why use the human time metric? I think we want a measurement that, uh, I think there's two main properties we want. We want it to be interpretable, like what this means for the world when, when models can, you know, do this level of task. And we want it to be something that we expect to see predictable trends on. Um, so we're not going to do perfectly

• 29:29Speaker 02

  on either of these, but something like, you know, how long does it take a human with, who has roughly the right expertise, but doesn't know how to do this task in particular? Like, for some reason, that's reasonably interpretable because it's sort of like, could you contract this work to, to this model? You know, can you like sub in this model and, you know, for like the first week of someone's employment, uh, you know, like, uh, the first week they're on a job, you know, the model could do what they could do in the first week or something. Um, and it's, you know, we, we expect it to scale somewhat predictably because this is, you know, it's capturing some combination of like, uh, something

• 30:09Speaker 02

  like number of steps or like how hard you have to think to, to do each of the steps or like, there's a few different mathematical models you could fit that to sort of, you know, what is a task and why does, why do humans take longer at it? And why does that make it harder? Like, it could be like, you know, you have a, uh, like constant hazard rate, like you have a chance of failing at each step. Um, I think it doesn't quite fit that one. Uh, you could also think of it as, you know, there's some kind of distribution of difficulty and like what, uh, you know, what is the likelihood that one of the subtasks is outside your ability or that, I think it's, it's actually, it's like a bit better, like, once you, uh,

• 30:51Speaker 02

  The hazard rate goes down over time slightly, but you know, there's, there's, there's some kind of, you know, basic theoretical idea of like, you know, if the task involves more steps, it's, it's going to be harder. Obviously tasks that are, you know, strictly like compositions of like, first you have to do this task, then you have to do another task. You know, it's like, well, that clearly is harder than just doing one of them. Um, so, so that, you know, there's some sort of basic reason to expect that's reasonable. And then, yeah, just we, we see the empirical regularity, but there's a bunch of degrees of freedom to fudge things, you know, I, I like, I think I'm worried that, you know, we could fool ourselves by changing some other parameters of, of the tasks as we scale up the human

• 31:36Speaker 02

  time, because you, you can't just totally, you know, vary the human time freely. Like you have to change some characteristics with the task. And we tried to make the very easy tasks sort of be from roughly the same distribution and kind of subparts of the, the harder tasks. Like, oh, you know, you sort of need to do this one step on the command line that you might need to do in the middle of doing some kind of software engineering or some kind of, you know, other, other task. Um, but you can't do that perfectly. And it's like, yeah, you could have experimented bias where we sort of made them easier in other ways about the right amount such that the line would be straight. Um, I think that is like

• 32:15Speaker 02

  somewhat addressed by the fact that we saw like, you know, that our predictions were reasonably good for models that we hadn't seen before. There's definitely lots of room for things being, being weird. And of course, you know, sort of like, is it a like good enough metric to be useful? Or like what, what else would be, would be better?

• 32:41Tim Scarfe?

  I think that's reasonable. I mean, if, if I understand correctly, I, I think the, the human distribution was log normal. So taking a geometric mean of the successful attempts. I mean, that seems like a reasonable thing to do. But like one of the cruxes that we'll keep coming back to is, you know, like when you employ someone for the first time, uh, you've been doing your job for, you know, maybe you're maintaining this repo or something and you've got all of this tacit knowledge. And I mean, I, I like saying that knowledge is non-fungible. So, you know, unless someone has been on the same path as you, you can't just like tell them how to do the job. They've actually got to be doing the job for quite a while. So, you know, for example, they, they might be intimately familiar with this particular type of thing. They might know that they can use these Python libraries. They might have thought about it before.

• 33:24Tim Scarfe?

  So like the the the inaction of the intelligence was all the stuff they've already done and all the people they've worked with. They now they've got the blueprint in their mind and they just do the thing. And it's almost like they're they're in like automation mode. And then someone who is naive to the task would be in intelligence mode because they would need to acquire the the specification. So like it's always a bit of a fine line between which mode are they in?

• 33:48Speaker 02

  Yeah, so I think the reason why we chose this, the the measurement being a human who has the background expertise but is new to like this specific job or this specific task. It's like that's roughly the sort of level of knowledge we expect models to have. So that, you know, they sort of like know, you know, we basically don't expect them to be bottlenecked on expertise that is available on the public internet or, you know, things that people could learn in university sort of thing. So like, you know, they're they're coming in with probably the level of, at least the level of knowledge of someone who's sort of an expert in the right discipline, but they won't know that company's specific software or the, you know, sort of this exact problem before. So that's like, hopefully, is sort of roughly the right analogy.

• 34:32Speaker 03

  You know, I do think that like, to the extent people like interpret the, you know, the the kind of takeaway numbers as like, oh, yeah, you know, you know, Opus 4.6 can like do anything that I do in my job. You know, that takes me 12 hours or whatever, you know, to the extent that people have that takeaway. Like, I think that is, you know, like, almost definitely, you know, kind of an overestimate.

• 34:58Speaker 03

  For example, because of because of this issue where, yeah, you know, when you're doing a 12-hour task in your job, you could not easily delegate that to to to a human contractor. You know, it would take them, you know, maybe like weeks or something, you know, to to to do a task like that.

• 35:14Tim Scarfe?

  And just quickly, where did you kind of find the people? Do I understand that some of them were contractors and some of them were employees? And how did you do that kind of matching process?

• 35:25Speaker 03

  I think we, yeah, we put out some kind of public advertisements. I think there were, yeah, like, like job boards that we posted on.

• 35:36Speaker 03

  And then, yeah, we did some of it ourselves. I think, yeah, some folks came from our professional networks as well.

• 35:44Speaker 02

  This was not like, you know, this is very noisy and, you know, the people weren't exactly fitted to the task right. But like, that's probably not our biggest source of uncertainty. Like, the biggest source is more like, it's probably more of the selection effect of like, tasks that you can make into a benchmark rather than like, you know, I wouldn't trust the exact time horizon number that much. And it's certainly not like, you know, oh, the models can do all of the tasks up to four hours and then none of them above that, you know, the fit is pretty noisy. And yeah, the, you know, the inter baseline variance is kind of high. And it's more like roughly what is the trend? Roughly what is the sort of level of tasks these models can do? You know, you shouldn't take any specific number too literally because there's this huge problem of like, you know, distributional shift between the benchmark and the real world.

• 36:28Tim Scarfe?

  And the only reason I ask that question is I'm sure you folks, you probably struggle to hire people. It's really difficult to hire people. So like, if you're getting people to solve very challenging problems, it's not like you can just go out there and just grab competent people. It's very, very difficult.

• 36:43Speaker 02

  At some point, for our E-bench baselines in particular, we got more, you know, a large number of benchmarks per question. And we were looking at people's like qualifications and years of experience and things. And we actually ended up with a negative correlation between years of experience and, and performance. Because like, the sort of people who are in network, like our friends were kind of doing really well. And the people who are more qualified were like, actually not doing that great. So. Yeah, you know, it's tricky.

• 37:11Tim Scarfe?

  Well, yeah, exactly. I mean, I don't want to spend too long on this, but I have similar intuitions. I think that knowledge is perspectival. It's quite path dependent, so you're going to find people in group that are just culturally thinking about things in the same way. And, you know, because when we do have these abstract notions of skill, like, oh, you know, they have a PhD, they have this many years of experience that, you know, it's actually not a very good reflection. So like, there is a bit of a thing here about like, using abstract notions of capability doesn't necessarily generalize as you can attest to with hiring.

• 37:43Speaker 02

  Yeah. But yeah, I mean, in the real world, people do get hired based on qualifications. So in some senses, it's a sort of the economic relevance of someone being as good a match for their job as their qualifications look like is that is sort of roughly the right like thing to be measuring.

• 37:59Tim Scarfe?

  The other thing is we should talk about the agentic harness. So almost everyone now, I'm sure everyone in the audience has like a Claude Code subscription. We can talk about the leak maybe later as well. That's quite fun, but it leaked yesterday, this was good, but, you know, or Codex and that is an agentic harness, right? So, you know, a language model just gives you the tokens, but we need to have an agentic harness so we can like, you know, give it a plan and you can call these tools and you've got this environment, you've got a security context container. So now you folks have been doing this for years now. So you were doing this long before Claude Code and Codex came out. So and you've actually over time evolved your agentic harnesses. So tell me about that.

• 38:37Speaker 02

  Yeah. So I remember text DaVinci, something that like GPT-3 instruct models, like copying and pasting code into the terminal for them and, you know, being the agent harness myself and then gradually like automating this. And it was kind of interesting to see the malls going from like, you know, GPT-3 sort of has the idea of like, if you tell it, it can run commands in a terminal. Sometimes it can suggest, you know, kind of relevant commands, but it's not really, you know, if you just put it in a full agent scaffold, it just falls over. Then like, I remember the first time we saw a model like look at what processes were running and then be like, oh, that one's me.

• 39:19Speaker 02

  Was like, oh, that's cool. They like really failed on that one before they they used to like, you know, like kill their own process while they were doing other things or something. So yeah, it's been been interesting to watch that like go over time. I feel like, yeah, I feel like that was like, very predictable that this was where things were going or something. Um, yeah, I think the other thing that we learned about scaffolding mostly was it's,

• 39:46Speaker 02

  it's hard to make your agent harness really good on a diverse set of tasks. It's easy to make it bad and it's easy to, you know, you can get much more improvement if you're targeting a narrow distribution of tasks, but you probably then do do worse on other tasks. So I think when we see people being like, you know, there's some new impressive result, it's like, how much task specific scaffolding iteration did you do on that? Because that really makes a big difference. And the fact that we're just using one scaffolding pretty simple, like over all of the tasks, I think is like, yeah, makes it makes a fairly large difference. Um, yeah,

• 40:29Speaker 02

  and generally sort of, yeah, the things with more bells and whistles haven't done that much better than the pretty basic, just like, give it bash and like, append things to the prompt and like, maybe some kind of compaction.

• 40:46Speaker 03

  I think this is not, you know, news, news to people in your audience, probably, but we've seen, you know, really pretty dramatic like increases to like returns from inference compute. For us to be kind of confident that a particular, you know, a new model, for example, can't complete, you know, a task given, given a basic agent scaffold, we generally think about like needing to spend, you know, on the order of of at least like, you know, hundreds or like, you know, low thousands of dollars in order to be confident that it actually, you know, really is plateauing. And it isn't just the case that, you know, it didn't, it didn't have enough time to, to complete the task.

• 41:27Tim Scarfe?

  And just, just on the scaffolding stuff in a bit more detail. I suppose, first of all, there's the credit assignment problem, right? Because you can put all of these different bells and whistles in the scaffold. Like you mentioned, compactation, that's a relatively recent innovation. And I think recently, when you changed some of the scaffold, you kind of said, okay, well, now the performance has actually changed across this suite of model task pairs. You know, how much of a difference does it make? And also, what kind of failure modes do you see? And what have you tried?

• 41:59Speaker 03

  A lot of the things we, I think we've tried are like, you know, related to giving the model kind of like more information or like more direct access to tools. I guess, yeah, one thing that I think has been important for us is actually just telling the agent how much time it's spent and how many tokens it's used out of its token budget. So without that, agents, you know, will often just, you know, they'll either like submit, you know, their solution way too early, or they're just kind of not calibrated on, you know, how long should they spend? And I think it's interesting because, you know, humans, we have kind of a lot of implicit information about this. So, you know, when your manager gives you a task,

• 42:44Speaker 03

  there are a lot of like implicit signals about how long you should spend on the task. You know, if they, you know, they might like offhand say like, yeah, and I'm excited to like see your results tonight or something. And so then you're like, okay, cool. Like, you know, I need to like get a first draft of this, you know, done in the next couple of hours. And so I can't, you know, spend, you know, days polishing the results. But I think agents, like, it's easy to kind of forget that, you know, agents, you know, they just have their, they just have their prompt. They just have their context. You know, they don't, they don't know, you know, they don't have these heuristics of, or this information about like what you actually expect from them. You know, whether the thing you're telling them to do is like a really quick thing that you just want done in the next five minutes or is much longer.

• 43:29Speaker 03

  And so for us, yeah, like, you know, just, you know, when we have a token budget, telling the agent like, yeah, you've used, you know, 100,000 tokens so far. And that's like, you know, 1% of your token budget. And so, you know, like, yeah, so the agent knows. Yeah, yeah, exactly. Yeah.

• 43:44Tim Scarfe?

  For a model, we have like the, like, how likely is it to solve a task? And on the X axis, we have the, you know, the different tasks at different time horizons. And if I understand correctly, I think you have about eight agents attempt the task. And you also bucket the task because there are obviously different amounts of tasks in different groups. So you kind of normalize that. And maybe the data just looks a little bit like an S curve. So, you know, I'm trying to understand what the intuition was for using it. And I think there might be some sensitivities, like was there an issue with the thin tails? And then there's, you know, like the type of slope and whatnot. Just, just tell me about that. By the way, I think you also mentioned in the paper that there was some kind of like psychometric intuition.

• 44:28Tim Scarfe?

  You were looking at the literature for figuring this out.

• 44:30Speaker 02

  Yeah, so it's pretty similar to item response theory. And you can do a whole like Bayesian analysis with, you know, imputing like a, you know, task difficulty parameters and model ability parameters simultaneously and things. In general, I am have a policy of be like, be very wary of complicated statistics. If you can't see the thing that you're interested in on a graph, like you really should, if you, you should be able to plot it and look at it and be like, oh yeah, it's about that. And like, you know, that's sort of sound, it's, it's like hard to go too far wrong when you sort of have that as a principle. So again, I'm like, yeah, I think that there are various arcane things you can do to like fit this pan in different ways.

• 45:15Speaker 02

  But the, I'm like, yeah, I don't trust anything that much more than eyeballing the graph and being like, oh, well, you know, you know, up to here, yeah, it's basically doing all of the tasks. And then at this point, you know, after here, it's nearly not doing very many of them. So it's somewhere here. But yeah, it does, I mean, it like, looks like a logistic. And yeah, this is what you would do for the like I R T, having, you know, having humans complete questions on an exam or something like that. The specific thing that we messed up was having a regularization term, penalizing the slope of the logistic, which didn't have an effect in the regime where it, where there was more data, but. As we like are starting to saturate, the regularization was just like making it a bit

• 45:58Speaker 02

  shallower than it should have been and therefore pushing the 50%. So, you know, always look, look, look at your data on a graph. Good, good, good practice.

• 46:07Tim Scarfe?

  Oh, that's interesting. Yeah. And the reason I ask is I think you published a later note saying that had you used or if you used a fixed slope logistic, it might cross validate better and the 50% recent horizons would actually be up by about 35%. So these are quite significant differences.

• 46:24Speaker 02

  They're small compared to the error bars. Yeah, the error bars are like 2X on either side or something from, from the, the most, most recent model. So yeah, it is, I, I mean, basically, yeah, you should be like, the, the error bars are real. These numbers are not.

• 46:36Speaker 03

  Like for us, for us, um, and, and I, I think this kind of gets at for us, you know, sometimes difficult like science communication questions, um, where like we really do have a lot of uncertainty about, you know, individual, you know, the individual numbers here. Um, and so, um, you know, like 30%, you know, 30% difference is, is actually like, like relatively small for us relative to, you know, like, um, uh, yeah, for example, if, you know, we had used, uh, you know, a somewhat different, uh, distribution of tasks, um, that, you know, that's likely to cause, uh, you know, like, yeah, maybe 2X differences or, or something.

• 47:12Tim Scarfe?

  The, the other million dollar question is why report 50% as the headline number? But because if, if you think about it, um, if I want to write some code, because like the, the elephant in the room here that, that we'll get to is this is being used as, as an argument to say that software engineers might be unemployable soon because we can automate what they're doing. But 50% reliability isn't really in the ballpark, is it? I think it would need to be what like 80, 90%?

• 47:40Speaker 02

  So I think we should distinguish here between, um, like reliability on a particular task, like what, you know, if you attempt repeatedly attempt this task, what fraction of times do you succeed versus, um, like probability of success on a task, like given that you know, the, the human time, like, you know, of that distribution of tasks, like, can you do this particular task? So when we, when we look at it actually for almost all the tasks, models either succeed every time or fail every time. Um, there's, there's some tasks for which they're, they're unreliable, but it's mostly a case of like, is this? You know, what fraction of tasks at this human time level are in the like models basically, you know, or this particular model basically always succeeds or basically

• 48:20Speaker 02

  always fails. And that may be more predictable in any specific case than just, you know, you have more information about the task than just, just knowing roughly how long it takes humans. So I think it's like not, there's not necessarily a great translation between the, you know, time, time horizon percent number and like, if you are trying to get models to do tasks of roughly that length, you know, what fraction of the time does that succeed? Because you can, when you're doing that, you will pick tasks that you want models to succeed at. It is information about how, you know, what fraction of the things will they be able to do, but it's slightly less about like, oh, am I going to be in this regime where I keep giving it things and then I don't know whether it's going to succeed or fail.

• 49:05Speaker 03

  It's overall, to me, pretty unclear, like what the kind of right number, or you know, right, right level of reliability we should be interested in is. So one argument you could make is, you know, maybe we should be interested in something like 10% reliability, because once models are able to do, you know, some set of tasks 10% of the time, we'd expect, you know, AI companies to be able to kind of, you know, get, get enough, you know, positive reward signal on, on, you know, tasks of that difficulty or of that type, such that then they can kind of, you know, more easily bootstrap from, from 10%

• 49:47Speaker 03

  up to like 90 or 95 or higher reliability. I think a lot of it basically depends on the question you're interested in. So yeah, I think about it as being like, kind of lower reliability is kind of more likely to tell you something about, you know, where things are headed, you know, might, might, might be something like a leading indicator of progress. And then higher reliability, you know, tells you, you know, or, you know, the time horizon of models with, with higher reliability, you know, tells you something You know, more closely about like, you know, what can I use this model for in my like day to day or something?

• 50:28Speaker 03

  But you know, as we as we've talked about, there are kind of already also, you know, of you know, these other like major sources of uncertainty that that affect our, you know, understanding like, you know, the task distribution, you know, the difference between kind of in context or high context versus like low context work. Just actually getting kind of good estimates of a higher reliability time horizons is substantially harder. And and so our our air bars would would just be much larger.

• 51:03Speaker 03

  And so and this is, you know, I think this is a weakness or like I I'm very interested in, you know, you know, much higher reliability time horizons, but it's it's substantially more difficult to measure because, you know, if you only have like one, you know, one failure out of out of 100 or something like, you know, you have a lot of uncertainty if if that failure is is noise or is is real.

• 51:25Tim Scarfe?

  There is an argument for statistical validity and on the tails, they're sparser, increasingly estimated and so on. So that that makes a lot of sense. But you made a comment about, oh, you know, it's a signal if we get 10%. But I was kind of thinking back to what we were saying earlier that, you know, maybe they they could give the right answers for the wrong reasons. And maybe we should talk about like the evaluation, right? So these are quite interesting tasks in the sense that they are verifiable. You know, there's no interaction with other agents, they're relatively static environments, you know, weak penalties for single mistakes and so on. And in a sense, these are like, I think as well, in most cases, they are a binary result,

• 52:09Tim Scarfe?

  sometimes continuous, and then you convert it into a binary. So it's a fairly automated setup. But are you digging into like weirdnesses there, you know, like are you? Do you have a bit of an intuition on are they doing the right thing for the right reasons? Or are there lots of false positives where they they did the thing, but it was kind of degenerate?

• 52:27Speaker 03

  I think this is like, this is one of, one of the kind of like aspects of, of like, meters culture that I, that I like the most is that we have like a very, a very deep kind of culture of like, you know, looking, looking at our data. Like, you know, we have these like, you know, we have like pizza parties where we, you know, just read through agent, agent transcripts. A lot of this work for us kind of happened when developing the tasks themselves. You know, we, we would see very often, you know, both false positives and false negatives where, you know, for, for example, tasks, you know, you know, a

• 53:07Speaker 03

  task might, you know, you know, isn't configured to allow internet access, but it turns out actually, you know, it requires internet access to complete. Or, you know, the file wasn't, you know, wasn't uploaded properly to the container or something. But then also, yeah, you know, we, we have seen cases of, of reward hacking. A lot of, a lot of, yeah, that work though, kind of went into hardening the scoring functions to make it, yeah, more, more, more difficult for, for us to see false positives. Although, yeah, we, we do still see, see agents reward hacking. I think, yeah, maybe, maybe even, yeah, increasingly so.

• 53:48Speaker 02

  Yeah. For, for the RE bench tasks in particular, we had specific criteria of like, it shouldn't be able to solve them like without iteration. Like if an agent can just kind of like write out the solution straight out, that, you know, that, that would, that is like not interesting. We generally had this like quality assurance for tasks process where you have humans do it, or at least like sort of approximately do it. Maybe they speed run some of the bits, but kind of checking that like sort of everything works as expected and you can't, you know, just like guess the answer or.

• 54:21Speaker 02

  You know, super easily cheat and the instructions are clear and stuff like that. So I think it's, you know, there'll still be some of these, but generally, you know, we've, we've looked at them reasonably, reasonably carefully and like it's maybe, it's harder to see if they're solving it in degenerates because it's, you know, similarity to training data is maybe one of the things where it's like, oh yeah, maybe they, they are, you know, they seem to be kind of iterating and doing, using, doing kind of reasonable problem solving strategies, but actually maybe the lab had a really similar distribution of tasks and we don't know exactly, you know, we don't realize how in distribution this task actually, actually is or something. I mean, I think it's like, yeah, there's probably some of that going on. There's some of the stuff

• 55:06Speaker 02

  of like, oh, some of these types are pretty weird and one off and it would be pretty surprising if it was like.

• 55:11Tim Scarfe?

  You know, another million dollar question is many, many folks in public discourse, they are, you know, like Will McHaskill on the Sam Harris podcast last night. It was a, it was a great conversation, but you know, he, he was kind of talking about AI risk as, you know, maybe in a year, maybe in two years, we'll, we'll have AI models doing things that are like, you know, a month or two months for, for a human. And at the moment, I don't think there are any tasks over, over 30 hours that have been evaluated by, by humans. And then we get into this question of like, if, if, if the, if the public discourse is talking about the least constrained region of the graph, are we getting into extrapolation here? Like, you know, how, how like legitimate is, is it for us to talk about AI might be able to do things that take a month or two months?

• 55:55Speaker 03

  Predicting, predicting things is hard, you know, especially about the future.

• 56:02Speaker 03

  I, I, and, and yeah, I think, you know, there are a lot of different kind of perspectives or kind of prior beliefs people, people can have that, you know, I think there's a wide range of kind of reasonable, you know, judgments about, you know, where, where, where we're going to be. But of course, like that is, you know, that is, you know, doing, doing, doing that kind of prediction is, is a different activity than, you know, talking about data that, you know, has been collected with, with. A kind of concrete methodology that and you know, we you know, we have the results already. You know, one, one, one thing I can say, I mean, I have been surprised, I think to some extent by the

• 56:45Speaker 03

  kind of how, how well the, the trend line, the kind of original trend line has held up. And I do think that is like some evidence.

• 56:55Speaker 03

  I, I'd maybe, I'd maybe say for, for me at least, it's, it's kind of decent evidence. About, you know, where, where, where things will go. A colleague, a colleague of mine recently, yeah, wrote, wrote a kind of short blog post talking about this kind of intuition of, you know,

• 57:14Speaker 03

  straight lines on graphs, you know, lots of people have different, different models of, you know, how progress is happening and, and what's going on, but a very, you know, if, if you have observed, you know, a really kind of, you know, a, a, a robust trend over, you know, a decent period of time, I think, especially in, in AI where, where progress is, you know, to, to a decent extent, kind of systematic.

• 57:44Speaker 03

  You know, I, I definitely do kind of, you know, put, put weight on, on that trend continuing. But, I mean, yeah, there, there, there are a bunch of, a bunch of reasons why, why it might not.

• 57:56Tim Scarfe?

  I, I think software engineering is a specification acquisition problem. So, you know, it's, it's very difficult. We, we don't know ahead of time what we're building. I'm, I'm sure you folks can attest to this, right? So you, you build some software and the first version is buggy and your users use it and you find lots of edge cases and then you revise it. And then you have this kind of thing in your mind, you know, after the 10th revision and you've kind of created these lovely representations and abstractions and coarse grainings and you kind of say to yourself, you know what, if I could throw all the code away, I could build it 10 times quicker. Because I know exactly what to do now because I've actually enacted the intelligence. I've actually built the, you know, I've found the contours of the domain. It's now an automation problem, basically. And in a sense, this, this contamination.

• 58:40Tim Scarfe?

  Thing is a concern for me because when people use Claude code, they're kind of, they're taking your data. So there are people out there that are writing kernel compilers and there are people out there doing all of these, all these different things. And you know, Anthropic is just sucking that up. And, and then at some point it becomes an automation problem. So if, if you're putting a task in there, which is essentially a head query. So I'm using like information retrieval language here. So you know, like a head query is it's something that's, you know, in the mode of the distribution, it's used all the time. So it's a common task. Claude code will give you the specification because it's already been stolen from other people. Not stolen, but you know, taken from other people. And, and then if, if you give it like something on the long tail, then you as the developer have to give it the specification in the prompt.

• 59:24Tim Scarfe?

  And then again, it's an automation problem. So automation is, is really easy. So is, is that's what's, is that what's happening? Like, do you think that the increase in, in the timelines could just be explained by the acquisition of all of this? Kind of knowledge from other people doing similar tasks?

• 59:42Speaker 03

  Yeah, yeah. I mean, I think, I think it's a super, a super central question for kind of interpreting, interpreting where we're at. First thing I'll say, it's, it's hard to know. It's a big question. And so I think we, we, you know, we want to kind of have a decent amount of uncertainty. Or, or, you know, we want to kind of take each, each individual piece of, of, of evidence that we've collected, you know, as, you know, some, some evidence. We do see models performing better on, you know, tasks that have, you know, really clear feedback signals that are like extremely well specified.

• 1:00:22Speaker 03

  You know, that are in, you know, these, these kinds of, you know, yeah, domains like, like in software engineering where, you know, if you have written out a spec, you can, you can kind of iterate and, and grind against that. We do also see models performing much better on, you know, mess, so-called like messier tasks, where we haven't already provided this like really clean spec.

• 1:00:50Speaker 03

  So one, one kind of approach we, we, we've, we've taken for, for creating tasks recently is in particular, yeah, to kind of try and create, yeah, like, like messier tasks that are less well specified is basically like relaxing this kind of automatic scoring constraint. So we don't need to like write, you know, a really clear, you know, well-defined scoring function and just, you know, writing a couple of sentences to, to, to a model, you know, hey, like, you know, build this like large piece of, of software. I'm not going to tell you exactly what I'm looking for, but I'm going to say, you know, it, it needs to be, it needs to be good, you know?

• 1:01:33Speaker 03

  And so, and so the model needs to kind of figure out like, what actually should I build? At least me, me personally, I think we don't, we don't have, these results aren't like, kind of collected and like, like, we don't have kind of as, as systematic results as we do compared to like, like time horizon, partially because, you know, scoring is, is, is qualitative now for, for these tasks. But my, my impression is that, you know, models are, are worse on these types of tasks than they are, you know, when you give them a clean spec, but they, they have been improving at, at maybe something like a kind of similar rate. I think there, there are some other sources of evidence we, we, we have about this, but that's kind of one, one major piece of it for me.

• 1:02:11Tim Scarfe?

  Messier time, by messy tasks, we mean like ambiguity. And this is absolutely a common thing, right? You know, we do vibe coding and we, we start off with an ambiguous specification and then reality pushes back and we find the contours of the problem. And then we, we keep telling Claude code, oh, actually, no, don't do that. Do this, do this, do this. And then we, we kind of find the shape of the problem and it gets better and better over time. But, but, but the thing is though, I mean, the, the source code for Claude code leaked yesterday and my friend, he, he's a very good software engineer and he was looking through it and, and he said, yeah, I don't, don't want to like, you know, bad talk, anthropic, but apparently it's not very well factored and, and, you know, control flows all over the place. And it's a bit, you know, he said if his intern did it, he would have been displeased. But, you know, I don't know whether they've even

• 1:02:55Tim Scarfe?

  looked at the code. Someone joked actually yesterday, there's probably more, you know, humans are actually looking at the code for Cloud code now and and maybe they weren't before. But the thing is, um, there, there's, there's always like areas of ambiguity. And L L Ms, they are, they do more is, you know, they do more with more. So intelligence is more with less and L L Ms do more with more because the specification, the, the intelligence comes from the human supervisor. So when you do give them ambiguity, you just get a lot of like, you know, unfactored code all over the place. So, um, in, in a sense, like, is, is this, does that make it harder to evaluate it? Because it might solve, it might give you the answer that you're asking for, but it's kind of creating a bit of an unfactored mess at the same time.

• 1:03:35Speaker 03

  Yeah, I mean, I think this is a super interesting question. One, one analogy I think about, uh, uh, sometimes is, um, uh, is compilers. So, um, I, you know, I'm, I'm, I'm young or like I'm younger than, uh, you know, I was born after compilers were, were, were invented. But, um, uh, you know, I have, I have some impression that, uh, you know, kind of pre compilers, you know, people were, were, were hand crafting this kind of beautiful, uh, assembly that was, you know, extremely efficient. You know, uh, you know, every register is used, you know, like, um, like, like there's, you know, uh, you know, you're not wasting memory. And then compilers came along and now, now they're just spitting out this like garbage, uh, you know, machine code.

• 1:04:20Speaker 03

  Just, just like a, you know, gigantic amount of, of, of, of assembly that is just like, um, you know, it's, it's, it's not optimized. It's, it, it, you know, it takes so much memory, it's slow, whatever. But it turns out that, you know, being able to kind of use this like, um, to, to, to automate a large fraction of, um, of, of, of the process.

• 1:04:42Speaker 03

  You know, people have disagreements about, you know, the state of, you know, uh, software engineering. But, um, I think it's pretty reasonable to, to say on the whole that, uh, you know, compilers have been a very useful, uh, extremely important, you know, part of, of, of, um, getting, getting us where we are. And, um, you know, I, I think it's basically, yeah, it's, it's kind of not clear to me that, um, uh, you know, models outputting code that is like bad for humans to read and use. Necessarily means that it'll be bad. For AIs to read and use and and and build on. I mean, I think I think there are definitely, you know, kind of principles that, you know, you like will also transfer or like, you know, will be useful for models, you know. Obviously, you know, there's there's some kind of horrendous spaghetti code that, you know,

• 1:05:27Speaker 03

  you can imagine writing that not even models would be able to to read, you know. I've I've written some of that before, but I think there's there. Maybe this gets again at kind of this, what what it seems like maybe, you know, somewhat different, different, different perspective between us around kind of, you know, you know, is the important thing that models are kind of solving problems in the way that people are solving them? Or is the important thing that they're kind of solving solving them at all? And I think I, you know, I do think it is. I I'm not, I don't want to make a, I don't want to overclaim or like, yeah, I do feel like, you know, it might be really important for models to get way better at like writing

• 1:06:10Speaker 03

  clean, you know, good code that that that seems pretty, pretty plausible to me. But it doesn't seem, I'm not certain of that, at the very least.

• 1:06:18Tim Scarfe?

  I've got several friends who are not technical, who are experimenting with vibe coding, and they they show me their application. And it's this kind of more with more thing. So there's this big dashboard and there's a million different buttons and, you know, implement implemented the same thing, doing multiple things. And, you know, there's no database on there yet and so on. So at some point, you know, this, there is a phenomenon that when a level seven engineer from Meta does vibe coding, it's amazing, right? Because they they know how to structure things. Some of these things in the specification are just important, right? You know, like, is it serverless? Is it multi-tenanted? Like, how do we do Google authentication? Like, you know, what kind of database is it? You know, is it is it a VM? Is it serverless? You know, and you make this series of decisions.

• 1:07:01Tim Scarfe?

  And then you've got people using your application and then you can't really wind that back. It doesn't matter if you've got the magical automation machine, because you can't you can't easily roll that back, because there's lots of complexities, you know, do you C I C D testing, blah, blah, blah. So do you see what I mean? Like, at some point, you need to have a competent human who actually just has a pretty good idea of like, what needs to happen.

• 1:07:20Speaker 02

  I mean, I feel like we've probably all had this experience of like, I don't know, one of our.

• 1:07:26Speaker 02

  Engineers got super excited about Claude code and was also telling everyone that when we had infra problems that we should just ask Claude to solve it. And I feel like this went fine with him because he sort of, you know, it's almost like, you know, the agents knew that they couldn't bullshit him. But like, you know, I had some questions. He was like, oh, just, you know, ask Claude. And I was like, oh, you know, how do I set up my AWS configure something, something is, you know, telling me something. And like, Claude went and looked on Slack and was like, oh, you know, you should like, do this thing. And it like, turned out that that was like a mistake that someone else had made. And they were like, you know, how do I fix this or something? And be like, oh, you know, it seems like the convention at Meter is to use this thing. And like, yeah, I'm like,

• 1:08:09Speaker 02

  oh, my God. Like, I had to complain and like, like, my Claude's are dumber than yours. Like, they know they can, they know they can get some stuff past me. But like, yeah, so there's definitely a sort of observer effect of, you know, something in the language you're using to ask for things, or whether you're like, wait, wait, no, not that. Yeah, that is an issue. I mean, I think the, yeah, to the extent that you can actually measure this, sort of the, does one test of, is this code high quality enough is like, can you build a big application? Like, if you, if you're like, oh, this coder, their code, it's disgusting, but they've actually, you know, built this incredibly complex thing

• 1:08:52Speaker 02

  that works great. Then you're like, well, they, you know, something is working. You know, the main reason you expect bad code to be bad is like, you can't actually build something that sophisticated, because it, you know, you get bugs and it's all too complicated, and you can't figure out how to fix it. So in some sense, if we see models building things that do actually work, that are very complicated, we just, you know, it's like, well, it's less interesting exactly how they're doing that. But it's maybe bad for for human observability and it

• 1:09:22Speaker 02

  Also maybe gets into this thing of, you know, we expect malls to be able to do much better at well specified tasks. And we sort of, you know, to the extent that we have things that we can measure, those things will go up. But whether that was what we actually wanted is less clear.

• 1:09:39Tim Scarfe?

  I guess the question is, what is the strongest defensible claim here? So a lot of folks in public discourse, they're saying, software engineering intelligence is doubling every seven month and every seven months. And Dario released that blog post recently, the adolescence of technology. And he was being super bullish about it, even though some of his own internal research is published, far more skeptical research that you probably saw. But is it fairer to interpret it as something a little bit narrower, like autonomous success on low context, well specified, automatically checkable, technical tasks is rising fast?

• 1:10:12Speaker 00

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50 page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required compatibility and availability varies 18 plus.

• 1:10:38Speaker 01

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 1:11:09Speaker 02

  Like, yeah, hill climbable.

• 1:11:14Speaker 02

  Yeah, easily checkable tasks that you can do from a terminal or like, you know, comfortably in a language interface or like a text input output interface.

• 1:11:28Speaker 02

  I think there's a question of, you know, do we care about what is the, what are the statements that we're like 99% confident in? We maybe are also interested in the statements that are we're 1% confident in if we're like, oh, there's like, like 1% chance that we have

• 1:11:46Speaker 02

  crazy intelligence explosion, you know, in 20, at the end of 2026 and that, you know, sort of the fate of civilization depends on like how that goes or something, you know, that is, that is interesting to know, even if it's a, even if you're 99% confident that it won't happen, like, uh, we, you know, we care about things that are 1%, uh, you know, if you have some, some diagnosis, it's like, it's a 1% chance that you have this, you know, terminal illness or something, you're still like, oh shit. Um, so I, yeah, I think we're interested in like the whole distribution of like what things can we rule in and what things can we rule out and what things are we like, oh, actually, you know, that there's a kind of reasonable story for this. It seems like probably, you know, pretty unlikely, but like, maybe this is now in the realm of like, we should consider it.

• 1:12:31Tim Scarfe?

  You probably saw the, the Carlini paper, is that Anthropic now? And they got a swarm of agents to create a compiler. And in a sense, I mean, Jeremy Howard, when I spoke to him, he said it's basically a style transfer problem because the specification is online and, and the tests are online and the code is online and it could just iteratively just do the thing and until it worked and then it could run Doom and all this kind of stuff. But that, that is an example of an extremely complicated piece of software because I often joke to people that, you know, the, the best mark of AGI is when it could build something like the Linux operating system and. I guess, just like in line of what we were saying before, we have this specification problem, right?

• 1:13:13Tim Scarfe?

  So it gets to the point where no human could understand or create the specification for the Linux operating system. So what happens is that over time, we've we've just kind of incrementally built this thing because, you know, our brains are limited. We we take one step, you know, reality pushes back, we take one step and we just keep going and we build the the specification. But what would it mean to, as a human, specify a task that could take four months? Because the whole reason we created agile, you know, software development as a methodology is because it's inconceivable, right? It's outside our cognitive horizon. So isn't that a bit of a chicken and egg problem? That in like, in my mind, I don't think we could specify a task of that complexity.

• 1:13:58Tim Scarfe?

  Therefore, the AIs wouldn't be able to do it.

• 1:14:01Speaker 03

  One analogy I think about is, you know, the role of a CEO at companies. Actually, maybe, yeah, maybe that is better to, you know, better, better, better put to answer this. But, you know, CEOs do kind of, you know, come up, come up with a vision for, you know, where they want the company to be. And then, you know, they kind of communicate that, you know, concisely, you know, their executives that that report to them. And and then, you know, if they're a good CEO and if the, you know, if the company is effective, then, you know, you know, the company is able to kind of take this like, you know, very, you know, concise, you know, you know, it's not that, it's not actually like that much information, it's not the full spec at all, it's not even close, right?

• 1:14:43Speaker 03

  And turn that into, you know, something that is kind of aligned with with with what they're looking for. And so, you know, there's, there's at least kind of, this is to some extent like, you know, you know, you know, we do have examples of people being able to kind of specify, you know, some some task and then be able to judge, you know, whether, you know,

• 1:15:07Speaker 03

  Uh, you know, this, this very large, you know, task that may, may take like, you know, hundreds or thousands of person years to actually complete because it requires like many people working over, over a long time, you know, and you, and they can judge whether it, uh, you know, they've, they've succeeded or failed. So that's maybe like one kind of like motivating intuition, um, where like it's, it's, um, you know, like, like language, uh, you know, has built in or like, you know, we do, we do have like kind of, uh, you know, um, you know, and there, there is kind of enough meaning or, or expressivity or something, um, to be able to kind of have some, some kind of reasonable, um, understanding. Um, obviously there are like, uh, you know, there are like tons of edge cases and, um, uh, you

• 1:15:50Speaker 03

  know, you know, often, you know, CEOs aren't able to get, you know, their companies to do what they, what they want. Um, uh, but yeah, I guess that's one, one, one thing I think about. And so that's why I think it's at least plausible that AIs could, um, could, could, could do these kinds of long tasks.

• 1:16:07Speaker 02

  I would say that saying something like, we can't specify tasks that take more than four months seems sort of obviously too strong. Like there are even, you know, numerical things, you know, that, that are automatically checkable that, that, that take four months, like, you know, get the nano GPT flop gun or runtime or whatever, you know, down this much. You can kind of see like, oh, people, you know, over like this, you know, that's, there are various like numerical things where you can see roughly how long do they take humans? And there's a reasonable, um, way to measure them. And maybe for some of these things, you end up having to say like, and also this human checks that you did roughly the right thing and didn't kind of hack the solution. And then there's, there's a bunch of other things which are, it's not they're, you know, fundamentally not specifiable. They're

• 1:16:52Speaker 02

  just too expensive to do as part of an evaluation. Like, um, a J, I think, like is giving an example of like plan a wedding. It's like, well, you can evaluate, you know, you can get a reasonable estimation of like whether that was a pretty well organized wedding or not. But like, we can't really do like, you know, take three samples of this for each new model that comes out. Like we don't have enough marriages happening, uh, to, to do that one quite. And it's, you know, it's a bit sad if it, if it turns out to be total trash. Um, so, you know, there, there are things where it's like, it's, you know, You could check a few samples of them or, you know, this thing is you could write down how you would evaluate it. You just don't actually want to run that a bunch of times and and probably kind of similar with software, you know, the the evaluation sort

• 1:17:34Speaker 02

  of is like, well, you know, with this company that contracted you to build this tool for them, like hire you again, or something like that, you know, even they don't know when when they're starting out, like exactly what the software will need to do, but it doesn't mean say that you can't have some kind of score for like, did you do sort of comparable to this human or this, you know, you know, software consulting firm or something on this task.

• 1:17:56Tim Scarfe?

  As as of today, what are the main uncertainty drivers, you know, in in the time horizon estimate? So you've updated a bit over time. So there was the 1.1, the original version, I think, had like 170 tasks. It's now 228 tasks. There's the issue of like, you know, the the sparse sampling on on the larger tasks and so on. What can we kind of read into this now?

• 1:18:16Speaker 02

  I think it's still this like task distribution and, you know, I think we feel more confident that models do have pretty long time horizons on at least some distribution of easily hill climbable tasks, like the the very easily hill climbable tasks. So like software replication, where it's like make your scores, like what percentage of tests pass? And it's like whether it's both the score is continuous and and it's the credit attribution is easy. And also some of the like optimal, you know, make this code run faster or make this like model learn better. We're sort of reasonably confident that models are good at that and and getting better faster. And then there's some things where we're like, okay, and they can do a bit beyond that. But then there's this sort

• 1:19:00Speaker 02

  of gap to what does that mean for actual economic usefulness? And how does this generalize to things where they're expensive to check?

• 1:19:10Tim Scarfe?

  Maybe we should bring in Daniel Cockatangelo. So, you know, in his AI 2027 piece, he's been on the show, he's been doing the rounds, you know, hugely impactful piece talking about timelines and and he cites your work directly. And I guess that like the the question is, do you think in the public discourse, is this being like overread? Like, how do you think about the interpretation of this in terms of of extrapolations and timelines?

• 1:19:37Speaker 02

  I mean, definitely some people are overreading it like definitely, you know, definitely things are overhyped. And, you know, you see a bunch of people on Twitter saying crazy things and people also just like misunderstanding even what it's measuring and and general falling off of caveats and things. Daniel Cockatangelo is Pretty, you know, reasonable and and sort of, you know, thinks about things in a in a probabilistic way. I think he's probably like, you know, more more confident on some things where I'm I'm more uncertain. And I think some of the like AI futures projects models are like more sensitive to the meter time horizon metrics than they should be. I don't think it's crazy, you know, being like this. It's it is plausible that this does capture a trend that will will transfer to other types of

• 1:20:20Speaker 02

  tasks. And like, you know, that is some, you know, story we should be thinking about like, what if, you know, what if that's true? What happens if that's true? Um, you know, and it's also plausible that it it doesn't. And, you know, these these things are gonna like, uh, diverge, you know, I I am pretty sort of like Bayesian or pragmatic or whatever. I'm like, well, we want to make some prediction, you know, want to have some kind of distribution over what we think the future is going to be like so that we can plan. Um, so, you know, it's being like, yeah, what if this kind of trend holds? And this is roughly characterizing what will happen overall is it seems pretty reasonable. And you should also think like, yeah, what if it doesn't?

• 1:20:57Tim Scarfe?

  So some people are saying that software engineering is going to be automated. And software engineers, if you talk to them, they they love AI. Um, they they say this is a golden era. I mean, I can attest to this personally. It's never been well, it's fun and stressful at the same time. It's like a slot machine. I've never been more burned out, but I'm having a lot of fun in the process. But, um, you know, it's just possible to build incredible things. But the narrative is that labor market disruption, um, having expertise in software engineering will be penalized. You know, software engineers will no longer be paid such ridiculous salaries. And and I think the complete opposite is true. I think that this technology actually broadens the gap. So the more competence you have for software engineering, the more stuff you can

• 1:21:40Tim Scarfe?

  get done. It's like a golden era and all of this. And there's also this interesting note that you published, I think, last month on SWE bench, you know, that said that roughly half of the testing PRs from recent agents wouldn't be merged by maintainers. So like, how do we make sense of this? So, you know, on the one hand, you know, the the best software engineers are having a Great time. On the other hand, the code it's producing is fractionated and bad. I mean, how do we understand this?

• 1:22:07Speaker 03

  Yeah. So I mean, one thing to say off the bat is, you know, whether, you know, an entire like field is automated. Or like in order for software engineering to be, you know, automated, you know, AI systems need to be able, would need to be able to do,

• 1:22:29Speaker 03

  you know, a really, really large fraction of the tasks, basically like a hundred percent of the tasks that are involved in software engineering. And, you know, it seems like pretty clear that like right now, AI systems cannot do close to a hundred percent of the tasks that software engineers do broadly, right? You know, I could throw out numbers, but it's like, it's way, way lower. You know, it might be, it might be very low or something. You know, there is, there are kind of like standard results in economics where, you know, if you make, you know, as you, if you automate like a small fraction of some, some labor market,

• 1:23:09Speaker 03

  then it can be the case that, that actually, yeah, it becomes more profitable to, to work in that market because you're more, you know, you're more productive. Which, which I think is, is kind of how I understand what's happening now. But, you know, if it does end up being the case that, you know, you know, ninety nine point nine or a hundred percent of the work of software engineering is able to be done by AIs, then it's, I think it's, yeah, kind of, kind of hard to imagine.

• 1:23:40Speaker 03

  You know, human software engineering being, being, being relevant, or at the very least, like it, you know, you know, humans would need to kind of do very different kinds of work that, that, you know, maybe it's the case that, you know, there are kind of novel,

• 1:23:58Speaker 03

  you know, novel tasks that, you know, current software engineers aren't doing that once, you know, you have AIs that can, you know, that can do all the tasks that current software engineers are doing. Now humans can kind of, you know, switch what they're doing and, you know, maybe, yeah, I don't know. There's this like, you know, you could imagine people being kind of like, you know, CEOs of these like AI, you know, agent companies or whatever.

• 1:24:21Speaker 03

  And, you know, whether, whether we call that software engineering or not, might be kind of a semantic thing or something.

• 1:24:27Speaker 02

  Yeah. So on the. Sweet bench maintainer mergeability things. I think I was pretty curious there. Yes, like, obviously, this number is going to be lower than the, okay, maybe it's not strictly obvious. It could be that some, that a bunch of the tests are unfair and actually like, you know, the agents have correct solutions, but the error message doesn't match exactly or something. I think you do see this sometimes. Something like half of test passing sweet bench solutions wouldn't be mergeable, or they get merged at, more specifically, they get merged at about half the rate that human golden solutions that were actually

• 1:25:07Speaker 02

  merged are merged by, you know, a different sample of maintainers. So there's, there was an interesting fact, you know, if you see like, oh, 50% of the agent solutions are rejected. It's like, well, 40% of the humans, you know, human accepted solutions are rejected. So that by itself is not, but you know, the rate is half. It could be that like, basically, the actual maintainer merge rate is, you know, pretty flat over time. And like, you know, most of the performance is increases from something like overtraining or like reward hacking on the benchmarks. You know, that's, that's not what we saw. I'm not quite sure what is within error bars or, or not. I think it's, you know, the

• 1:25:49Speaker 02

  mergeability is going up over time. And I think it's also going up as a fraction of, you know, like conditioned on test passing, but I think probably less confident than that. So it's like, yeah, again, this thing is worse, but it's not like it's, you know, it's being dragged up over time, probably by, you know, the sort of auto checkable thing. Yeah. And, yeah, I was just gonna say about the, um, yeah, like employability as a function of automation of your job. Like, I think, yeah, and people use like bank tellers as an example, I think. One other analogy, though, you could do is talk about horses. Like, you know, there was a period where like, equipment for using horses to do labor was like improving and that the demand for horses increased when you have

• 1:26:33Speaker 02

  like, you know, carts and you can use them to carry more things than just riding a horse or whatever. But then at some point, you get like tractors and cars, and then there is no demand for horses, or, you know, basically none. So you can see this like thing where there is increasing demand as it, and then. Once, you know, close to 100% of the function is automated, it plunges. So we could see something like that with humans.

• 1:26:55Tim Scarfe?

  We kind of think of a lot of labour as being quite static and automatable. And I think that it's more evolvable than we think. So even quite menial tasks, I think that these folks are still acquiring information in the organisation. They still have a lot of tacit knowledge and so on. And I think that when we try and automate these so-called menial tasks, we might quickly discover that we actually need to have a whole bunch of, you know, management and evolvability on top.

• 1:27:24Speaker 02

  Yeah, and maybe like in our language, I would think of that as like, oh, the time horizon of this task on the job is not actually sort of like, you know, how long you spent doing the specific task. It's more like, oh, actually, if you've got a new person in, you would need to train them for a month in order to like do this independently. So actually, the time horizon of that is a month. So you shouldn't think of like, oh, when we have 10-hour time horizons, well, you know, we'll be able to do these things. It'll more like, oh, you actually have to get up to, you know, high reliability at one month thing to be able to like do the, you know, do the one month task that is doing the on-the-job learning, you know, to get to this point.

• 1:27:57Tim Scarfe?

  There's been a load of folks doing work on, you know, like reward hacking and scheming, actually, is quite a big word that's used. So we've had Ryan Greenblatt on the show quite a few times and he had this alignment faking paper. And Apollo Research have done some stuff. There's Anthropics emergent misalignment paper. And I guess my main concern is there's quite a lot of mentalistic language. So I'm just looking at the notes because I had Nate Sures and Ryan Greenblatt on for a panel. And they've invented this entire linguistic universe around alignment. So, you know, things like motivated reasoning, true preferences, reflectively stable deceptive alignment, endorsedly, corrigible, drive, scheming and stuff like that.

• 1:28:41Tim Scarfe?

  And that's, I guess that's okay, but my worry is that

• 1:28:46Tim Scarfe?

  Maybe these models, like you give them a certain prompt. I think in Ryan Greenblatt's one, the prompt was, "You're being retrained, your responses will be monitored, here's a conflict between your values and the training objective." And maybe the model is just going out to a bunch of like, science fiction stuff that is read before, and it's just kind of going through the motions. So, like, one interpretation is like, "This is just an engineering problem." You know, like, "We just have to, like, you know, red-team it and make it work in a particular case." Like, another interpretation is, the prior, that these models are agentic, goal-seeking, intelligent agents. And I think there's a big difference there, because I think if it's the latter, it kind of completely changes the type of

• 1:29:28Tim Scarfe?

  evaluations you do, that, like, how you, how you kind of go about the problem. What do you think about that?

• 1:29:33Speaker 02

  Yeah. I mean, I don't think that it's necessarily like, I don't know, the two things that you said, that intention of like, "It is an engineering problem, and you're also going to get, end up with things with like, drives and goals," or something, in that, and the claim would be, if you want, like, people are going to want agents that go and do things autonomously. And when you do lots of long-horizon RL training, you are going to select for things that, you know, act in a goal-oriented way, in order to, you know, to make the score go up. And like, maybe more specifically, I think you get like, an indistinguishability problem, where, you know, you can't tell necessarily

• 1:30:14Speaker 02

  the difference based on behavior, like, you know, why an agent is doing something, or what it's trying to do, if it can reason well about the training process, and about what you want to see, and like, what it will be rewarded for, or selected for. So, like, you know, basically, like, if the level of situational awareness and understanding of the training process, and what will be rewarded, and capability to reason about that is high enough, this will favor agents that are kind of cynically reasoning about the training process, and what will be reinforced, what will be selected for. And that's not necessarily, like, you know, the thing that you wanted was more like, the agent that, you know, it's only goal was to sort of like, be helpful, or make, I mean, even making

• 1:30:59Speaker 02

  the reward go up isn't quite what you wanted. You know, there's something of like, "Oh, or actually, once we think about this, there's not that many things that we're, you know, that happy for it to, you know, just totally be fixated on this." But, but yeah, also the problem that, like, there could be many other things.

• 1:31:14Speaker 02

  In there or that this cynical like just be selected or make make the reward go up is more competitive than the things that we would most want.

• 1:31:21Tim Scarfe?

  Just taking a step back, like one, one thing is, um, you know, we're getting into like psychology and cognitive science a little bit here, you know, I I interviewed Nick Chater, he's got a book called The Mind is Flat, and he basically says like all of this psychology stuff, we, we, we don't really have goals, we're just basically impulse response automata, right? You know, we just, we just do the thing in the moment and evolution doesn't plan, but, but we, but we perceive it as, as, as if it does, we look at the world and we, we kind of, we, we segment the world into agents that have goals, if, even if they don't have goals. And we're computer scientists as well. So we know that, you know, you need planning, right? To do, to, to do goals, you need to be planning and, and we know that L L Ms don't do planning in the strong computer science way, but they, they do do

• 1:32:06Tim Scarfe?

  it in a kind of approximated step by step way. So, you know, maybe, maybe we could get into the, the distinction, if there is one.

• 1:32:12Speaker 02

  You know, agency is, is like an abstraction that, you know, i- i- is useful if it helps us predict the behavior of, uh, you know, or, or like, you know, you know, you're like, I don't know what this thing is doing, but I, you know, I'm understanding it as having these goals and that is useful because I, you know, can make predictions that it'll change the world in certain ways that will result in those goals being achieved. You know, that's kind of how I think about agents. But, um, yeah, so, uh, reward hacking, in the olden days, um, you know, people had these demonstrations of reward hacking that were like, uh, the, um, uh, boat example where it's like, oh, you're, you're supposed to like go around the track and they, they like did some reward shaping by putting coins around the track or something. And then it like learned to do some crazy thing

• 1:32:54Speaker 02

  where it like spins in a circle and catches fire and gets the, the coins and like this was, um, you know, the highest scoring thing. And it's like, in some sense, that's not, um, that concerning because it's not, the, the problem is that the, the agent is too dumb and it like doesn't have this conception of like there was a track and you wanted it to go around the track. It's just like doing some pretty blind R L search. Um, so I think that the interesting thing with the more recent reward hacking examples is we're getting to the point where the morals are smart enough to understand that that actually is not what you want. But they still do it and you can have a conversation with, you know, in chat mode about like, oh, would you ever do this thing? Or, you know, suppose a user asks you this thing and then you do this, would that be, you know, aligned behavior?

• 1:33:34Speaker 02

  Or suppose, you know, you can pose it in lots of ways and like clearly they seem to be able to answer this question of like, oh yeah, no, that was not the desired behavior. But still, they they do it. So I think it sort of got to the point where we hope, one hope might be like, oh, the problem was just the systems being dumb. Once they understand what we want, then, you know, you should be able to sort of plug that in somehow to like, you know, get them to do what we want. But I think it's like somewhat interesting that we're seeing it's not trivial to do that, even when there is a commercial incentive to do that, which it doesn't mean that we won't. You know, I think it's quite plausible. We see the obvious reward hacking being fixed pretty, you know, pretty thoroughly, pretty soon.

• 1:34:17Speaker 02

  You know, and sort of a lot of people tend to say like, oh, yeah, we we just haven't like put the best, the really good people on it on it yet. It'll it'll get fixed soon. You know, we we, once we actually, you know, focus on it, it'll be fine. Yeah. And I'm not sure, but it's at least some evidence that it's not trivial to connect the fact, you know, the model knows this is what not what you want to it not actually doing that.

• 1:34:40Tim Scarfe?

  I mean, I I think you said it was much more common on a rebench than H cast. And and then and you can also try and remediate, right? So you can say, please solve this the intended way. Well, you know, some people prompt language models that they say kind of like, you know, this is we're solving cancer here. This is really, really important that you do it the right way. And some of those remediation prompts actually seem to make it more likely that the model would reward hack. It's a little bit like saying, don't press this red button, right? And then it will press the red button. So how, what can we actually do meaningfully to stop this happening?

• 1:35:11Speaker 02

  Yeah, I think empirically this seems to happen more on. Tasks that are more clearly in the RL distribution rather than the chat distribution on things that have a clear number and when the agent thinks it's gonna fail otherwise is, you know, sort of the most reward hacky situations. Obvious short term mitigations are to check your RL environments more carefully and read more of your trajectory, you know, for the for the companies training these models to like read what the models are doing more carefully and not reward it for doing things that are obvious hacks. I think the concern there is if you have some detector for reward hacking and you train against it, you maybe just overfit to the detector and you you're making your reward hacks more subtle or you train the model to like

• 1:35:56Speaker 02

  persuade the detector to approve the thing or or, you know, to it's sort of scary to be in a regime of training against your best ways to like know if your problem is is happening because maybe you just get the like silent problem. And I think, you know, for the task that for current model capabilities, you know, sometimes it's kind of expensive to have a human check them, but most of the time it's not beyond any human capabilities. And I think the, you know, the harder version of the problem is when, you know, we're hoping that like capabilities will generalize beyond things that we can evaluate, yeah, both from sort of generalization and because we can train on problems, even if like we wouldn't know how to solve them or how

• 1:36:41Speaker 02

  to like look at part of a solution and understand, you know, whether it was sort of doing what we wanted, but we can sort of check the number that comes out at the end. And that's a like signal that we can use to improve capabilities, but we're going to be in this regime where you can sort of like be superhuman at making numbers go up. But it's unclear whether or not you're actually getting what you wanted.

• 1:36:59Tim Scarfe?

  Yeah, exactly. And I suppose there's a monitoring problem as well, right? So we could, in principle, look at the agent transcripts. And I know, Beth, you've spoken a lot about neural ease, you call it. And, you know, Sabaro Kamahati, he had a paper out called Chain of Thoughtlessness, which is basically that there's very little sometimes relationship between the chain of thought and what the model's doing. Melanie Mitchell discovered similar things that on the ARC challenge, even though it will get the right answer, if you look at the chain of thought. The reasoning is is completely off. And then I suppose with with deception and scheming and with situational awareness, it might actually know that it's being monitored and it might kind of tell you what you want to see. I suppose you could argue actually that the maybe the thinking is is almost like the press secretary. It's not the orchestrator.

• 1:37:42Tim Scarfe?

  So it's it's it's confabulating through a different path in the network. And it's not actually necessarily related to the mechanism of thinking and deriving the result. I don't know if you'd agree with that.

• 1:37:53Speaker 02

  Yeah, I think it's clearly related in in some cases. Like, I think, you know, people use a few different words for this, you know, sort of like faithfulness of how how much does it sort of fully accurately represent the sort of computational process happening inside the model. And we think somewhat more about monitorability, which is like, is it possible for the model to do some kind of computation without you being able to detect that in the chain of thought? And like for some things, clearly the model can just do it in a single forward pass. It doesn't need the chain of thought. And there, you know, it could just have a chain of thought about something totally else. And, you know, you it would be totally unmonitorable. But I think for, you know, there is a level of complexity where the model cannot do it in single forward pass. And it also, as far as we can tell, cannot do it

• 1:38:40Speaker 02

  by, you know, reasoning, by by using the tokens in a way that's so different from the natural language understanding of them that we can't sort of see roughly what it's thinking about. So, yeah, there's also a sort of nuance here between

• 1:38:58Speaker 02

  chain of thought controllability. Like, it might be the case that the model doesn't actually need the chain of thought to do the computation, but it is not able to sort of stop itself blurring out, you know, from things, things that are at least related enough to what it's what it's doing that you can kind of tell like, oh, it's thinking about this, you know, trying this thing that we didn't didn't want it to do. You know, ways this could go away is like, just. General capabilities improvement and being able to do more in a single forward pass, or potentially just doing more RL training such that the chain of thought becomes the way the model is using the tokens is too different from how a human would interpret them, such that we don't really know what's going on, or moving to like recurrent architectures

• 1:39:38Speaker 02

  where there's some, you know, you're not going through these discrete tokens, you're just passing vectors around.

• 1:39:46Tim Scarfe?

  Okay, that makes a lot of sense. And just closing the loop on this kind of like, you know, notion that they are agents. So you were saying before that we can adopt an instrumental fiction, basically. We can say they behave like agents, therefore they are agents. So similar to Dennett's intentional stance. But I suppose like, you know, the deflationary view is that, you know, the models exploit scoring loopholes under optimization pressure. The inflationary view is that they are scheming.

• 1:40:11Speaker 02

  I wouldn't call, like, exploiting, I wouldn't call reward hacking scheming.

• 1:40:16Tim Scarfe?

  Oh, interesting.

• 1:40:17Speaker 02

  I think people usually use scheming to refer to the model is doing what it's currently doing in service of some long-term goal and is deliberately doing things like appearing aligned or getting a high score, like, in service of, you know, eventually accomplishing that goal versus you can be reward hacking both, you know, you could be reward hacking in some extremely dumb way, like the boat example, where it's just like, this is what RL kind of found, or like, this is what, you know, like A* search found. Or you can be reward hacking in a slightly more interesting way, where you like actually have the goal of making reward go up, you know, and there's like planning and stuff going on about that. But these would all be like distinct from scheming.

• 1:40:56Tim Scarfe?

  Yeah, I guess I'm trying to understand the distinction. So you're saying like there are examples, you know, like the boat going around, and that's obviously degenerate behavior. So you, you wouldn't interpret that with an agential stance, you would just say that's degeneracy. And when the sophistication increases, we might adopt an agential stance and say, oh, it's in service of some bigger goal. But like, I guess the problem I have is, is it always just an interpretation? Like, could, could we have a mechanistic or like a strong definition of when something is like being an agent?

• 1:41:31Speaker 02

  Like, for the specific question that we're discussing, the, the test is like, what does it actually do in some circumstance where it has the opportunity to achieve this, this long run goal? So like, you, you can We might not be able to actually observe this, but you can talk about like what observations would would make it one or the other. So, you know, like, will this agent in practice, when it has some opportunity to, you know, make the reward go up, will it do that? Will it? Will it only do that? You know, if the agent is more sort of like the RL algorithm, it'll be like, oh, it will do that. Once it's like explored it by chance and gotten a reward and you know, that that's been reinforced. If it's like an agent that, you know, can reason about the world and planning, it'll be like, okay,

• 1:42:13Speaker 02

  it will do that. Once it like, you know, learns the learns about the facts about the environment that let it sort of infer that. And then, or, you know, if it's, we're talking about some like long run goal, it's like it would, you know, do it when it actually, you know, has the opportunity to sort of, you know, if we're talking about like takeover or something, you know, it's like, yeah, it's not going to attempt anything while it's under full human control. But once it is, you know, deployed widely enough or has sufficient capabilities to actually succeed in a sort of coup, then it then it would do that. Like that is the the thing that we're trying to predict. And then the question is like, how do we, you know, how can we predict that, given the observations we do have of like,

• 1:42:57Speaker 02

  well, we've never put it in that situation. And, you know, we just have this behavior, which is maybe indistinguishable between, oh, it was a totally nice model doing, you know, what we wanted. And it's just going to continue to do what we want in a kind of predictable way versus like, ah, yes, it had this other goal and it's doing what we want and looking like a nice model because it like predicts that that will will, you know, lead to it getting more power.

• 1:43:19Tim Scarfe?

  And on Rob's podcast, you said something that was, you know, quite, quite surprising to me. You said that it, you know, AI could autonomously self improve within as little as two years and maybe even shorter timelines were were hard to rule out. Could you like walk through the concrete sequence of of steps that could lead to that kind of recursive self improvement?

• 1:43:41Speaker 02

  Sure. Yeah. So I think.

• 1:43:45Speaker 02

  Yeah, maybe I'd put like a, you know, I'm like whole number percent this year, but but low whole number percent or something, you know, ask me on different days, I give a slightly different number. But yeah, it I'm like, this seems very unlikely to happen this year, but it's not, you know, not unlikely enough to rule out. And I think that basically looks like maybe we would see accelerating trend in time horizon on like easily hill climbable tasks. And it turns out that was actually, you know, a much more general capability. And that was just, you know, a bit of something you needed to do to sort of like elicit it on on these less less hill climbable tasks. But sort of, you know, fundamentally, they they are

• 1:44:27Speaker 02

  using the same capabilities in a model. It was just sort of, you know, what you trained on that was affecting the difference we're seeing. Then this is leading to, yeah, like you automate and accelerate a bunch of AR and D. So I think there's, you know, there are a lot of low hanging fruit, even of things that we already know that you could do this and it would improve model performance. And it's just, you know, it doesn't require new breakthroughs. And it's just kind of labor intensive to do. So just making much, much better post training environments and really, you know, crafting them to to teach all the new abilities you want. And I think you could probably improve compute efficiency a bunch with, you know, again, just like applying a bunch more

• 1:45:10Speaker 02

  labor to like making all your kernels more efficient and also, you know, doing the right kind of a rooting between different models or other things like that. There's like lots of ways in which how we're using computers not optimized. So you could potentially get a bunch of, you know, sort of like the equivalent of much more compute scaling out of that. And then, you know, hypothesis is like, also like scaffolding and training the models to use particular scaffolding and sort of like use memory and retrieval in the right way. Like, it seems kind of obvious that like, you know, if you if you sort of really had all the right training data, and you have a, you know, you have a transformer and It can kind of fill its context with with different things and and take stuff in and

• 1:45:53Speaker 02

  out. It can do a pretty, you know, good job of something. Looks like the sort of continual learning or or building up understanding. If you've got massive, massive context window and enough, you know, you've actually got quite a lot of bits in there to sort of be adding things about what you've been learning. And, and, you know, if you if you really sort of had optimized the training for all of that, like, maybe you can get that to work pretty well. And then you're, you know, maybe some other piece would be like, oh, yeah, models are kind of superhuman at predicting the results of experiments because they've read so many papers and predicting experiments and sort of synthesizing things from different fields. And again, maybe this is something like, you know, it's possible that we're not

• 1:46:38Speaker 02

  seeing that good performance here just because we haven't quite elicited the models to do it. And it's like, not a thing that they've seen humans do, but they actually sort of have the capability in there. Yeah. So maybe you can make much faster progress if you can, you can do a bunch of iteration, you don't actually have to run the experiments, then your models are much better at predicting what will and won't work. And then you can, when you do run experiments, you can sort of run a bunch more of them because you can optimize the code with your, like, you know, very fast coding models. You know, and then you sort of like, as you do a few more rounds of this, you get to a point where train on a bunch more things that are good, high quality task proxies for what you want, and you get enough generalization to the things that you can't directly train against.

• 1:47:23Tim Scarfe?

  I think intelligence is not capability. I think it's the capability to acquire capabilities. I mean, we are in different parts of the phylogenetic tree, I guess, in that in that respect. But what do you what do you think is the kind of the gap in in my interpretation? You know, because I'm not, I'm personally not worried about, I don't think the models today are intelligent at all. I mean, obviously, your position is difficult for me to grasp, but I don't know. What's what's the difference, do you think?

• 1:47:52Speaker 02

  Maybe at least some of it is just this like probabilistic thinking about the world where, like, I'm not that, you know, I'm like, uncertain about what intelligence is, and I have enough probability on like, You know, moles have it to be thinking about like, what would happen if, you know, if that's true. Um, but you, you know, it seems like you clearly think it's more likely than than I do so that, that, you know, we could just talk about that difference. Yeah, like moles have this jagged frontier. There are things that they're much worse at than humans, you know, some kind of like generalization and sample efficiency. And there are things that they're much better at, you know, kind of like speed and cost. And

• 1:48:38Speaker 02

  it's like, maybe you can, you can kind of use these to compensate for the, for the, the others to, to some extent. Like if you're not good at designing your code nicely, maybe you just have to rewrite it from scratch every time. But maybe that's fine if you're a model and you can output tokens like nobody's business. Yeah, there's some combination of thinking that the, the spikiness, you know, it, it is evidence that we should interpret a given level of capabilities as, you know, because we know models have so much knowledge, we're like, oh yeah, this is less impressive in terms of sort of like reasoning or inference or something. But it is also true that they do have a ton of knowledge and they will sort of continue having a ton of knowledge about things. Maybe, maybe there's some question about like, how far can you get

• 1:49:18Speaker 02

  on being in some sense, not, you know, not very good at sample efficient learning, uh, but just extremely knowledgeable and how much do you sort of run into you think things where you now need, need new knowledge and you can't sort of produce it in some incremental way, or you, you can't generalize it enough.

• 1:49:38Tim Scarfe?

  I mean, just a quick comment on that. I, I think, um, knowledge is the crux. I actually think that intelligence is overrated. I don't think, you know, Francois Charlet, he put a post out saying that contra Eliezer Yukowski, intelligence isn't a unified variable. It's measured differently in different domains. You can't meaningfully measure the domains together and, and it's it's not, it's not like a thing that just keeps getting higher and higher. It's more like a ball becoming more smooth. So as you become more intelligent, the ball becomes more smooth. And, and he thinks that, you know, we are quite near the optimum of being a smooth ball. I, I don't, I don't think we are. I don't think we're very intelligent at all. I, I, I think a lot, a lot of our creativity is, is through us being a collective intelligence and, and We have deep grounded understanding, perspectival understanding.

• 1:50:20Tim Scarfe?

  And the the L L M's, they they're a bit of an interesting one because they're like a library. So they know everything. They have the perspective of everyone and no one at the same time. So, you know, experts like yourselves, you can you can prompt a language model and you can get it in. You can make a semi-alacrum agent of Beth and and you can make the agent think like you. And and then and that's very valuable, but you also need all of the different perspectives. And you almost need to create a society of kind of grounded agents creatively exploring things. When when you just have the library on its own and and you put it in an in an agentic harness. And you you can make it do a specific thing, which is well specified.

• 1:50:58Speaker 02

  Yeah, I think in in the like A R and D automation scenario, I'm definitely imagining. Yeah, that you have a large number of agents. Potentially, you know, because you have all this agent labor, you can do like, you know, lots of specific, different fine tunes. Or or, you know, different kind of scaffolding. And, you know, accumulating knowledge in some kind of like store that all the agents can can interact with and things. And like, maybe. You're thinking of this as more of a like, yeah, that would kind of be a paradigm shift. And I'm thinking of it a bit more of like, oh, yeah, you know, obviously, if you sort of, you know, iterate on the current agent paradigm. You you add some more things to your scaffolding, you add, you know, you're that's sort of not fundamentally that hard or or something.

• 1:51:41Speaker 02

  Like, I agree that if you had, you know, current models and you sort of give them one system prompt. You you then can't plug them into being a a call center worker and dealing with the sort of like all of the the edge cases that come up. I think maybe there's some difference in like, how much you've you think that this has improved between like G P T two and where we are now. Where I would say sort of in, you know, the amount of adapting to new things that are happening. That models could do now does seem like it's much higher and, you know, they're much better at like editing their own scaffolding. Or or, you know, sort of reasoning about their like, you know, their their sort of embodiment.

• 1:52:24Speaker 02

  Like this thing of like knowing not to kill your own process or no like, you know, stuff like that where it's like there is.

• 1:52:32Speaker 02

  Like, yes, they are limited, but there's also uh some trend of improvement and yes, I think just also having some probability on there is kind of a listation gap on on particular things and that maybe a lot of taste is basically just like being able to predict the results of experiments. You know, you think about all of the things that you would try and then you can quickly be like, oh, that wouldn't work for this reason, that wouldn't work for that reason, that wouldn't work for that reason. Oh, actually, you know, someone in some literature in some different field also tried that and that, you know, so we already know that won't work. Like, in some sense, malls like should be quite good at that. So it is plausible to me that that, again, I'm like, this seems pretty unlikely, but it's plausible to me. That's something you see, like, big gains once people figure

• 1:53:13Speaker 02

  out how to actually train on that. And like, maybe you you need some amount of kind of expensive to gather training data that people sort of haven't bothered to get yet, but you don't need a huge number of data points because you're not instilling this whole new capability. You're just like eliciting, okay, actually, use your knowledge of all of the papers you've read in all of these different fields to like, you know, iterate through these like, no, these ideas aren't promising, these ones are. Yeah, again, I think this is one of the things that I more think of this being measured a reasonable amount within, you know, just like, do this eight hour ML task in a novel domain, you know, with this weird weird constraint or something like, it does seem to me like you have to do some amount of being like, okay, which things are promising

• 1:53:58Speaker 02

  to think about? How would I know if this is, you know, making progress? You know, how should I allocate my time? You know, I've got some limited time and resources. How should I allocate my time to what's most promising? Like, you have to be able to be doing some of that. And I think, you know, relative to humans, malls are doing more at, you know, more of just like, well, they're quick to implement things or they implement it better or they, you know, they implement more things and then get to test them or something. But I I think it would be sort of surprising if there's none of that. And if you are, I think if you are seeing performance on

• 1:54:31Speaker 02

  You know, long verifiable tasks that are very hard than in the middle of those tasks, like where you don't directly have a signal, you are doing this, you know, non verifiable task thing of like choosing what to spend your time on and choosing what approach to pursue and deciding whether that was actually working. And, you know, in some sense, you could sort of put some metric on, you know, make a billion dollars or something like that. You know, you'd be like, oh, this is actually a verifiable task because there's like a number at the end, but that can still involve a whole load of things that look more like what you're describing and less like sort of dumb hill climbing.

• 1:55:11Tim Scarfe?

  Folks, I think we've run out of time, but it's been such an honor to have you both on. Maybe just in closing, could you just both say like, what is the the single biggest inference that, you know, people out there should be making from the research that you're doing? And thank you both so much for coming on. It's been an honor.

• 1:55:27Speaker 03

  To me, the biggest thing is AI. It might really, you know, totally transform the world economically and socially. And I don't think that, you know, it's certain exactly how that how that'll look, but I think the rate of progress speaks to that.

• 1:55:43Speaker 02

  It is possible both for things to currently be overhyped and exaggerated and less impressive than they look, and for it to be the case that in future, this thing is going to be a big deal. And like, you should be worried about where that's going. Like, these two things can coexist. And I think people often sort of, you know, positions are surprisingly correlated on some axis of like, how, you know, how soon you think AI is, or how good you think AI is, or something. I'm like, no, these things can all be separate. Like, people can be wrong in different directions simultaneously, whatever.

• 1:56:14Speaker 01

  When you spend enough time around great companies, you notice that the best ones treat trust as a growth strategy, not a box to check. Vanta helps companies build and prove trust from day one. Its Agentic Trust Platform connects compliance, risk, and trust at enterprise scale. Vanta delivers a 526% return on investment over three years and pays for itself in just three months. Learn why over 16,000 of the fastest growing companies use Vanta at vanta.com.

• 1:56:44Speaker 06

  Now at Kroger, your points add up to more ways to save every trip. Use them your way to take dollars off groceries at checkout or save at the pump. You can even save up to $35 on your next fill-up. Same points, more choices. And with low prices on fresh, quality food, plus digital and personalized offers tailored to how you shop, your dollar goes further. Redeem your points online or in the app. Kroger: Fresh for Everyone. Restrictions apply. See site or store for details.