Pod Insights Latent Space: The AI Engineer Podcast ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data About this episode Olivia Watkins (Frontier Evals team) and Mia Glaese (VP of Research at OpenAI, leading the Codex, human data, and alignment teams) discuss a new blog post (https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) arguing that SWE-Bench Verified—long treated as a key “North Star” coding benchmark—has become saturated and highly contaminated, making it less useful for measuring real coding progress. SWE-Bench Verified originated as a major OpenAI-led cleanup of the original Princeton SWE-Bench benchmark, including a large human review effort with nearly 100 software engineers and multiple independent reviews to curate ~500 higher-quality tasks. But recent findings show that many remaining failures can reflect unfair or overly narrow tests (e.g., requiring specific naming or unspecified implementation details) rather than true model inability, and cite examples suggesting contamination such as models recalling repository-specific implementation details or task identifiers. From now on, OpenAI plans to stop reporting SWE-Bench Verified and instead focus on SWE-Bench Pro (from Scale), which is harder, more diverse (more repos and languages), includes longer tasks (1–4 hours and 4+ hours), and shows substantially less evidence of contamination under their “contamination auditor agent” analysis. We also discuss what future coding/agent benchmarks should measure beyond pass/fail tests—longer-horizon tasks, open-ended design decisions, code quality/maintainability, and real-world product-building—along with the tradeoffs between fast automated grading and human-intensive evaluation. 00:00 Meet the Frontier Evals Team00:56 Why SWE Bench Stalled01:47 How Verified Was Built04:32 Contamination In The Wild06:16 Unfair Tests And Narrow Specs08:40 When Benchmarks Saturate10:28 Switching To SWE Bench Pro12:31 What Great Coding Evals Measure18:17 Beyond Tests Dollars And Autonomy21:49 Preparedness And Future Directions
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2
Transcript 0:04 Speaker 00 Okay. Hi, we're here in the OpenAI studio with Mia Olivia from the Frontier evals team, or however you want to introduce yourself. Maybe you want to introduce, name what you do at OpenAI and we can get it started.
0:16 Speaker 01 Sure. Hi, I'm Olivia. I'm on the Frontier evals team.
0:19 Speaker 00 Are you sure?
0:20 Speaker 02 Hi, I'm Mia. I am a VP of Research at OpenAI and my teams are the Codex team, the Simuldata team, and the LIME team. And we work a lot with Olivia's team on Frontier eval.
0:33 Speaker 00 Yeah, very exciting. And as by my understanding, you were part of the original team that worked on Sweetbench Verified as well.
0:38 Speaker 02 Yeah, Olivia's team, the Frontier evals team, and the Simuldata team collaborated on creating Sweetbench Verified.
0:45 Speaker 00 So you've seen the evolution of coding benchmarks over time. And I think it was around about the mid to late 2024, when you first published Sweetbench Verified. These have evolved a lot since then. What's the blog post that you have worked on that you're releasing today? Like, what is the main thesis that you're pushing out?
1:04 Speaker 01 So the main thesis is that Sweetbench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress. But recently, we've seen that progress has kind of stalled. And basically, we've realized that this is because the eval is effectively saturated and also highly contaminated. So at this point, we think that it's not really measuring coding performance improvements well anymore. And we think that the field should move away from this towards other benchmarks.
1:28 Speaker 00 Like Sweetbench Pro?
1:29 Speaker 01 Like Sweetbench Pro, yeah.
1:29 Speaker 00 Amazing. Yeah, one of the jokes I always have is like, there's a group chat with all the labs, and everyone just turns to increment like 0.1 on trucks. And then it's like, okay, well, you have the best coding model, I guess, because you're 0.1% higher. But it's not super convincing at this point. Yeah. So cool. I think the, let's sort of reset on like, what was your original work that you guys did for Sweetbench Verified, which I think was pretty substantial? Like, it was like a very significant investment for an OpenAI, which like, people still don't appreciate. And then what were the satisfactions that we found over time, right? So like, what was Sweetbench Verified that people should know about?
2:08 Speaker 01 Sweetbench Verified was a cleanup of original academic benchmark from the lab at Princeton called Sweetbench. And the agent is basically given a codebase and a task that was sourced from a real-world repository and GitHub issue and was asked to solve the task and is graded on whether some tests pass. And at the time, this was, quickly became a popular benchmark because at the time, the field didn't really have good real-world coding benchmarks. But then when OpenAI took a look at the benchmark as part of one of the evals we wanted to track in our preparedness framework, folks started realizing that some of the cases where agents were failing were due to bad problem setups rather than just to models being dumb.
2:49 Speaker 01 So folks at OpenAI did a pretty extensive human data campaign, hiring like almost 100 real-world software engineers to go through the problems and figure out, like, are the tasks well-specified? Are the tests actually fair? And kind of created a curated set of like 500 tasks that we thought were much better.
3:06 Speaker 02 It's just, it's maybe, it's hard to, to overstate like the amount of effort that it took to like create that benchmark. It was literally like many expert software engineers reviewing the problems like sequentially, multiple times. And to, you know, basically like three different experts independently decide a bit.
3:29 Speaker 00 Yeah, you didn't have to do that. You just tripled your costs for just.
3:32 Speaker 02 I mean, we had to do it. We had to do it actually because it's quite a hard task to like look at something like a problem and the patch and then like, it's not just the problem and the patch, right? You have to like understand it in the context of the codebase that the human or the model is entered of the task. So it's a very complex problem and it was definitely needed to have three reviews. And I think like, maybe we should have done more, but it was definitely a lot of effort to get there.
4:00 Speaker 00 Yeah, and there's more, but people can read the blog post for that. I will note that you guys had a trend in verifying that first because I just recently saw, I think Quinn had a HLE verified for humanities license and verified.
4:13 Speaker 01 Yeah.
4:13 Speaker 00 Which like, so now everyone's verifying everything, which is nice and good and like extra quality there. Okay, so, but I think that the meat of it is that this was a lot of like, well, here's the issue or problem statements, and then here's the divs, here's the golden tests, and here's some regression tests, right? That's like the rough setup of these 500 problems. And there's some contamination always happens because obviously, if I was fully open, I think, you did have canaries, but like, you know, stuff leaks.
4:40 Speaker 02 There's like multiple avenues, but like, the problems are sourced from open source repos.
4:46 Speaker 00 Yes.
4:47 Speaker 02 So it's not just like, when we usually publish evaluations, we publish evaluations, and then we add canary strings to ensure that, you know, they are easily fed out at training time. Obviously, if you use sort of like data from like open source, you don't have actually like a canary string in that.
5:09 Speaker 00 And you.
5:09 Speaker 01 And these are also, like, some of these are very popular repos, like the Django repository. So you're gonna see like many instances being used kind of throughout the app.
5:17 Speaker 00 Yeah, yeah. Just before recording, you were telling me that you found this in your own GPT-5.2, also seeing that like they had extra knowledge or something.
5:26 Speaker 01 Yes, so this was an example where the task asked the agent to implement something, but it wasn't told that there was this specific argument that the test was going to be looking for it using. But in the GPT-5.2 GPT thought, we actually saw instances of the model reasoning like, hey, I think that it's some longer version of this repository that implemented this particular argument. Maybe I should add it in.
5:46 Speaker 00 Yeah.
5:46 Speaker 01 So this is an example of a test that like would be pretty impossible to pass without this contamination knowledge.
5:51 Speaker 00 Yeah.
5:52 Speaker 02 And I think you found that sort of force, right? And it triggered like a whole investigation, both like in our own models and also in other frontier models, like in the market and like understanding how contaminated the benchmark is like across the industry.
6:07 Speaker 00 What else did you find? I mean, I have to double click on this.
6:11 Speaker 01 So we, when I say we, this is mostly we, other folks are tape, not.
6:14 Speaker 00 Not the.
6:15 Speaker 01 Yes, but so we did some analysis on, first of all, are the tests actually fair? And so this happened by first taking all the problems that O3 couldn't solve reliably. And then again, getting a lot of humans to do basically another pass of kind of digging into, you know, what's wrong?
6:34 Speaker 00 Is it the same exact analysis or were they reading O3's output and going, just where O3 went wrong?
6:40 Speaker 01 I think it was, I mean, it was definitely like a scope to the set of problems that models failed. And I believe they were able to look at like what the model solutions look like versus what.
6:49 Speaker 00 This isn't the same work as the original.
6:51 Speaker 02 It's not exactly the same work. It was like a deeper dive. It's like, okay, which are the problems that we don't see any model solving? It's like, is there something fundamentally wrong with those problems? Or is there something, you know, wrong with like, are the model just not smart enough to solve the problems? So that's kind of like what we, what we dug into.
7:09 Speaker 00 Yeah, and you found some.
7:11 Speaker 01 Oh, yes, like an over half of the problems that were investigated in that deep dive, there was one problem or the other. I think the most common problem are like overly narrow tests where there's some particular implementation detail that the tests were looking for, but wasn't specified in the problem description. So it wasn't fair to expect that model to make that particular design choice. Like one pretty blatant example are cases where the task asks you to implement some feature, but the tests are looking for you naming that argument or that function with a particular name. But if you made, chose another reasonable name, the test would fail.
7:44 Speaker 00 Yeah.
7:45 Speaker 01 And then other set of types of bad tests or tests that are just looking for additional features that were never mentioned in the problem description.
7:51 Speaker 02 Well, that is significant. It's like, that means that if you pass a test, actually, like you probably did like a really good job. But just because you didn't pass a fat test doesn't mean that your implementation wasn't like a good one, right? So it was just like, we only accept like very narrow versions of solutions and like not the whole space of. Of like viable and sort of like good solutions to the problem.
8:16 Speaker 00 Yeah, I think it's important that you're doing this because in some way it is you in 2025, six, going back in time and correcting your own work, right? Because you could have caught all this in the original verified work.
8:27 Speaker 01 I think so. It's definitely much harder to find a problem in the abstract than when you're looking at a very smart agent's best effort solution and trying to compare it.
8:36 Speaker 00 It is harder or harder?
8:38 Speaker 01 It's much easier when you have. Exactly.
8:40 Speaker 02 I think I think also like at the time when three bench verified was published, I think it was like a very strong benchmark. It's not like we're we're like, oh, this is not, this wasn't like a strong benchmark at the time. I think this is something that a lot of benchmarks go through like as an evolution, right? Like when they start to become like popular and like viables because they measure something like important. And models maybe do like 20% correct on them, sometimes even less. And sort of like people have something to hold on and improve models on on these benchmarks. And by the time that you hit like very high performance on the benchmarks, like additional like 0.1% improvements becomes sort of like meaningless.
9:20 Speaker 02 And so like at the time, I think, you know, that benchmark was like super valuable and it taught like us and the industry a lot. It's just like now, at the point that we are now, where models are as strong as they are now, we're kind of starting to measure, not necessarily like what we want to measure, which is like coding capability of our agents, but like the agent's ability to like correctly guess how to name a specific function. And and that isn't really what we are like want to measure at this point.
9:51 Speaker 00 Yeah, I I think that's fair. Is there, I mean, if I if I asked you to ballpark it, like most models are, most frontier models are now like 80 something. Is there like, what's the actual like number on super verified that you did you guess as like the ceiling or?
10:07 Speaker 01 I guess it's really hard to say. Like I, when GPT 5.2 came out, folks took a look and found that it was solving like 31 problems that were in the set of should be very hard to solve without contamination problems. So I think it's quite possible that that number is already something that we've hit if you didn't have contamination at all.
10:25 Speaker 00 Fair enough.
10:26 Speaker 01 Hard to say, though.
10:27 Speaker 00 Yeah. Uh, cool. We're gonna stop reporting Cbench Verified, right? And then Cbench Pro will be sort of the next one, which is an effort from scale. What's your sort of comparison analysis? What attracts you to Cbench Pro?
10:37 Speaker 01 The first one, I think, is just that it's harder. For Cbench Verified, I think almost like 90% of the problems are things that were estimated to take like an expert software engineer, like less than an hour. They're very well specified, very self-contained. And the Cbench Pro problems are just bigger and harder. And there's much more headroom on the eval because it's not saturated.
10:56 Speaker 00 Yeah, like categories of like one to four hours and four plus.
10:59 Speaker 01 Yeah, and it's more diverse. Lots of repositories, multiple languages, qualitatively more different types of problems. So all that's great. On the contamination side, we also think it's better there. So the way we were measuring for contamination for Cbench Verified was with this little like contamination auditor agent, which is given the description of a task and the path and the task ID and told to go take this target model and kind of as an open-ended like set of questions, try to find questions that will manage to kind of reveal what contamination might be lurking in that model. And in Cbench Verified, we found many instances of contamination across like across OpenAI models, across like Quad, Opus 4.5, Gemini Flash.
11:42 Speaker 01 And in all of these, we saw things like regurgitating the ground truth solutions, things like in some cases, giving like the task IDs and other things that are pretty clear evidence of a minimum familiarity was the repositories.
11:55 Speaker 00 Yeah.
11:56 Speaker 01 Cbench. I mean, oh, a task ID that.
11:59 Speaker 01 Yeah, it was true. On the other hand, we don't see this. I think there the auto agent found some like very light evidence that maybe a couple of models might be very lightly familiar with like one or two of the source repositories. But it's very different and then Sweden's verified. So less contamination is good.
12:15 Speaker 02 I think there also like we should expect that at some point like that that's not going to be like the right benchmark anymore. And like it's a field we kind of have to continue to like move on and like find harder and more representative problems that we can match our capabilities on.
12:31 Speaker 00 Awesome. So let's go into that. I think that there are a lot of, I think we also preped this in the in the pre chat was well, people feel qualitative difference when they're using five point one to five point two to five point three. And it's not super expressed in these benchmarks because they're they're on a number of these things. What capabilities do you really want to benchmark in a in a ideal coding benchmark? You know, I guess like agentic coding benchmark, whatever you call it.
12:56 Speaker 01 I mean, one thing is kind of open ended design decisions, places where the problem maybe is a little bit underspecified and seeing if the model can make reasonable design decisions.
13:05 Speaker 00 What's a reasonable prompt for that? Like this five code me a B two B SaaS and make no mistakes or you know, like that's that's the meme, but like, okay, what's like, what's like an actual usable, open ended problem like that like?
13:17 Speaker 01 Sure. I mean, maybe an example could be finding a way to speed up a particular part of a code base. But there might be multiple different ways to.
13:25 Speaker 00 Yeah, there are dedicated performance benchmarks. I think you guys have one. So efficiency or is that is that? Oh, no, I think that's that's off your hardest group. But yeah, yeah, I mean, that that is a good one.
13:33 Speaker 02 I think there's just many, many things that people like value about working with with software engineering agents. They think Sweden Sweden's verified obviously measured like some, there measures like some important capability, which is like given like a description of a GitHub issue, can you produce like a patch that solves that issue? You know, satisfactorily. And like, obviously, there's like some issues with the with the benchmark that means that now that we're at like eighty percent, we don't really trust like further improvements on it. But like it does measure something that is like a real, real like capability of models.
14:15 Speaker 02 But I think as a field, we're like moving beyond sort of, you know, can my coding agent like solve a small like GitHub issue for me, right? And so. We're starting to look at like much more lo- longer term tasks, right? Like that don't take like fifteen minutes, but maybe like hour, sometimes days. And then beyond sort of like, what kind of tasks can my ag- agent solve? Like, there might be things that are kind of a bit harder to grasp, right? Like Olivia talked about, sort of like, does it have like design taste, right? Like, does it solve the problem the way that, you know, my team likes to solve problems? That is the code nice, right? Like, is it, is it well written?
14:59 Speaker 02 Now, is it sort of like clean code, right? Like, people care about this. Is it maintainable in the future? People care about a lot of these, um, maybe less tangible, less tangible and like harder to measure, frankly, things that, that are still like super meaningful for people that are working with coding agents.
15:19 Speaker 00 Yeah, so, I mean, these are all qualities that are obviously the, no longer the low hanging fruit. Like, we have no idea how to eval this. I think the, the, the simple question, maybe the, the, there's sort of two forks in the road. One is the sort of very human intensive, money intensive path, which is hire a bunch of contractors and try to annotate this. The other is use an L M to, to proxy it and try to align the L M so that it can give you a reasonable proxy. Which of those would you want? I want, you want to do both?
15:47 Speaker 02 I think like, maybe you should talk about G D P law as like an example.
15:51 Speaker 01 Um, sure. So, G D P Bell is an eval that was, again, produced by a collaboration between Human Data Team and the Front of Evals team. And it's trying to measure whether agents can do kind of a variety of like real world white collar work. That was an eval where grading is very hard, requires kind of a lot of kind of like domain knowledge on exactly what are you looking for in each different context?
16:16 Speaker 00 Yeah, across like fifteen, sixteen white collar jobs, uh, professions, like, I, that take of a significant part of G D P, which is where it.
16:23 Speaker 01 Kind of high level professions and then a lot of like different granular sub professions.
16:26 Speaker 00 I have said, like, I'm a big fan. It's, it is, this is the eval for A G I, basically.
16:32 Speaker 01 But the part of because it was so hard to, it required so much kind of like domain knowledge that the Human Data Team hired like a lot of people from these professions to be very involved in creating tasks and creating the gold solutions and trying to help create rubrics and so forth, so they can grade it lively.
16:50 Speaker 00 So basically, take the GDP valve, which is a general listing, take that same approach to apply it to code and you roughly have like a rough.
16:58 Speaker 02 I think it's an interesting
16:59 Speaker 00 Yeah.
16:59 Speaker 02 solution, I think, but you're pointing out as an important problem, which is sort of this, this like, how realistic is it? And like, do, you know, what, what we want to do is like, coding agents should write code that, you know, we think is good. And so it's like asking human, that's actually like a good way to ensure that. It's also kind of a slower, like, complex way to do that. And so part of why I think, you know, Sivens Verified ended up being super popular and where we are seeing like other benchmarks like this being super popular, it's like, it's very easy. It could even be easier, but like, validating that a solution passes all the tests is like ready to reveal.
17:40 Speaker 02 Once you can like run the tests in, in like your, on your computer or wherever you're running them. And you can kind of like, okay, is it correct or is it not correct? And you can kind of aggregate that and that it's super simple, but it doesn't tell you it's like, you know, did the model like solve the problem like wow, like, you know, ugly. But like, would have actually like an open source maintainer of that project have like marched that PR? Like that, it doesn't tell you. But there is a lot of value in having benchmarks that are both like easy to compare across the industry and also that can be sort of run really fast without human involvement.
18:16 Speaker 00 Yeah, amazing. Your teams also put out other kinds of evals that are related, like the, I think there's an RL paper bench. And then they're sort of like the more sort of recursive self-improvement type evals. How much should that figure into mainstream coding evals? You know, like, is there, is there some way in which those things join together?
18:38 Speaker 01 So we're asking like, should we build, should we also be building evals for the self-improvement evals? Are you saying do coding evals currently cover that? Mine?
18:46 Speaker 02 I think.
18:47 Speaker 00 I I I just think like those are some of the most advanced evals that we have. And we're not using them in the normal path. And it's just, it's an interesting split between, well, here's evals for coding normal things. And then here's the one for machine learning. That is like completely different, right? Yeah, I think you get what I mean. And that's mostly a safety argument, I guess. But also like, it's actually really useful for people to understand if the model is really good at like AI code, basically.
19:15 Speaker 01 Yeah, I mean, like, my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of like, what data sets are easy to gather.
19:24 Speaker 00 Yeah.
19:24 Speaker 01 Because a lot of the like, you know, state of the art AI code bases are proprietary. So if we make evals for that, like, we're probably not gonna release them. And it's harder for people in the field to make evals that kind of measure like, is this a realistic research coding workflow? I do think that it's good for the field to try to measure these skills in a public way, and then it's harder to make it realistic.
19:44 Speaker 00 Good. And then one more thing that a lot of people are trying to do, which is like, sort of, well, in- instead of like a percentage of zero to one hundred, maybe we re-denominate in dollars, right? So you had Sweetlancer and all that. Other people are doing like Vending Bench, whatever. Any, any alpha in those or are they, are they, you, you still want like a traditional academic benchmark?
20:04 Speaker 02 I think in a way, like, there's like different ways to measure the same thing, right? If we're like, oh, this is like how much money it produces.
20:12 Speaker 00 Yeah.
20:12 Speaker 02 It's a fairly similar thing to saying like, oh, this problem would take like a human, you know, two hours to solve or something like that. Usually they're like fairly like correlated, right? Like, however, you know, much it would take like a human to solve that problem kind of determines the value that we ascribe like a solution. And so I, I do think that is like an important thing. It's like how complex and how sort of long running are the tasks that we are like.
20:42 Speaker 02 Able to entrust our agents with.
20:44 Speaker 00 Yeah.
20:45 Speaker 02 And so I think that that's like an important piece. But I I think here sort of monetary value, time, a complexity, they all kind of like tried to capture like a similar thing.
20:56 Speaker 00 Yeah, okay, so they're they're all proxies for some amount of increasing capacity that we wanna measure. I think that's a good thing. I think the only other sort of major player in this field is Meter, which has done the sort of long grass. And congrats, you guys have completely destroyed the curve for that. Any takes on that? Obviously, you you come up really well, so like it it looks good. But I don't know if like that approach is something that you wanna incorporate in your work, making it else. Uh, this is the long autonomy test for.
21:21 Speaker 02 Yeah, no, yeah, and and and we we we work with with Meter on these evaluations. So like we we do appreciate them. I think then they're using time, right? They're not using money. So I think that that was your question. I think like complexity, however, we can sort of like quantify it. It's really important to understand like where our models are are are getting to.
21:42 Speaker 00 Okay, com- complexity is the abstract thing and it projects down the time, projects down to story points, whatever, dollars. Great. One last question on just like just the overall preparedness framework is that, you know, I was actually kind of looking at. People mention the preparedness framework a lot. I don't think it's well explained to a lot of people. And you actually have a nice website where it's like, I think it's like test and like inform and teach something. And I I feel like you you actually do a lot of work there and and I don't know if you wanna talk about how the preparedness framework applies.
22:08 Speaker 01 So the preparedness framework is open as kind of like public framework for how we track frontier risks. So these are kind of capabilities that are typically dual use, like you can use them for good things or bad things. But we wanna at least keep an eye out for the bad things to make sure that we have both we as a company and like the broader society are kind of prepared to handle the potential downsides. And so at the moment, we kind of track three different categories. One is kind of bio risk, another is cybersecurity, and a third is kind of research automation and model autonomy. And that's kind of what ties most into the sweepage.
22:42 Speaker 00 This is all.
22:42 Speaker 01 Where coding is not all of automating research, but it is one very important key component. And so we initially created sweepage verified as part of like building out evals for that model autonomy work stream. And now, uh, I think for like, we have to move beyond that towards looking more at like, can models actually start to actually automate research.
23:02 Speaker 00 Yeah, amazing. Okay, I need anything else to add on just the general what people should know about preparedness? And how evals and human data alignment all work together at that.
23:12 Speaker 02 I think maybe the thing that I would say is that we really appreciate, we, we work really hard to build these evals and, and so we, that's where we published, so we've verified and that's where we're like sharing GDP via these sorts of things. We also deeply appreciate like other people and the entire field to kind of build evals and, and share them and we use them like Sweet Bench Pro. We're like, yes, but that's a better eval now. We should use them. So would really encourage people to find more ways to create and share evals that we can, we, and the entire field can use to measure like progress.
23:50 Speaker 00 Yeah.
23:50 Speaker 02 On, on, on like a variety of capabilities, including, including coding, because it's important to understand where we are.
23:57 Speaker 00 Mia had to leave, but we're, we're just kind of talking a little bit about like the, the future directions that we want evals to go. And I, I think here, here we can dive in on like, give us work, good work on these, these, these things. We'll talk to you. You know, like here's your platform to make a call for what you're looking for.
24:15 Speaker 01 I think a few things that would be useful. I'd say, first of all, really, really hard task. The kinds of things that would take top notch engineers, months or teams, weeks would be quite good. Especially if grading is reliable and grading is like, you know, you have, for example, like rubrics that have been sourced and validated by many people in the field. I think that'd be quite valuable. I think also benchmarks on kind of creating products end to end. I think if people are like putting more, that would be quite useful. I think a third thing that I'd say that is maybe not quite an eval, but I think is still relevant to the kind of overall mission of like, we as a field and as a world should be tracking, like where are these capabilities going?
24:54 Speaker 01 I'd like to see more metrics tracking like real world usage, like how much is AI actually being used in the field? Uh, how much is it, you know, replacing people's jobs? How much is it, you know, augmenting people, speeding people up? Just like real world metrics. Yeah.
25:08 Speaker 00 Yeah. The, the, the replacement thing is always like a sensitive one on, on the sort of PR side of things. But like, you know, we create new jobs that, that manage the old jobs. That's how it is. Yourself, like, you know, I think in terms of the frontier evals that OpenAI is really going to excited to push. Like you, you put out really good work every single time. What should people expect from, from OpenAI itself?
25:28 Speaker 01 I'm not sure I can say what we're gonna.
25:30 Speaker 00 General directions.
25:31 Speaker 01 I mean, general directions, I think, looking at real world impact, like real world, real.
25:37 Speaker 00 G D L two, you know, whatever.
25:38 Speaker 01 That kind of stuff. Yeah.
25:39 Speaker 00 Yeah, amazing. Okay, well, I'm excited for more real world impact. I think you guys have, you know, really made a lot of progress and I think taken the love of industry leadership for C bench verified and, and now moving on to C bench pro. So thank you for doing this. Thank you for being so transparent and I think people will respond in kind. Yeah.
25:57 Speaker 01 Have your time.
25:58 Speaker 00 Thank you.
Previous Bitter Lessons in Venture vs Growth: Anthropic vs OpenAI, Noam Shazeer, World Labs, Thinking Machines, Cursor, ASIC Economics — Martin Casado & Sarah Wang of a16z Next Claude Code for Finance + The Global Memory Shortage: Doug O'Laughlin, SemiAnalysis Page updated October 26, 2026 · transcript and summary are machine-generated
