OpenAI’s IMO Team on Why Models Are Finally Solving Elite-Level Math

In just two months, a scrappy three-person team at OpenAI sprinted to fulfill what the entire AI field has been chasing for years—gold-level performance on the International Mathematical Olympiad problems. Alex Wei, Sheryl Hsu and Noam Brown discuss their unique approach using general-purpose reinforcement learning techniques on hard-to-verify tasks rather than formal verification tools. The model showed surprising self-awareness by admitting it couldn’t solve problem six, and revealed the humbling gap between solving competition problems and genuine mathematical research breakthroughs. Hosted by Sonya Huang, Sequoia Capital

• 0:00Speaker 00

  The pace of progress is really, I think you see it so clearly in math. And I think Alex tweeted about this, where, you know, even a few years ago, these models were struggling with like grade school math. And, you know, and then we, you know, I remember even in 2024, that like GSM AK was used as like the standard eval when everybody would release a model. And then it was like math for a short period of time, and then it became AMI, and then it became USAMO. And the pace that it's just gone, blown through all of these math benchmarks is astonishing.

• 0:48Speaker 01

  Today, we're joined by Alex Wei, Sheryl Hsu, and Noam Brown, the trio behind the OpenAI model that just achieved gold medal performance at the International Math Olympiad. The IMO gold is one of the most important milestones in the race to artificial superintelligence. And what makes this breakthrough particularly fascinating isn't just the mathematical chops, but the underlying architecture. General-purpose techniques for scaling test-time compute and handling hard-to-verify tasks that extend far beyond competition math. We've now gone from models that can reason about math for a tenth of a minute just a year ago, to systems that can reason and concentrate on the order of 100 minutes. The hope for superintelligence is that as we scale reasoning to thousands or hundreds of thousands of hours, we can begin to solve humanity's greatest unsolved problems in math, the sciences, and more.

• 1:32Speaker 01

  Alex, Sheryl, and Noam join us on Training Data to talk about their approach and share some of the behind-the-scenes fun and learnings behind this historic result. Enjoy the show.

• 1:42Speaker 01

  Alex, Sheryl, Noam, thank you so much for joining us today. We have with us the team behind OpenAI's first gold medal at the IMO. Congratulations to you all. It's a momentous achievement.

• 1:54Speaker 02

  Thanks.

• 1:55Speaker 00

  Yeah, thank you.

• 1:56Speaker 01

  I'd love to get into a little bit of the origin story behind this. I know that, you know, the IMO gold has just been this, you know, elusive thing that everyone in AI has been chasing for a long time. I remember back when Sam pitched us in 2021, it was on the slides, and I remember thinking, oh, that seems really far away. I'd love to understand, you know, the more immediate origin story for this specific effort. When did you guys start thinking about this, and how did it come about?

• 2:21Speaker 02

  Yeah, I think it's like one, sort of like... Something that we've been thinking about for a long time. I I remember in my, you know, first week at OpenAI. Noam asked me like, you know, when when do you think the model will get IMO gold? I thought, you know, like, it was really unlikely in 2025. Um, but I feel like it's something that's always been on our minds, as you said, like, you know, Sam, like many years ago as well. But this specific effort, um. I think, you know, we thought, I think it was really only like, you know, maybe like a couple of months since like.

• 2:55Speaker 01

  Just just a couple of months.

• 2:57Speaker 02

  Uh, like the sort of last sprint to like, get everything ready for uh this year's IMO. Um, and of course, we've been working on like improving our our algorithms. The ideas for this started coming together maybe like six months ago, but like really like the last push, like, you know, we're gonna try to do something for this year's IMO was was only a couple of months long.

• 3:18Speaker 01

  It's amazing. And how big is the team involved?

• 3:21Speaker 02

  I mean, so it's we're like, you know, definitely building on like a lot of folks work at OpenAI, like this is not possible without like, you know, a lot of help from people from, you know, the the people working on like inference and the scaling org, the people who train the do the pre-training and the RL training. Um, but in terms of like the core team, I would say it's just the three of us. Um, so it's a it was a super small, like scrappy effort here.

• 3:46Speaker 01

  That's crazy. Just the three of you.

• 3:48Speaker 00

  Also, it was mostly mostly Alex. Alex had been working Alex had been working on this technique, uh, you know, for for a while and Cheryl and I were happy to help out uh as we were getting closer to the IMO to like make it a reality.

• 4:00Speaker 01

  That's so cool. And how does this even come about? Like, do you self-direct and self-choose, you know, I want to work on IMO gold and I'm gonna get us there and like, how does, you know, how do you even raise your hand to work on something like this?

• 4:11Speaker 02

  I think it was something where it just felt like, you know, maybe maybe it's possible, like, maybe if we like, you know, push a bit for a couple of months, we can just like, you know, get there.

• 4:19Speaker 00

  One of the nice things about OpenAI is that I think the researchers are really empowered to, you know, do the kinds of research that they think is impactful. And, you know, so Alex, you know, had this like pitch that like, hey, you know, there's this new technique that I, you know, I think could, could help out a lot. And, you know, honestly, like, there's a decent amount of skepticism, you know, I think some people were supportive, uh, but, you know, everybody felt like we should give them the, you know, the freedom to be able to explore this and pursue it. And, uh, and then it started showing some strong evidence and, you know, I think people still were a little skeptical, but like more people were getting excited about it. And, uh, you know, eventually it turns into something more substantial. And I think now people are obviously very excited about it.

• 4:57Speaker 01

  Can you say a little bit more about the strong evidence? Like, what were some of the early signs that you all were seeing that made you really lean in?

• 5:03Speaker 02

  I think it's just like, you know, like progress on hard to verify tasks. Where I think previously we, you know, um, a lot of our was more focused around just like, you know, if you have like these verifiable rewards, like, you know, what can you do? You know, we were just seeing more improvement on these like harder to verify tasks is I think what made us excited.

• 5:27Speaker 01

  Maybe on that front, how did you even verify that the results you had were right? Um, and I, I saw that you published the, the proofs on, on GitHub, but can you just say a little bit more about how, how you, how you even know, um, that, that what, that, you know, you've discovered the answers? Because my understanding is that, you know, they're done a bit differently from how a human might answer them.

• 5:49Speaker 02

  Yeah, yeah, I do think like, you know, like the style of the model outputs is a little atrocious.

• 5:56Speaker 01

  Atrocious isn't the word I was gonna use. It's creative, like an alien language.

• 5:59Speaker 02

  Yeah, yeah, it's a little, yeah, I think it was, we, I think, you know, we could have, I think, you know, it was a very like small scrappy effort. And so, um, we, we didn't optimize as hard for like, you know, human readability, but that's something that, you know, we know how to do, like, you know, we can, like, we can do the same stuff to like, in the same way that like, you know, chat GPT like is very readable, we can, we can do the same things here.

• 6:22Speaker 01

  Do you even need to optimize for human readability? Like, is that even important?

• 6:26Speaker 00

  I think if you're showing this to humans, they prefer readability. We were actually discussing, like, you know, we got the proofs, like, okay, because you could actually just like run them through ChatGPT and like ask ChatGPT to like rewrite them in a more readable way. And it's like, the proofs are still correct, they're just like a little bit more readable. And we were like, oh, should we, when we post these online, should we like post the more readable version? That's like run through ChatGPT or should we just post the raw version? And we decided, you know, I think for full transparency, we'll just like post the originals and people will figure it out.

• 6:57Speaker 01

  You guys have a bunch of IMO medalists and participants in the staff at OpenAI, right? Or do you guys like moonlight in your spare time grading the answers that the model produces?

• 7:06Speaker 02

  Like during, yeah, I mean during like, you know, the testing, like, you know, yeah, we like, we read a lot of samples. But for grading these specifically, like, we hired external former IMO medalists. So each proof was graded by three medalists and for each one, they, you know, reached unanimous consensus on the correctness.

• 7:27Speaker 00

  I should also say that like for me, I don't know about Cheryl, but for me, like, the proofs are beyond my ability to comprehend. Like, I was a math major and like, I never really did competition math. And I already like, the stuff that this model is like writing about is beyond, beyond my ability to grade.

• 7:43Speaker 03

  Yeah, same. I think that's what makes it like even more amazing, just like how, how smart the model is.

• 7:49Speaker 01

  Totally. What about problem six? How come none of the models at this year's IMO had a solution? And your model didn't even attempt problem six. Can you say more about what makes that problem? And traditionally, problem six is always the hardest at the IMO, is that right?

• 8:02Speaker 02

  Yeah, I think problem three or problem six usually.

• 8:04Speaker 01

  Okay. Why, just say a bit more about what made problem six different and what you learned from, you know, the, I think you tweeted that the fact that your model knew that it couldn't solve problem six was one of the things that gave you hope. So just say a bit more about that as well.

• 8:17Speaker 02

  For problem six, I mean, it's just a really tough problem. I think like, you know, if you gave me like months to think about it, if you even gave me like a big hint about like, you know, the main idea to solve problem six, I don't think I'd be able to get there. It's, it's just like crazy, like tough problem where there's so many things you can do. And there's like, you know, a very narrow path to, you know, finding the proof. And I think, you know, it's one of those things, I think, like math is just hard.

• 8:45Speaker 03

  Yeah, and we like threw a lot of compute at problem six. Um, but I think it was good to see like the model doesn't like try to hallucinate or like try to just like make up some solution, but instead we'll say like no answer. I mean, it is kind of disappointing when you feel like it's done so much work um just to like say no answer. But I think, you know, it's good that it actually like acknowledges that.

• 9:07Speaker 01

  Yeah, that's an amazing level of self-awareness of your own kind of ceiling. Because I mean, I remember at least a couple of years ago with these models, they'd always try to be helpful and make up an answer, right? And so to see this is just like, I think, an amazing level of self-awareness from these models.

• 9:24Speaker 00

  When we released the reasoning models, um, you know, I talked to some professors at mathematicians, computer scientists, and I was asking them like, you know, are you finding value in these models? And the answer was, you know, frequently yes, but the one thing that they would complain about is like, if they would ever ask the model a question that it didn't know the answer to, it would just like output a very convincing but wrong answer. Um, and they would have to like go through it very carefully to figure out, was it exactly correct or was there like, you know, some flip of an inequality or something that the model snuck in there? And it's nice to see that this model like, if it, if it doesn't know, it will just like acknowledge that it doesn't know, at least more frequently.

• 10:02Speaker 01

  I guess internally, did you guys have like a betting, like a polymarket or something going on whether you guys were gonna win IMO gold this year? And like, what was the internal vibe?

• 10:11Speaker 02

  I think we, we felt like we had a strong shot. Um, but I think we also felt that it wasn't like a lock, where there's definitely a distribution of questions where the models would probably struggle more than the humans. But then, you know, there's another distribution of questions where the models would be really, really strong. And, you know, I think this year was somewhere in the middle, um, where, you know, like problem six, like, I think is just out of reach of state-of-the-art models today. And I think maybe in general, like, you know, like these hard, like combinatorics problems, which problem six was, are, I think, more challenging. And that's still something that the models struggle with.

• 10:54Speaker 01

  What is it about combinatorics that makes it challenging versus, you know, the like geometry, for example, which seems like you guys do well at?

• 11:02Speaker 02

  I think for combinatorics, it's probably because it's a little more like abstract, a little more high dimensional. And I think oftentimes like combinatorics problems sort of require like leaps of faith or leaps of insight that, you know, the models aren't good at, I think the models are more good at like, you know, problems that require like a bunch of smaller steps, for example.

• 11:29Speaker 01

  What about from your guys' perspective? Was the internal vibe optimistic or not that you all were going to get gold?

• 11:33Speaker 03

  I feel like it wasn't super optimistic. Like, I think they definitely knew that like it could happen. But I think like even like a month or like two months back, it definitely felt like we'd have to like improve quite a bit, which I guess we did.

• 11:46Speaker 00

  I remember I was talking to another researcher at OpenAI, like maybe two months before the competition. And we were like, you know, saying like, okay, if we were to bet, you know, I'm a betting man.

• 11:58Speaker 01

  Yes, you are.

• 11:59Speaker 00

  And I was saying like, what odds would you take? Because I was willing to bet. I'm like, we were going to get gold here. And he was like, there's, there's really no chance. And, and, you know, and, you know, he said that he would gladly take like two to one odds against like the model winning. So like, you know, less than one third chance. But he didn't want to bet against us. So, you know, he thought it'd be bad, bad vibes to bet against the team winning. So he didn't go for the bet.

• 12:25Speaker 01

  So did you make some, some pocket change to him?

• 12:27Speaker 00

  I wish I, I wish I did.

• 12:29Speaker 01

  I mean, you need it. So.

• 12:33Speaker 01

  Because I mean, you guys were at, I think you tweeted 12% on Amy like 15 months ago, right? So it's even, even though you want to never want to bet against Scall and OpenAI, it's just, it's just an astounding slope of what you all have accomplished here.

• 12:47Speaker 00

  The pace of progress is, is really, I think you see it so clearly in math. And I think Alex tweeted about this, where, you know, even a few years ago, these models were struggling with like grade school math. And, you know, and then we, we, you know, I remember even in 2024, that like GSM AK was used as like the standard eval when everybody would release a model. And then it was like math for a short period of time, and then it became Amy, and then it became USAMO. And the pace that it's just gone blown through all of these math benchmarks is really astonishing.

• 13:17Speaker 03

  Yeah, I remember training a model on GSM AK two years ago.

• 13:21Speaker 01

  Yeah, we're past those days, huh? Saturated the evals. What's next? Do you think, I mean, at this point next year, you think we'll be solving millennium prizes?

• 13:30Speaker 02

  I think it's, I think those are still very far away. I think on one hand, you know, you think about like how much math progress has been made since like GSM 8K, which is like, you know, like, just like two years ago was sort of a standard that people were trying to push on. You know, that's like an astounding level of progress. But also you think about like how much time it takes for people like, you know, GSM 8K problems, they're like grade school math, you know, it takes someone to do that math like a couple of seconds. And now we've gone from like a couple of seconds to something that takes like, you know, these brilliant students an hour and a half per problem on average, or like, you know, the IMO is three problems, four and a half hours.

• 14:11Speaker 02

  And then, you know, research math is going to be like, you know, these same, you know, brilliant students, they've grown up, they're researchers, it's going to take them like 1,500 hours. So there's like, you know, a thousand X of like more thinking time. And then millennium prize problems have taken entire fields, like, you know, people's lifetimes of thinking and, you know, we still don't have much progress on most of those. And so it's, on one hand, like, you know, super exciting that we've made so much progress. On the other hand, it's sort of also like humbling to see like how much further, you know, progress has to go from like an hour and a half to like, you know, tens of thousands, hundreds of thousands of hours of human thinking.

• 14:55Speaker 01

  Totally. Noam, I think you deserve a lot of credit for seeing the future on this. I remember you visited us before you even joined OpenAI, talking about the results from gameplay and, you know, what happens if you let a model think for hours and tens of hours. And credit you, you've really seen the future on this.

• 15:14Speaker 00

  Thank you, yeah. I mean, it's exciting to see it actually happen.

• 15:18Speaker 01

  What are the hard things that happen as you scale compute time, inference time from the order of 0.1 minutes to the order of 100 minutes? I guess at a high level, because not everyone, most of our listeners are not AI researchers, but what are the hard things that happen to, to keep the model kind of on the rails, so to speak?

• 15:39Speaker 00

  The one thing I think we can point to is like, pretty clearly a challenge is that if you have the model thinking for like 1500 hours, then in order to evaluate, you have to have it think for 1500 hours. And so eventually, the evaluation of the models becomes a significant, you know, a speed bump on progress. So we're not really at that point, you know, like if we have the model thing for an hour and a half, it's no big deal, you know, we can, we can run those tests. But to run a test where the model is thinking for a month, it takes a month to finish that test. And so progress can only advance so fast if you want to wait for those kinds of results.

• 16:16Speaker 01

  I think both of you are on the multi-agent team. Help me understand like what the role that multi-agent systems play in this is.

• 16:24Speaker 00

  Yeah, so in addition to having the model like think for a very long time and, you know, make a lot of progress on, you know, hard to verify tasks. This also involved scaling up parallel compute. And so that's, there's a multi-agent component to that. We're probably not going to be able to go into too much detail about the exact techniques. But that was certainly like one way that we were able to scale up test on compute for the IMO. By the way, one thing I'll add for the, the multi-agents, you know, scaling parallel computing is that the way that we did it, you know, we, we really tried to prioritize generality in our, in our techniques. I, for example, like, you know, I, I worked on AI for poker. Alex and I actually both worked on AI for diplomacy.

• 17:07Speaker 00

  So Alex was, was on the, the team that worked on Cicero.

• 17:10Speaker 01

  Cicero, yeah.

• 17:10Speaker 00

  Yeah.

• 17:11Speaker 01

  Nice.

• 17:11Speaker 00

  And, you know, those were projects that I'm, I'm really proud of. But they were also projects that we spent years working on to like achieve that result. And with the pace of AI progress being so fast, it felt like that wasn't the best use of time to like develop a very bespoke system that could only do that one task. And so we all like really prioritized general purpose techniques in all this. And, you know, the techniques that we used for everything, for scaling up the thinking time, for working on hard to verify tasks and for the parallel compute are all general purpose techniques that we're either planning or have used for, you know, other systems as well.

• 17:53Speaker 01

  And is that the reason you all chose not to do this in lean? Like my understanding is the official kind of IMO AI track was, was a, was a lean interpretation this year. Is that why you guys chose not to go with lean?

• 18:05Speaker 00

  Yeah, that's right. I mean, there is certainly, I think there is a lot of value in lean as a tool. You know, mathematicians find it useful, for example. But the priority for us is really general purpose reasoning capabilities. And lean has its limitations. And so that's why we wanted to prioritize natural language.

• 18:24Speaker 01

  My layman's understanding is lean is a formal verification tool. Does your result here basically say that like informal verification with scale can, you know, can perform at the same level or even surpass formal verification? Is that the right takeaway?

• 18:37Speaker 00

  I wouldn't say you, I would not say that's the right takeaway. And Alex, do you have thoughts?

• 18:42Speaker 02

  I see that these are just like, you know, sort of two like orthogonal sort of components here. Where like, I think, you know, I think we found the informal math sort of an interesting problem because it represents sort of like a kernel of difficulty around like, you know, like scaling up, test time compute, hard to verify tasks. That represented, you know, something like difficulties from like, you know, a very broad set of tasks that we were like, you know, interested in from like a general purpose standpoint. I think like lean is a little bit more narrow where like, I think a lot more of the world can be approached with like informal reasoning than is like formalizable.

• 19:23Speaker 00

  I don't think there's anything wrong with narrow AI. Like narrow AI can be very effective and obviously like far surpass general purpose AI in certain domains. And I think the right way to think about it is in the same way that humans, human mathematicians find a lot of value in lean.

• 19:41Speaker 00

  General AI can be compatible with, with, you know, a more narrow system that's, that's focused on like formal, formal mathematics. And, you know, the combination, I think, can be better because of it.

• 19:55Speaker 01

  I think I saw on Twitter from multiple folks at OpenAI, and I think you guys have mentioned this as well, that, you know, this system was built with a very similar approach and infrastructure to many of the recent launches from OpenAI. Like we had ESA from the ChatGPT agent launch on, on the podcast last week. Can you say a little bit more about what the similar kind of foundation and approach is?

• 20:18Speaker 03

  I think like infrastructure wise, like. I mean, like, we all kind of just use the same infrastructure. Um, but I think as far as like the core of this question, like, you know, like, like Noam Alex said, there's nothing like that's very bespoke to IMO here. And the hope is really that we can use the techniques that Alex worked on, um, as far as like non-verifiable tasks and as far as just scaling up test time compute, um, and be able to like, apply this to other areas of reasoning or other areas of like model capabilities in general, um, and just build like stronger models, you know, like, keep, keep improving agent, keep improving chat GPT and everything else.

• 20:55Speaker 01

  Uh, tell, tell me about the actual experience of IMO day. What, what was it like?

• 21:00Speaker 00

  Yeah, I mean, uh, uh, we were waiting for the problems to come through because like, you know, once the, once the like participants finish the exam, then, then they get, uh, posted. And so we like, you know, plug the problems into our model and, uh, that was around like, I guess, pretty late at night, maybe like 1:00 AM or something. And honestly, I went to sleep because it's like, you know, it's 1:00 AM, I'm not going to stay up for four and a half hours to like see the, I'll just wake up in the morning and see. Uh, but I think these two like actually stayed up and like, um, got to watch the model and, um, see it come in in real time.

• 21:32Speaker 03

  Yeah, it was, it was a lot of fun.

• 21:34Speaker 01

  Did anybody call Noam? Like, wake up, wake up, we got, we got this.

• 21:38Speaker 00

  There were, there were a couple of moments where like Alex was like so exhausted that he decided to like take a nap, but we like told him like, okay, just make sure your, so your phone is on silence so that we need to like, we need to like call you. And, uh, at one point we did actually have to call him, but I don't think he woke up.

• 21:54Speaker 01

  That's awesome. It must've been such a thrill and such a high, especially for that to come through at like, so you got, you started at 1:00 AM, so you must've known like 9:00 AM then?

• 22:03Speaker 03

  Oh, it's four and a half hours.

• 22:04Speaker 01

  Four and a half hours.

• 22:05Speaker 03

  So it's like, yeah, I don't know. I mean, we can kind of see the problems coming. Uh, so I'd just be like making sure the systems are like staying stable and Alex is like over there reading and seeing what they're not, how, how the model's doing.

• 22:18Speaker 01

  Uh, so you were doing the, you were doing the live human proof checking to see if it was actually.

• 22:22Speaker 02

  Yeah, I was, I was, you know, you're naturally very like anxious about the results. So I was just like looking at the, you know, like the partial progress the model was making, you can sort of. Like, we can sort of observe that. Um, and then like, you know, I also like, you know, hand check things like, you know, we were, we were gonna send these out to the graders, but like, I was also just like, hand check them because I was so curious, like.

• 22:49Speaker 01

  Okay, well, call me next time. I want, I want to come hang out for that. I'm not gonna go to sleep. That sounds awesome.

• 22:54Speaker 00

  One of, one of the cool things about these models is like, you know, I can't understand the proofs. Um, but when you see the model, like, thinking about it, it, it will express its uncertainty or its confidence in natural language throughout, throughout the process. And it will just kind of say words that will like hint at its like, you know, if, if it's like really confident that it figured out, it'll say good a lot, you know? Um, and if it's like unsure, it'll like throw in a lot of question marks. Um, and so it's like, cool that I can kind of follow along and see how the model is like, you know, feeling about, about its progress, even though I can't really tell if it's like got it correct or not.

• 23:29Speaker 03

  Yeah, you get the dreaded seems hard.

• 23:35Speaker 01

  You got that on problem six?

• 23:37Speaker 03

  Just got that a lot. No progress. Hard.

• 23:40Speaker 01

  Seems hard. Keep going. Too bad.

• 23:44Speaker 01

  Wonderful. Um, I guess looking ahead, what, you, you've got, you've gotten like the pinnacle results in competition math. I guess you can go do Putnam next year, but you're basically at the top, right? And so what's next?

• 23:56Speaker 02

  Yeah, so actually for, um, for Putnam, the problems,

• 24:01Speaker 02

  I think since the exam is like, you know, less time per problem than the IMO, and it's a little more, uh, knowledge heavy, uh, we actually found in our evals that the model, you know, was like, really, really good at putting them problems, like better than it was at IMO problems. And so I think, you know, the frontiers here are really not about like, you know, these like very like time boxed competition problems anymore, but it's about like problems that really take like longer periods of time and more deep thinking to solve.

• 24:32Speaker 01

  That's really cool. Okay, so you're gonna start proving novel theorems now?

• 24:36Speaker 02

  Okay, I think there's like, there's this, I think, very intimidating gap between like, you know, these like very like time box competition problems and like a, you know, real research breakthrough, which, you know, takes like a year's worth of work, like, you know, years, that's like on the order of like fifteen hundred hours instead of one point five.

• 24:56Speaker 01

  Yeah, totally. I guess relatedly, I was listening to the Demis podcast last night, and he mentions that, you know, the hardest thing is actually coming up with the interesting problems to solve. And I'm curious if you all agree with that.

• 25:11Speaker 00

  I think there's some truth to that, that, you know, these models are really good now at solving these problems, coming up with them is, you know, still a challenge. But I think it's also worth noting the incredible pace of progress that we're seeing. And, you know, there's always, there's always a next hurdle, you know. And originally, when L M's came out, it was like, well, how do we get them to reason? And then we got them to reason, but then how do we get them to reason on hard to verify tasks? And now they can reason on hard to verify tasks. And I think the next hurdle is going to be like, okay, well, how do we get them to come up with these novel questions? You know, like even creating an I M O question is a challenge and, you know,

• 25:56Speaker 00

  it takes a lot of expert mathematicians, a lot of work to do that. But I don't see any fundamental barriers that block us from getting there.

• 26:06Speaker 01

  I love that. Do your results in math, do they just fully generalize to, you know, you're just going to be better at scientific reasoning, you're going to be better at general reasoning? You know, does being great at competition math make you, you know, be great at everything else?

• 26:22Speaker 02

  I think how we approached this was not like, you know, we should be like, you know, great at competition math. We really, I think it's like, we were focused on like, developing like general purpose techniques.

• 26:35Speaker 02

  To make up like reinforcement learning better. And I think those we are, you know, very excited to like improve our models in other domains beyond math. And so, and you know, hopefully like make models more useful for like, you know, us in like everyday usage.

• 26:54Speaker 00

  This is like a, you know, it's a pretty late breaking result. It's honestly, it was a surprise even to people internally at OpenAI. And so, um, the next step is to incorporate this more broadly into our models and, you know, improve the reasoning capabilities across the board. But, you know, it's going to take some time to go through that process and deploy it to the world. So, um, I think it's going to come, but yeah, it'll just take a little bit more time.

• 27:22Speaker 01

  Is it harder for these ma- these models to do the IMO or the Physics Olympiad?

• 27:27Speaker 02

  I think definitely the Physics Olympiad because the Physics Olympiad has, I think, like an experimental section.

• 27:34Speaker 01

  Oh.

• 27:34Speaker 02

  Or you have to.

• 27:36Speaker 01

  We need to solve robotics first. I didn't realize that. Okay. I thought it was just on a piece of paper.

• 27:42Speaker 02

  Yeah, so I think it's, the model will probably be good at the on the paper part, but yeah, I think the will be a little bit of, a bit of time before it can, you know, do the experiments.

• 27:52Speaker 01

  Not with like a world model?

• 27:56Speaker 01

  Okay, cool. Are you going to release this model for customers to play with? Ruloff's son is a math Olympiad kid and he's like, I want access to the math Olympiad model. Like, will people be able to play with this?

• 28:08Speaker 00

  So we, we want to make this accessible to mathematicians to use. We're still trying to figure out the exact details of how we make that happen. But, you know, I think it's really cool that we've developed this system that is incredibly good at math. And it makes sense that we want to see what mathematicians can do with it. I've actually already been emailing with like the Stanford professor, mathematics professor. He actually emailed me like about a year ago before we announced O1. And he was like, hey, do you want to do a collaboration on like solving hard math problems? And basically what I told him is like, I think we just got to advance general reasoning capabilities. And eventually they're going to be able to help you with your like hard math problems. And I think that's actually the most promising route to getting there. He was a little skeptical, but every model release, like every reasoning model release, he's like emailed me with a follow up and is like, can it solve this problem now?

• 28:54Speaker 00

  And I've been plugging them in. And I don't know what the output is, but I like email it back to him and he says like, "Yeah, that's wrong." And he emailed me like a follow-up this time with like, you know, the same problem like asking like, "Hey, can it solve it now?" It still can't solve it, but at least this time it like recognizes that it can't solve it. So I think that's like a big step. But we're curious to see if there's like a lot of other problems out there that mathematicians want to challenge this model with and see if it can take them on.

• 29:21Speaker 01

  Amazing. Congratulations to you all. I think this is a momentous result that the entire field has been waiting for for a very long time. And the fact that it was accomplished by a team of three people in a span of two months, it's extraordinary. Congratulations and thanks for joining us on Training Data.

• 29:37Speaker 00

  Thank you. Thanks for having us.