Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

Solving Poker and Diplomacy, Debating RL+Reasoning with Ilya, what’s *wrong* with the System 1/2 analogy, and where Test-Time Compute hits a wall Full Video Episode Timestamps 00:00 Intro – Diplomacy, Cicero & World Championship 02:00 Reverse Centaur: How AI Improved Noam’s Human Play 05:00 Turing Test Failures in Chat: Hallucinations & Steerability 07:30 Reasoning Models & Fast vs. Slow Thinking Paradigm 11:00 System 1 vs. System 2 in Visual Tasks (GeoGuessr, Tic-Tac-Toe) 14:00 The Deep Research Existence Proof for Unverifiable Domains 17:30 Harnesses, Tool Use, and Fragility in AI Agents 21:00 The Case Against Over-Reliance on Scaffolds and Routers 24:00 Reinforcement Fine-Tuning and Long-Term Model Adaptability 28:00 Ilya’s Bet on Reasoning and the O-Series Breakthrough 34:00 Noam’s Dev Stack: Codex, Windsurf & AGI Moments 38:00 Building Better AI Developers: Memory, Reuse, and PR Reviews 41:00 Multi-Agent Intelligence and the “AI Civilization” Hypothesis 44:30 Implicit World Models and Theory of Mind Through Scaling 48:00 Why Self-Play Breaks Down Beyond Go and Chess 54:00 Designing Better Benchmarks for Fuzzy Tasks 57:30 The Real Limits of Test-Time Compute: Cost vs. Time 1:00:30 Data Efficiency Gaps Between Humans and LLMs 1:03:00 Training Pipeline: Pretraining, Midtraining, Posttraining 1:05:00 Games as Research Proving Grounds: Poker, MTG, Stratego 1:10:00 Closing Thoughts – Five-Year View and Open Research Directions This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Alessio Fanelli?

  Hey everyone, welcome to the Late and Space Podcast. This is Alessio, partner and CTO at Dasebo, and I'm joined by my co-host Spix, founder of Small AI.

• 0:12Speaker 01

  Hello, hello, and we're here recording on a holiday Monday with Noam Brown from OpenAI. Welcome.

• 0:18Speaker 02

  Thank you.

• 0:19Speaker 01

  So glad to have you finally join us. A lot of people have heard you. You've been rather generous of your time on the podcast, Lex Friedman, and you've done a TED Talk recently just talking about the thinking paradigm. But I think maybe perhaps your most interesting recent achievement is winning the World Diplomacy Championship.

• 0:39Speaker 02

  Yeah.

• 0:39Speaker 01

  In 2022, you built Cicero, which was top 10% of human players. I guess my opening question is, how has your diplomacy playing changed since working on Cicero and then now personally playing it?

• 0:53Speaker 02

  When you work on these games, you kind of have to understand the game well enough to be able to debug your bot. Because if the bot does something that's really radical and that humans typically wouldn't do, you're not sure if that's a mistake, or if that's a bug in the system, or it's actually just the bot being brilliant. When we were working on diplomacy, I did this deep dive, trying to understand the game better. I played in tournaments, I watched a lot of tutorial videos and commentary videos on games. And over that process, I got better. And then also seeing the bot, the way it would behave in these games. Sometimes it would do things that humans typically wouldn't do. And that taught me about the game as well. When we released Cicero, we announced it in late 2022.

• 1:36Speaker 02

  I still found the game really fascinating, and so I kept up with it. I continued to play, and that led to me winning the championship, the world championship in 2025. So just a couple of months ago.

• 1:46Speaker 01

  There's always a question of Centaur systems where humans and machines work together. Was there an equivalent of What Happened in Go where you updated your playstyle?

• 1:55Speaker 02

  If you're asking if I used Cicero when I played in the tournament, the answer is no. Seeing the way the bot played and taking inspiration from that, I think, did help me in the tournament. Yeah.

• 2:06Speaker 01

  Yeah. Do people now ask Turing questions every single time when they're playing diplomacy?

• 2:13Speaker 02

  To try to

• 2:13Alessio Fanelli?

  Because it's public now.

• 2:15Speaker 02

  If the person they're playing with is a bot or a human.

• 2:17Alessio Fanelli?

  Yeah, like that's the one thing you worried about when you started.

• 2:20Speaker 02

  It was really interesting when we were working on Cicero because like, you know, we didn't have the best language models. We were really bottlenecked on the quality of the language models. And sometimes the bot would do, would say like bizarre things like, you know, 99% of the time it was fine. But then like every once in a while, it would say this like really bizarre thing. Like it would just hallucinate about something. Somebody would reference something that they said earlier in a conversation with the bot. And the bot would be like, I have no idea what we're talking about. I never said that. And then the person would be like, look, you could just scroll up in the chat. It's literally right there. And the bot would be like, no, you're lying.

• 2:51Speaker 02

  And when it does these kinds of things, like people just kind of like shrugged it off as like, oh, that's just, you know, the person's tired or they're drunk or whatever. Or they're just like trolling me. But I think like, that's because people weren't looking for a bot. They weren't expecting a bot to be in the games. We were actually really scared because we were afraid that people would, would figure out at one point that there's a bot in these games. And then they would just like always be on the lookout for it and they would always be, and if you're, if you're looking for it, you're able to spot it. That's the thing. So I think now that it's announced and that people know to look for it, I think they would have an easier time spotting it. Now that said, the language models have also gotten a lot better since 2022.

• 3:27Alessio Fanelli?

  It's adversarial.

• 3:27Speaker 02

  Yeah. So at this point, like, you know, the truth is, you know, GPT-4O and like 03, these models are like passing the Turing test. So I don't think they can really ask that many Turing complete questions that would actually make a difference.

• 3:39Alessio Fanelli?

  And Cicero was very small, like 2.7B, right?

• 3:42Speaker 02

  It was a very small language model. Yeah. It was one of the things we realized over the course of the project that like, oh yeah, we, you really benefit a lot from just having like larger language models.

• 3:50Alessio Fanelli?

  Right. Yeah. How do you think about today's perception of AI and a lot of like, maybe the safety, the scores of like, you know, you're going to build a bot that is really good at persuading people into like helping them win a game. And I think maybe today labs want to say they don't work on that type of problem. How do you think about that dichotomy, so to speak, between the two?

• 4:08Speaker 02

  You know, honestly, like after we released Cicero, a lot of the AI safety community was really happy with the research and like the way it worked. Because it was a very controllable system. Like we conditioned Cicero on certain concrete actions. And that gave it a lot of steerability to say like, okay, well, it's going to pursue a behavior that we can like very clearly interpret and it very clearly define. It's not just like, oh, it's a language model like running loose and doing whatever it feels like. No, it's actually like pretty steerable and there's this whole reasoning system that steers the way the language model interacts with the human. Actually, a lot of researchers reached out to like, reached out to me and said, like, we think this is like potentially a really good way to achieve safety with these systems.

• 4:50Speaker 01

  I guess the last diplomacy related questions that we might have is have you updated or tested like O series models on diplomacy? And would you, would you expect a lot more difference?

• 5:02Speaker 02

  I have not. I think I said this on Twitter at one point that I think this would be a great benchmark. I would love to see all the leading bots play a game of diplomacy with each other and see like who does best. And I think a couple of people have like taken inspiration from that and are actually like building out these benchmarks and like evaluating the models. My understanding is that they don't do very well right now. But I think, I think it really is a fascinating benchmark and I think it would be, um, yeah, I think it'd be a really cool thing to, to try out.

• 5:25Speaker 01

  Well, we're going to go a little bit into O series now. I think the last time you did a lot of publicity, you were just launching O1, you did your TED Talk and everything. How has the vibe, how have the vibes changed just in general? You said you were very excited to learn from domain experts like in chemistry, like how they review the old series models, like how have you updated since, let's say, end of last year?

• 5:48Speaker 02

  I think the trajectory was pretty clear pretty early on in the development cycle. And I think that everything that's unfolded since then has been pretty on track for what I expected. So I wouldn't say that my perception of where, where things are going or has honestly changed that much. I think they were going to continue to see, I said before that we're going to see this paradigm continue to progress rapidly. And I think that that's true even today, that we, we saw that with like going from O1 preview to O1 to O3, consistent progress and we're going to continue to see that going forward. And I think that we're going to see a broadening of what these models can do as well. You know, like we're going to start seeing agentic behavior. We're already starting to see agentic behavior.

• 6:31Speaker 02

  Like honestly, for me, O3, I've been using it a ton in my day to day life. I just find it so useful, especially the fact that it can now browse the web and like, you know, do meaningful research on my behalf, like it's kind of like a mini deep research that you can just get a response in three minutes. So yeah, I think it's just going to continue to become more and more useful and more powerful as time goes on and pretty quickly.

• 6:51Alessio Fanelli?

  Yeah, and talking about deep research, you tweeted about if you need proof that we can do this in number of viable domains, the research is kind of like a great example. Can you maybe talk about if there's something that people are missing, you know? I feel like I hear that repeated a lot. It's like, you know, it's easy to do encoding in math, but like not in these other domains.

• 7:08Speaker 02

  I frequently get this question, including from pretty established AI researchers that, okay, we're seeing these like reasoning models exceed in math and coding and these easily verifiable domains, but are they ever going to succeed in domains where success is less well defined? I'm surprised that this is such a common perception because we've released deep research and people can try it out. People do use it. It's very popular. And that is very clearly a domain where you don't have an easily verifiable metric for success. It's very like, what, what is the best research report that you could generate? And yet these models are doing extremely well at this, at this domain. So I think that's like an existence proof that these models can succeed in tasks that don't have as easily verifiable rewards.

• 7:51Alessio Fanelli?

  Is it because there's also not necessarily like a wrong answer? Like there's a spectrum of deep research quality, right? You can have like a report that like looks good, but the information is kind of so and so, and then you have a great report. Do you think people have a hard time understanding the difference when they get the result?

• 8:07Speaker 02

  My impression is that people do understand the difference when they get a result. And then I think that they're surprised at how good the deep research results are. There certainly, it's not, not a hundred percent. It could be better and we're going to make it better. But I think people can tell the difference between a good report and a bad report and certainly and a good report and a mediocre report.

• 8:24Alessio Fanelli?

  And that's enough to kind of feed the, the loop later to build the product and improve the model performance.

• 8:30Speaker 02

  I mean, I think if you're in a situation where people can't tell the difference between the outputs, then it doesn't really matter if you're like, you know, hill climbing on, on progress. These models are going to get better at domains where there is a measure of success. Now, I think this idea that it has to be like easily verifiable or something like that, I don't think that's true. I think that you can have, you can have these models do well, even in domains where success is a very difficult to define thing, could sometimes even be subjective.

• 8:56Speaker 01

  People lean on a lot, you've, you've done as well, is the thinking fast and slow analogy for just thinking models. And I think it's reasonably well diffused now, the idea of that this is kind of the next scaling paradigm. All analogies are imperfect. What is one way in which thinking fast and slow or system one, system two kind of doesn't transfer to? How we actually scale these things.

• 9:22Speaker 02

  One thing that I think is underappreciated is that the models, the pre-trained models need a certain level of capability in order to really benefit from this like extra thinking. This is kind of why you you've seen the reasoning paradigm emerge around the time that it did. I think it could have happened earlier, but if you try to do the reasoning paradigm on top of G P T two, I don't think it would have gotten you almost anything.

• 9:43Speaker 01

  Is this emergence?

• 9:44Speaker 02

  Hard to say if it's emergence necessarily, but like I haven't done the, you know, the measurements to really define that clearly, but I think it's pretty clear. You know, people tried chain of thought with G P D like really small models and they saw that it just didn't really do anything. Then you go to bigger models and it starts to to give a lift. I think there's a lot of debate about like the extent to which this kind of behavior is emergent, but clearly there is a difference. So it's not like there are these two independent paradigms. I think that they are related in the sense that you need a certain level of system one capability in your models in order to have system two, is it be able to benefit from system two?

• 10:22Speaker 01

  Yeah, I have played, tried to play amateur neuroscientists before and tried to compare it to the evolution of the brain and how you have to evolve the cortex first before you evolve the other parts of the brain. And perhaps that is what we're doing here.

• 10:35Speaker 02

  Yeah, you could argue that actually this is not that different from like, I guess, the the system one, system two paradigm, because, you know, if you ask like a pigeon to think really hard about playing chess, you know, it's not going to get that far. It's, you know, it doesn't matter like things for a thousand years. It's like not going to be able to be better at playing chess. So maybe you do still also also in like with animals and humans that you need a certain level of intellectual ability, just in terms of system one, in order to benefit from system two as well.

• 11:00Speaker 01

  Yeah, just a side tangent, does this also apply to visual reasoning? So let's say we have now we have the four O like natively Omni model type of thing. Then that also makes O three really good at geoguesser. Does that apply to other modalities too?

• 11:18Speaker 02

  I think the evidence is yes. It depends on exactly the kinds of questions that you're asking. Like, there are some questions that I think don't really benefit from System 2. I think GeoGuessr certainly won, where you do benefit. I think image recognition, if I had a guess, it's like one of those things that you probably benefit less from System 2 thinking.

• 11:35Speaker 01

  Because you know it or you don't.

• 11:36Speaker 02

  Yeah, exactly.

• 11:37Speaker 01

  There's no way.

• 11:37Speaker 02

  Yeah, and the thing I typically point to is just like information, like retrieval. If somebody asks you, like, when was this person born? And you don't have access to the web, then you either know it or you don't. And you can sit there and you can think about it for a long time. Maybe you can make an educated guess, so you can say like, well, this person was like, probably lived around this time. And so this is like a rough date. But you're not going to be able to like, get the date unless you've actually just know it.

• 12:01Speaker 01

  But like spatial reasoning, like tic-tac-toe, might be better because you have all the information there.

• 12:05Speaker 02

  Yeah, and I think it's true that like with tic-tac-toe, we see that like GPT-4.5 falls over. You know, it plays decently well. I shouldn't say it falls over. It does reasonably well, can draw the board, it can make legal moves. But it will make mistakes sometimes. And if you really need that System 2 to enable it to play perfectly. Now, it's possible that if you got the GPT-6 and you just did System 1, it would also play perfectly. You know, I guess we'll know one day. But I think right now you need a System 2 to really like do well.

• 12:34Alessio Fanelli?

  What do you think are like the things that you need in System 1? So obviously, general understanding of like game rules. Do you also need to understand some sort of like meta-game of like, you know, usually this is like how you value pieces in different games, even though it's a, you know, how do you generalize in System 1 so that then in System 2, you can kind of get to the gameplay, so to speak?

• 12:53Speaker 02

  I think the more that you have in the System 1, like, this is the same thing with humans, you know, like humans are when they're playing for the first time a game like chess. They can apply a lot of System 2 thinking to it. And if you apply a ton of System 2 thinking to it, like if you just present a really smart person with a completely novel game, and you tell them like, okay, you're going to play this game against like an AI or like a human that's like master this game. And you tell them to like, sit there and think about it for like, three weeks about how to play this game? My guess is they could actually do pretty well, but it certainly helps to build up. Up that system one thinking like build up intuition about about the game because it will just make you so much yeah so much faster.

• 13:35Alessio Fanelli?

  I think the Pokemon example is a good one of like the system one can ask maybe all this information about games. And then, once you put in the game, it still needs a lot of harnesses to work. And I'm trying to figure out how much of can we take things from the harness and have them in system one. So that then system two is as harness free as possible. But I guess that's like the question about generalizing games and AI.

• 13:57Speaker 02

  Yeah, I guess I view that as a different question. I think the question about like harnesses, in my view, is that the ideal harness is no harness.

• 14:05Alessio Fanelli?

  Right.

• 14:05Speaker 02

  I think harnesses are like a crutch that eventually we're going to be able to move beyond.

• 14:11Speaker 01

  So only two calls.

• 14:12Speaker 02

  And you could ask, you know, you could just ask O three. And actually, you know, it's interesting because like when this playing Pokemon thing kind of like emerged as as this like, you know, benchmark. I was actually like pretty opposed to evaling this with our with our like opening I models because my feeling is like, okay, if we're going to do this eval, let's just do it with O three, you know? How far does O three get without any harness? How far does it get playing Pokemon? And the answer is like, not very far, you know? And that's fine. I think it's fine to have an eval where the models do terribly. And I don't think the answer to that should be like, well, let's build a really good harness so that now it can do well in this eval. I think the answer is like, okay, well, let's just like improve the capabilities of our models so they can do well at everything. And then they also happen to make progress on this eval.

• 14:56Alessio Fanelli?

  Would you consider things like checking for a valid move a harness? Or is this in the in the model, you know, like chess? It's like you can either have the model learn in system one, what moves are valid and what it can and cannot do versus in system two figuring out.

• 15:12Speaker 02

  I think I think there's like, a lot of this is design questions. Like for me, I think you should give the model the ability to check if a move is legal if you want. Like that that could be an option in the environment of like, okay, here's a, you know, an action that you can like a tool call you can make to see if an action is legal. If it wants to use that, it can. And then there's like design question of like, well, what do you do if the model makes an illegal move? And I think it's totally reasonable to say like, well, if they make an illegal move, then they lose the game. Like, I don't know, what what happens when a human makes an illegal move in a game of chess? I actually don't know.

• 15:41Alessio Fanelli?

  Just not allowed to?

• 15:42Speaker 02

  Yeah, like, do you just lose the game? So if that's if that's the case, and I think it's totally reasonable to say like, yeah, we're gonna have an eval where that's also the criteria for the AI models.

• 15:51Speaker 01

  Yeah, but I think like maybe one way to interpret that in sort of researcher terms is, are you allowed to do search? And one of the famous findings from DeepSeek is that MCTS wasn't that useful to them. But I think like there are a lot of engineers trying out search and spending a lot of tokens doing that. And maybe it's not worth it.

• 16:09Speaker 02

  Well, I'm making a distinction here between like a tool call to check whether a move is legal or illegal is different from actually making that move and then seeing whether it ended up being legal or illegal, right? So if that tool call is available, I think it's totally fine to make that tool call and check whether a move is legal or illegal. I think it's different to have the model say, oh, I'm making this move. And then, you know, it gets feedback that like, oh, you made an illegal move. And so then it's like, oh, just kidding. Like, I'm gonna do something else now. So so that's that's the distinction I'm drawing.

• 16:40Speaker 01

  Some people have tried to classify that second type of playing things out as test time compute. You would not classify that as test time compute?

• 16:48Speaker 02

  There's a lot of reasons why you would not want to rely on that paradigm when you're going to imagine you have a robot, you know, and your robot like takes some action in the world and it like breaks something. And you're just like, oh, you can't say like, oh, just kidding. I didn't mean to do that. I'm gonna undo that action. Like, the thing is broken. So if you want to simulate what would happen if I move the robot in this way, and then in the simulation, you saw that this thing broke, and then you decide not to do that action, that's totally fine. But you can't just like undo actions that you've taken in the world.

• 17:14Speaker 01

  There's a couple more things I wanted to cover in this rough area. I actually had an answer on the on the thinking fast and slow side, which maybe I'm curious what you think about. Like, a lot of people are trying to put in effectively model router layers. Let's say between like the the fast response model and the the long thinking model. Anthropic is explicitly doing that. And I think there is a question about always, do you need a smart judge to route? Or do you need a dumb judge judge to route because it's fast? So when you have a model router, let's say, let's say, you're passing requests between system one side and system two side. Does the router need to be as smart as the smart model or dumb to be fast?

• 17:53Speaker 02

  I think it's possible for a dumb model to recognize that a problem is really hard and that it won't be able to solve it and then route it to a A more capable model.

• 18:02Speaker 01

  But it's also possible for a dumb model to be fooled or to be overconfident.

• 18:06Speaker 02

  I don't know. I think there's a real trade off there. But I will say, like, I think, I think there are a lot of things that people are building right now that will eventually be washed away by scale. So I think harnesses are a good example where I think eventually the models are going to be. And I think this actually happened with the reasoning models, like before the reasoning models emerged. There was like all of this work that went into engineering, these like agentic systems that like made a lot of calls to GPT-40 or like these non-reasoning models to get reasoning behavior. And then it turns out like, oh, we just like created reasoning models and they, you don't need this like complex behavior. In fact, in many ways, it makes it worse. Like you just give the reasoning model the same question without any sort of scaffolding and it just does it.

• 18:49Speaker 02

  Now that you can still, and so people are building scaffolding on top of the reasoning models right now, but I think in many ways, like those scaffolds will also just be replaced by the reasoning models and models in general becoming more capable. And similarly, I think things like model, like these routers, you know, we've said pretty openly that we want to move to a world where there is a single unified model. And in that world, you shouldn't need a router on top of the model. So I think that the router issue will eventually be solved also.

• 19:18Speaker 01

  Like you're building the router into the model kind of weights itself.

• 19:23Speaker 02

  I don't think there will be a benefit for, like, I shouldn't say, because it's, I could be wrong about this, like, you know, and certainly maybe there's, you know, reasons to route to different model providers or whatever. But I think that routers are going to eventually go away. And I can understand why it's worth doing it in the short term, because like the fact is it is beneficial right now. And if you're building a product and you're getting a lift from it, then it's worth doing right now. One of the tricky things I'd imagine that a lot of developers are facing is that like you kind of have to plan for where these models are going to be in six months and 12 months. And it's like very hard to do because things are progressing very quickly. You know, you don't want to spend six months building something and then just have it be totally washed away by scale.

• 20:07Speaker 02

  But I think I would encourage developers, like when they're, you know, building these kinds of things like scaffolds and routers. Keep in mind that the field is evolving very rapidly, you know, things are going to change in three months, let alone six months, and that might require radically changing these things around or tossing them out completely. So don't spend six months building something that might get tossed down in six months.

• 20:29Speaker 01

  It's so hard, though. Everyone says this and then like, no one has concrete suggestions on how.

• 20:36Alessio Fanelli?

  What about reinforcement fine tuning? Is this something that, obviously, you just released it a month ago, Erobinai? Is this something people should spend time on right now or maybe wait until the next?

• 20:46Speaker 02

  I think reinforcement fine tuning is pretty cool and I think it's like worth looking into because it's really about specializing the models for the data that you have. And I think that something that's like worth, worth looking into for, for developers. Like, we're not, we're not suddenly gonna like have that data baked into the raw model a lot of times. So I think that's kind of like a separate question.

• 21:11Alessio Fanelli?

  Yeah. So creating the environment and the reward model is the best thing people can do right now. I think the question that people have is like, should I rush to fine tune the, the model using RFT? Or should I build the harness to then RFT the models as they get better?

• 21:26Speaker 02

  I think the difference is that like for reinforcement fine tuning, you're collecting data that's going to be useful as the models improve as well. So if we come out with like future models that are even more capable, you could still fine tune them on your data. That's, I think, actually a good example where you're building something that's going to complement the model's scaling and becoming more capable, rather than necessarily getting washed away by the scale.

• 21:50Speaker 01

  Yep. One last question on Ilya. You mentioned on, I think, the Sarah and Alad podcast where you had this conversation with Ilya a few years ago about more RL and reasoning and language models. Just any speculation or thoughts on why his attempt, when he tried it, it didn't work? Or the, the timing wasn't right and why the time is right now?

• 22:14Speaker 02

  I don't think I would, I would frame it that way that like, this is his attempt didn't work. In many ways it did. So Ilya, for me, I saw that in all of these domains that I'd worked on, in poker and Hanabi and diplomacy, having the models think before acting made a huge difference in performance, like orders of magnitude difference.

• 22:36Speaker 01

  Like 10,000 times, isn't it?

• 22:37Speaker 02

  Yeah, like, you know, 1,000 to 100,000 times, like, it's a The equivalent of a model that's like a thousand to 100,000 times bigger. And in language models, you weren't really seeing that, that the model, the models would just respond instantly. Some people in the field, in the L M field were like convinced that like, okay, we just keep scaling pre-training. We're going to get to superintelligence. And I was kind of skeptical of that perspective. In late 2021, I was having a meal with Ilya. He asked me what my H I timelines are, very standard S F question. And I told him like, look, I think it's actually quite far away because we're going to need to figure out this reasoning paradigm in a very general way. And with things like L M's, L M's are very general. But they don't have a reasoning paradigm that's very general. And until they do, they're going to be limited in what they can do. We're going to scale it sure, we're going to scale these things up by a few more orders of magnitude.

• 23:22Speaker 02

  They're going to become more capable, but we're not going to see superintelligence from just that. And like, yes, if we had a quadrillion dollars to train these models, then maybe we would. But like, you're going to hit the limits of what's economically feasible before you get to superintelligence. Unless you have a reasoning paradigm. And I was convinced incorrectly that the reasoning paradigm would take a long time to figure out. Because it's like this big unanswered research question. And, you know, Ilya agreed with me and he said like, yeah, you know, I think we need this like additional paradigm. But his take was that like, maybe it's not that hard. I didn't know it at the time, but like he and others at OpenAI had also been thinking about this. They'd also been thinking about R L, they'd been working on it. And I think they had some success, but like, you know, with most research, like it does, you have to iterate on things.

• 24:07Speaker 02

  You have to try out different ideas, you have to, yeah, try different things. And then also, as the models become more capable, as they become faster, it becomes easier to iterate on experiments. And I think that the work that they did, even though it didn't like result in a reasoning paradigm, it all builds on top of previous work, right? So they built a lot of things that over, over time led to this reasoning paradigm.

• 24:31Speaker 01

  For listeners, no one can talk about this, but the rumor is that that thing was codename G P T zero. If you want to search for that, that line of work. I think there was a time where like basically R L kind of went through a dark age. When everyone like went all in on it and then nothing happens and they gave up. And like now it's like sort of the golden age again. So that's what I'm like trying to identify like why, what is it? And it could just be that we have smarter base models and better data.

• 24:57Speaker 02

  I don't think it's just that we have smarter base models. I think it's that, yeah, so I, we did end up getting a big success with, with reasoning. But I think it was in many ways, a gradual thing. To some extent, it was gradual, you know, like there were signs that there were signs of life. And then we like, you know, iterated and tried out some more things. We got like better signs of life. I think it was around like November 2023 or October 2023, when I think I was convinced that we had like very conclusive signs of life that like, oh, this is, this was going to be, this is the paradigm and it's going to be a big deal. That was in many ways a gradual thing. I think what OpenAI did well is like when we got those signs of life, they recognized it for what it was and invested heavily in scaling it up.

• 25:40Speaker 02

  And I think that's, that's ultimately what led to reasoning models arriving when they did.

• 25:45Alessio Fanelli?

  Was there any disagreement internally, especially because like, you know, OpenAI kind of pioneer pre-training scaling, you know, and kind of like computers are all you need. And then you're kind of saying, maybe that's not how we get there. Was it clear to everybody that like, okay, this is going to work or was it controversial?

• 26:01Speaker 02

  There's always different opinions about this stuff. I think there were some people that felt that pre-training was all we need, that we scaled it up to infinity and we're there. I think a lot of the leadership actually at OpenAI recognized that there was another paradigm that was needed. And that was why they were investing all this like research effort into this like RL stuff. And I think that's also to the credit of OpenAI that like, okay, yes, they figured out the pre-training paradigm and they were very focused on scaling that up. In fact, the vast majority of resources were focused on scaling that up. But they also recognized the value that that's something else was going to be needed. And it was worth researching, putting researcher effort into other directions to figure out what that extra paradigm was going to be. There was a lot of debate about, first of all, like, what is that extra paradigm?

• 26:40Speaker 02

  So I think a lot of the researchers looked at reasoning and RL was not really about scaling test on compute. It was more about data efficiency. Because, you know, the feeling was that, well, we have tons and tons of compute, but we actually are more limited by data. So there's, there's the data wall and we're going to hit that before we hit limits on the compute. So how do we make these algorithms more data efficient? They are more data efficient, but I think that also like, they are also just like. The equivalent of scaling up compute also by a ton. That was interesting. There was like a lot of debate around like, okay, what exactly are we doing here? And then I think also, even when we got the signs of life, I think there was a lot of debate about the significance of it. That like, okay, how much should we invest in scaling up this paradigm?

• 27:23Speaker 02

  I think especially when you're when you're in a small company, like, you know, OpenAI, like, in 2023 was not as big as it is today. And compute was more constrained than it is today. And if you're investing resources in a direction, that's coming at the expense of something else. And so if you look at these signs of life on reasoning and you're saying like, okay, well, this looks promising. We're going to scale this up by a ton and invest a lot more resources into it. Where are those resources coming from? You have to make that tough call about where to draw the resources from. And that is a very controversial, very difficult call to make that makes some people unhappy. And I think there was debate about whether we're focusing too much on this paradigm, whether it's really a big deal, whether we would see it generalize and do various things.

• 28:06Speaker 02

  And I remember it was interesting that I talked to somebody who left OpenAI after we had discovered the reasoning paradigm, but before we announced O1. And they ended up going to a competing lab. I saw them afterwards, after we announced O1. And they told me that like, at the time, they really didn't think this like reasoning thing, like this, these O series, the strawberry models were like that, that big of a deal. It was like, they thought we're making a bigger deal of it than it really deserved to be. And then when we announced O1 and they saw the reaction of their coworkers at this competing lab about how everybody was like, oh crap, like this is a big deal. And they like pivoted the whole research agenda

• 28:44Alessio Fanelli?

  Oh my god.

• 28:45Speaker 02

  To focus on this, that then they realized like, oh, actually, like, this maybe is a big deal. You know, a lot of this seems obvious in retrospect, but at the time it's actually not so obvious and be quite difficult to recognize something for what it is.

• 29:00Alessio Fanelli?

  I mean, OpenAI is like a great history of just making the right bet. I feel like GPT models are kind of similar, right? Where like it started with games, NRL, and then it's like, maybe we can just scale these language models instead. And I'm just impressed by the. Leadership and obviously the research team that keeps coming up with these insights.

• 29:19Speaker 02

  Looking back on it today, it might seem obvious that like, oh, of course, like these models get better with scale. So you should just scale them up a ton and it'll get better. But it really is, the best research is obvious in retrospect. And at the time, it's not as obvious as it might seem today.

• 29:35Speaker 01

  Follow up questions on data efficiency. This is a pet topic of mine. It seems that our current methods of learning are so inefficient still, right? Like compared to the existence proof of humans, we take five samples and we learn something. Machines, 200, maybe, you know, per like whatever data point you might need. Anyone doing anything interesting in data efficiency? Or do you think like there's just a fundamental inefficiency that machine learning has? That will just always be there compared to humans?

• 30:05Speaker 02

  I think it's a good point that if you look at the amount of data these models are trained on and you compare it to like the amount of data that a human observes to get the same performance. I guess pre-training, it's a little hard to make an apples to apples comparison because like, I don't know. How many tokens does a baby actually absorb when they're developing? But I think it's a fair statement to say these models are less data efficient than humans. And I think that that's an unsolved research question. And probably one of the most important unsolved research questions.

• 30:32Speaker 01

  Maybe more important than algorithmic improvements. Cause you can just, you can, we can increase the supply of data out of the existing set of the world and humans.

• 30:43Speaker 02

  I guess this is a good, so a couple of thoughts on that. Like one is that the answer might be an algorithmic improvement. Like maybe, maybe algorithmic improvements do lead to greater data efficiency. And the second thing is that like, it's not like humans learn from just reading the internet. So I think it's certainly easiest to learn from just like data that's on the internet. But I don't think that's like the limit of what data you could collect.

• 31:06Speaker 01

  The last follow up before we change topics to coding. Any other just anecdotes or insights from Ilya, just in general? Cause like you've worked with him, so there's not that many people that we can talk to that have worked with him.

• 31:17Speaker 02

  I think I've just been very, very impressed with his vision that I think like, especially when I joined and I saw, you know, the internal documents at OpenAI of like, what he had been thinking about back in like 2021, 2022, even earlier. I was very impressed that he had a clear vision of like where this was all going and what was needed.

• 31:36Speaker 01

  Some of his emails from 2016, 17, when they were founding OpenAI was published. And even then, he was talking about how he thinks like one big experiment is much more valuable than 100 small ones. That was like a core insight that differentiated them from Brain, for example. It just seems very insightful that he just sees things much more clearly than others. And I just wonder what his production function is like. How do you make a human like that? And how do you improve your own thinking to better model it?

• 32:05Speaker 02

  I mean, I think it is true that, I mean, one of OpenAI's big success was betting on the scaling paradigm. It is just kind of odd because, you know, they were not the biggest lab, you know? It was like difficult for them to scale. Back then, it was much more common to do like a lot of small experiments, more academic style. People were trying to figure out these various like algorithmic improvements. And OpenAI bet pretty early on like large scale.

• 32:27Speaker 01

  We had David Luhan on, who I think was VP Eng at the time of GPT-1 and 2. And he talked about how the differences between Brain and OpenAI was basically the cause of the Google's inability to come out with a scaled model. Like just structurally, everyone had allocated compute and you had to pool resources together to make bets. And you just couldn't.

• 32:47Speaker 02

  I think that's true that OpenAI was structured differently. And I think that really helped them like OpenAI functions a lot like a startup. And other places tended to function more like universities or, you know, research labs as they traditionally existed. The way that OpenAI operates more like as a startup with this mission of building AGI and superintelligence. That helped them organize, collaborate, pool resources together, make hard choices about like how to allocate resources. And I think a lot of the other labs like have now been trying to adopt paradigms more like that, like setups more like that.

• 33:22Alessio Fanelli?

  Let's talk about maybe the killer use case, at least in my mind, of these models, which is coding. You released Codex recently, but I would love to talk through the Noam Brown coding stack. What models you use, how you interact with them.

• 33:34Speaker 01

  Cursor, WinSurf.

• 33:36Speaker 02

  Lately, I've been using WinSurf and Codex, like actually a lot of Codex. I've been having a lot of fun, like you just give it a task and it just goes off and does it and comes back five minutes later with like a, you know, pull request.

• 33:46Speaker 01

  And is it core research task or like side stuff that you don't super care about?

• 33:49Speaker 02

  I wouldn't say it's like side stuff. I would say basically anything that I would normally try to code up, I try to do it with Codex first.

• 34:02Speaker 01

  Well, for you, it's free, but yeah, for everybody is free right now.

• 34:04Speaker 02

  And I think that's partly because it's the, it's the most effective way for me to do it. And also it's good for me to get experience working with this technology and then also like seeing the shortcomings of it. It just helps me like better understand like, okay, this is the, the limits of these models and like what we need to push on next.

• 34:20Speaker 01

  Have you felt the AGI?

• 34:21Speaker 02

  I felt the AGI multiple times, yes.

• 34:24Speaker 01

  Like, like, um, how should people push Codex in ways that you've, you've done? And, you know, I think you, you see it before others because obviously you were closer to it.

• 34:34Speaker 02

  I think anybody can use Codex and feel the AGI. It's kind of funny how like you feel the AGI and then you get used to it very quickly.

• 34:43Speaker 02

  So it's really like.

• 34:44Speaker 01

  Dissatisfied with like where it's lacking?

• 34:46Speaker 02

  Yeah, I know. It, you know, it's, it's magical. One day I was actually looking back at the old Sora videos when they were announced. Cause like, remember when Sora came out, it was just like.

• 34:55Speaker 01

  The biggest news ever.

• 34:56Speaker 02

  It was just magical. You look at that and you're like, it's like, it's really here. This is AGI, but if you look at it now and it's kind of like, oh, you know, the people like don't move like very organically. And it's like, there's like a lack of consistency in some ways. And you see all these flaws in it now that you just didn't really notice when it was first came out. And yeah, you get used to this technology very quickly and, but I think what's cool about it is that because it's developing so quickly, you get those feel the AGI moments like every few months. So something else is going to come out and just like, it's magical to you and, and then you get used to it very quickly.

• 35:28Speaker 01

  What are your WinSurf pro tips now that you've immersed in it?

• 35:33Speaker 02

  I think one thing I'm surprised by is how few people, I mean, maybe your audience is going to be more comfortable with reasoning models and like use reasoning models more. But I'm surprised at how many people don't even know that O3 exists. Like, I've been using it day to day. It's basically replaced Google Search for me. Like, I just use it all the time. Like, and also for things like coding, like I tend to just use the reasoning models. My suggestion is like if people are not, have not tried the reasoning models yet, because like honestly, like we do, like people love them, people that use it love them. Obviously, a lot more people use GPT-4O and just like the default on ChatGPT and that kind of stuff. I think it's worth trying the reasoning models. Like, I think people would be surprised at what they can do.

• 36:15Speaker 01

  I use WinSarve daily and they still haven't actually enabled it as like a default in WinSarve. Like, I always have to dig up, like, type in O3 and then it's like, oh yeah, that exists. It's weird. I would say like my struggle with it has been that it takes so long to reason and actually break out a flow.

• 36:34Speaker 02

  I think that is true, yes. And I think this is one of the advantages of Codex, that like, okay, you can give it a task that's kind of self-contained and like it can go off and do its thing and come back 10 minutes later. And I can see that if you're doing, if you're using this thing as like more like a, like a pair of programmer kind of thing, then yeah, you want to use GPT-4.1 or something like that.

• 36:51Alessio Fanelli?

  What do you think are the most broken part of the development cycle with AI? Like, in my mind, it's like, um, pull request review. Like, for me, like, I use Codex all the time and then I got all these pull requests and it's kind of hard to like go through all of them. What other thing would you like people to build to make this even more scalable?

• 37:09Speaker 02

  I think it's really on us to build a lot more stuff. These models are very limited in some ways. I think I find it frustrating that, you know, you ask them to do something and then they spend 10 minutes doing it. And then you ask them to do something pretty similar and then they go spend 10 minutes doing it and like, you know, it's, I think I describe them as like they're geniuses, but it's their first day on the job. You know, and that's like kind of annoying. Like even the smartest person on earth, when it's their first day on the job, you know, they're not going to be like as useful as you would like them to be. So I think being able to get more experience and like act like somebody that's actually been on the job for like six months instead of one day, I think would make them a lot more useful. But that's really on us to build, to build that capability.

• 37:51Speaker 01

  Do you think a lot of it is GPU constraint for you? Like, if I think about Codex, why is it asking me to set up the environment myself? When like the model, if I ask O3 to like create an environment setup script for a repo, I'm sure it'll be able to do it. But today in the product, I have to do it. So I'm wondering, in your mind, could these be a lot more if we just, again, put more test time compute on them? Or do you think there's like a fundamental model capability limitation today? That we still need a lot of like human harnesses around it?

• 38:19Speaker 02

  I think that we're in an awkward state right now. We're like, progress is very fast and there's things that are like, clearly we could do this and the models will be better. We're gonna get to it. It's, um, you're just limited by how many hours there are in the day.

• 38:30Speaker 01

  Right.

• 38:31Speaker 02

  You know, so progress can only proceed so quickly. We're trying to get to everything as fast as we can. And I think that the O3 is not where the technology will be in six months.

• 38:41Speaker 01

  I like that question overall in like, there's a software development lifecycle, not just generation of the code, like from issue to PR. Basically, it's like the typical commentary of that. And then there's the WinSurf side, which is inside your IDE. Like, what else, right? Pull Request Review is like something that people don't really... There are startups that are built around it. It's not something that Codex does. And it could. And so like, then there's like, what else is there? You know, that is sort of rate limiting the amount of software you could be iterating on. It's an open question. I don't, I don't, I don't know if there's an answer. Anything else on ASWE in general? Like, where do you think this goes? Just in form factors? Or what will we be looking at this time next year in terms of how things are, how,

• 39:26Speaker 01

  what models are able to do that they're not able to today?

• 39:29Speaker 02

  I don't think it's gonna be limited to ASWE. You know, I think, I don't think it's gonna be limited to software engineering. I think it's gonna be able to do a lot of remote work kind of tasks.

• 39:37Speaker 01

  Yeah, like Swielancer type upwork?

• 39:40Speaker 02

  Yeah, or just like even things that are not necessarily software engineering.

• 39:43Speaker 01

  Okay.

• 39:43Speaker 02

  So the way that I think about it is like... Anybody that's doing a remote work kind of job, I think it's valuable to become familiar with the technology and like kind of get a sense of like what it can do, what it can't do, what it's good at, what it's not good at. Because I think the breadth of things that it's going to be able to do is going to expand over time as well.

• 40:00Speaker 01

  I feel like virtual assistants might be the next thing after ASV then. Because they're the most easily like, you know, virtual system like hire someone in the Philippines, someone who just looked through your email and all that. Because that is entirely, you can intercept all the inputs and all the outputs and train on that. And maybe OpenAI just buys a virtual assistant company.

• 40:20Speaker 02

  Yeah, I think what I'm looking forward to is that for things like virtual assistants, the models, like if they're aligned well, they could end up being like really preferable for that kind of work, you know? If, there's always this like principal agent problem, where if you delegate a task to somebody, then like, are they really aligned with like doing it as you would want it to be done?

• 40:40Speaker 01

  And Just as cheaply, as quickly as they can.

• 40:43Speaker 02

  Yeah. And so if you have an AI model that's like actually really aligned to you and your preferences, then that could end up doing a way better job than a human could. Well, not that it's doing a better job than a human could, but like it's doing a better job than a human would.

• 40:56Speaker 01

  That word alignment, by the way, I think there's like an interesting overriding or homomorphism between safety alignment and instruction following alignment. And I wonder where they diverge.

• 41:09Speaker 02

  Okay. So I think where it diverges is like, what do you want to align the models to? Like that's, I think, a difficult question, you know? Like you could say like, you wanted to align it to the user. Okay. Well, what happens if the user wants to build a novel virus that's going to wipe out half of humanity?

• 41:22Speaker 01

  That's safety alignment.

• 41:23Speaker 02

  Yeah. So there's a question of like, I think alignment, I think they're related, you know? And I think the big question is like, what are you aligning towards?

• 41:30Speaker 01

  Yeah, there's like humanity goals and then there's your personal goals and everything in between.

• 41:35Alessio Fanelli?

  So that's kind of, I guess, the individual agent. And you announced that you're leading the multi-agent team at OpenAI. I haven't really seen many announcements, maybe I missed them on what you've been working on. But what can you share about interesting research directions or anything from there?

• 41:51Speaker 02

  Yeah, there's, uh, hasn't really been announcements on this. I think we're working on cool stuff and I think we'll get to announce some cool stuff at some point. I think the team, in many ways, is actually a misnomer because we're working on more than just multi-agent. Multi-agent is one of the things we're working on. Some other things we're working on is just like being able to scale up test time compute by a ton. So how, you know, we get these models thinking for 15 minutes now, how do we get them to think for hours? How do we get them to think for days, even longer, and be able to solve incredibly difficult problems? So that's one direction that we're pursuing. Multi-agent is another direction. And here, I think there's a few different motivations. We're interested in like both the collaborative and the competitive aspect of multi-agent. I think the way that I describe it is people

• 42:34Speaker 02

  often say in AI circles that humans occupy this very narrow band of intelligence and AIs are just gonna like quickly catch up and then surpass like this band of intelligence. And I actually don't think that the band of human intelligence is that narrow. I think it's actually quite broad, because if you compare anatomically identical humans from, you know, caveman times, they didn't get that far in terms of like, you know, we would consider intelligence today, right? Like, they're not putting a man on the moon, you know, they're not like building semiconductors or nuclear reactors or anything like that. And we have those today, even though we as humans are not anatomically different. And so what's the difference? Well, I think the difference is that you have thousands of

• 43:18Speaker 02

  years, a lot of humans, billions of humans cooperating and competing with each other, building up civilization over time. The technology that we're seeing is the product of this civilization. And I think similarly, the AIs that we have today are kind of like the cavemen of AI. And I think that if you're able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization essentially, the things that they would be able to produce and answer would be far beyond what is possible today with the AIs that we have today.

• 43:57Alessio Fanelli?

  Do you see that being similar to maybe like Jim Phan's Voyager skill library idea? Resaving these things? Or is it just the models then being retrained on this new knowledge? Because the humans then have it, a lot of it in the brain as they grow.

• 44:11Speaker 02

  I think I'm going to be evasive here and say that like, we're not gonna, yeah, we're not gonna,

• 44:16Speaker 02

  we're, until we have something to announce, which I think that, I think that we will in the not-too-distant future. I think I'm going to be a bit vague about like, exactly what we're doing. But I will say that the way that we are approaching multi-agent in the details and the way we're actually going about it, is I think very different from how it's been done historically and how it's being done today by other places. I've been in the multi-agent field for a long time. I've kind of felt like the multi-agent field has been a bit misguided in some ways and the things that, the approaches that the field has taken and like, the way that's been approached. And so I think we're trying to take a very principled approach to multi-agent.

• 44:52Speaker 01

  Sorry, I gotta add, like, so you can't talk about what you're doing, but you can say what's misguided? What's misguided?

• 44:57Speaker 02

  I think that a lot of the approaches that have been taken have been very heuristic and haven't really been following like, the bitter lesson approach to scaling in research.

• 45:08Alessio Fanelli?

  Okay. I think maybe this might be a good, a good spot. So, obviously you've done a lot of amazing work in poker. And I think as the reasoning model got better, I was talking to one of my friends who used to be a hardcore poker grinder, and I told them I was gonna interview you. And their question was, at the table, you can get a lot of information from a small sample size about how a person plays. But today, GTO is like, so prevalent that sometimes people forget that you can play exploitatively. What do you think is the state, as you think about multi-agent and kind of like, competition? Is it always gonna be trying to find the optimal thing? Or is a lot of it trying to think more in the moment, like, how to exploit somebody?

• 45:46Speaker 02

  I'm guessing your audience is probably not super familiar with poker terminology, so I'll just like, explain this a bit. A lot of people think that poker is just like, a luck game, and that's not true. It's actually, there's a lot of strategy in poker. So you can win consistently in poker if you're playing the right strategy. So there's different approaches to poker. One is game theory optimal. This is like, you're playing an unbeatable strategy and expectation, like, you're just unexploitable. It's kind of like in rock, paper, scissors. You can be unbeatable in rock, paper, scissors if you just randomly choose between rock, paper, and scissors with equal probability. Because no matter what the other guy does, You know, they're not going to be able to exploit you or you're going to win, you're going to like not lose in expectation. Now, a lot of people hear that and they think like, well, that also means that you're not going to win an expectation because you're just playing totally randomly.

• 46:31Speaker 02

  But in poker, if you play the equilibrium strategy, it's actually really difficult for the opponents to figure out how to tie you. And they're going to end up making the stakes that will lead you to win over the long run. It might not be a massive win, but it is going to be a win. If you play enough hands for a long enough period of time, you're you're going to win in expectation. Now, there's also exploitative poker and the idea here is that you're trying to spot weaknesses in how the opponent plays. You know, maybe they're, maybe they're not bluffing enough or maybe they fold too easily to a bluff. And so you start adapting from the game theory optimal balance strategy of like you bluff sometimes, you you don't bluff sometimes. To then playing a very unbalanced strategy that's like, oh, I'm just going to like bluff a ton against this person because they always fold whenever I bluff.

• 47:15Speaker 02

  Now, the key is that there's a trade off here because if you're taking this exploitative approach, then you're opening yourself up to exploitation as well. And so you have to choose this balance between playing a defensive game theory optimal policy that guarantees you're not going to lose, but might not make you as much money as you potentially could versus playing an exploitative strategy that can be much more profitable. But also it creates weaknesses that the opponents could take advantage of and trick you. And there's no way to perfectly balance the two. It's kind of like in rock, paper, scissors. If you notice somebody is like playing paper for five times in a row. You might think like, oh, they're they have a weakness in their strategy. I should just be throwing scissors and I'm going to take advantage of them. And so on the sixth time you throw scissors, but actually that's the time when they throw rock, you know, so and you never really know. So you always have this trade off.

• 47:58Speaker 02

  The poker AIs that have been extremely successful and like my background is like, I worked on AI for poker for several years during grad school and made the first superhuman no limit poker AIs. The approach that we took was this game theory optimal approach where the AIs would play this unbeatable strategy and they would play against the world's best and beat them. Now, that also means they they beat the world's worst, like they would just beat anybody. But if they were up against a weak opponent, they might not beat them as severely as a human expert might, because the human expert would know how to adapt from the game theory optimal policy to be able to exploit these weak players. And so there's this kind of unanswered question of like, how do you make an exploitative poker AI? And a lot of people had pursued this research direction. I had like dabbled in it a little bit during grad school.

• 48:43Speaker 02

  And I think fundamentally, it just comes down to AIs not being as sample efficient as humans, you know, we discussed earlier. If a human's playing poker, they're able to get a really good sense of the strengths and weaknesses of a player within a dozen hands. It's like honestly really impressive. And back when we were working on AI for poker in like the, you know, mid 2010s, you'd have to, these AIs would have to play like 10,000 hands of poker to like get a good profile of like who this player is, like how they're playing, where their weaknesses are. Now, I think with more recent technology, that has come down. But still, the sample efficiency has been a big challenge. Now, what's interesting is that after working on poker, I worked on diplomacy. I think we talked about this earlier. And diplomacy is this, you know, it's a seven-player negotiation game.

• 49:26Speaker 02

  And when we started working on it, I took a very game theory approach to the problem. I felt like, okay, we're, it's kind of like poker. You have to compute this game theory optimal policy and you just play this. You're gonna not lose an expectation, you're gonna win in practice. But that actually doesn't work in diplomacy. And it doesn't work, again, for a question of like, how much of a rabbit hole do we want to go down on this? But like, basically, when you're playing like the zero-sum games like poker, game theory optimal works really well. When you're playing a game like diplomacy where there's like, you need to collaborate and compete, and you need, there's there's room for collaboration, then game theory optimal actually doesn't work that well. And you have to understand the players and adapt to them much better.

• 50:06Speaker 02

  So this ends up being very similar to the problem in poker of like, how do you adapt to your opponents? In poker, it's about adapting to their weaknesses and take advantage of that. In diplomacy, it's about adapting to their play styles. It's kind of like, if you're at a table and everybody's speaking French, You don't want to just keep talking in English. You want to adapt to them and speak in French as well. That's the realization that I have with diplomacy that we need to shift away from this game theory optimal paradigm towards modeling the other players, understanding who they are, and then responding accordingly. And so in many ways, the techniques that we developed in diplomacy are exploitative. Like, they're not exploitative. They're really, you know, just adapting to the opponents, to the other players at the table.

• 50:49Speaker 02

  But I think the same techniques could be used in AI for poker to make exploitative poker AIs. If I didn't get, you know, AGI-pilled by the incredible progress that we were seeing with language models and like shifting my whole research agenda to focusing on like general reasoning, probably what I would have worked on next was making these like exploitative poker AIs. It would be a really fun research direction to go down. I think it's still there for anybody that wants to do it. And I think the key would be taking the techniques that we use in diplomacy and applying them to things like poker.

• 51:17Alessio Fanelli?

  I think to me, the core piece is when you play online, you have a HUD, which tells you, you know, all these stats about the other player and like, you know, how much they participate, pre-flap, blah, blah, blah. And to me, it's like, a lot of these models, from my understanding, are not really leveraging the behavior of the other players at the table. They're just kind of looking at the board state and kind of working from there.

• 51:36Speaker 02

  That's correct. The way the poker AIs work today, they're just kind of like sticking to their pre-computed

• 51:42Alessio Fanelli?

  GTO.

• 51:43Speaker 02

  GTO strategy, and they're not adapting to the other players at the table. And like, you can do various like kind of hacky things to get them to adapt, but, you know, they're not very principled. They don't work super well.

• 51:55Alessio Fanelli?

  Yep.

• 51:56Speaker 01

  Okay. Any grad students listening? If you want to work on that, I think that is a very, very reasonable research direction that'll at least get in front of you and, you know, get some attention at least.

• 52:06Speaker 02

  Yeah.

• 52:07Speaker 01

  The other thing that this conversation brings up for me is, yeah, well, one of the hypotheses for like, what is the next step after test time compute is world models? Is world modeling important or worthwhile research direction? Like, Yann LeCun has been talking about this nonstop. But like, basically, no LLMs have, like, they have internal world models, but like, not explicitly a world model.

• 52:29Speaker 02

  I think it's pretty clear that as these models get bigger, they have a world model, and that world model becomes better with scale. So they are implicitly developing a world model, and I don't think it's something that you need to Explicitly model. I could be wrong about that.

• 52:47Speaker 01

  When dealing with people or multi agents, it might be because you have entities that are not the world. And you're resolving hypotheses of what, which are the many types of entities you could be dealing with.

• 53:00Speaker 02

  You know, there was this like long debate in the multi agent AI community for a long time about, and it's still going on about whether you need to explicitly model other agents, like other, like other people. Or if they can be implicitly modeled as part of the environment. For a long time, I was like on the, took the perspective of like, of course you have to like explicitly model these other agents because they're behaving differently from the environment. Like they, they take actions, they're unpredictable. You know, they, they have agency. But I think I've actually shifted over time to thinking that like, actually, if these models become smart enough, they develop things like theory of mind. They develop an understanding that there are other agents that like can take actions and have motives and all this stuff. And these models just develop that implicitly with scale and more capable behavior broadly.

• 53:45Speaker 02

  So that's the perspective I take these days.

• 53:47Speaker 01

  So like what I just said was an example of a heuristic that is not bitter lesson filled. And you just, it just goes away.

• 53:53Speaker 02

  Yeah, it's really all come back to the bitter lesson.

• 53:56Speaker 01

  Got to cite them every, every podcast. So one of the interesting findings and most consistent findings, you know, I, I think you were at ICLR. And one of the hit talks there was about open endedness. And this guy, Tim, who gave that talk has been doing a lot, bunch of research about multi agent systems too. One of the most consistent findings is always that it's better for AI to self play and improve competitively. As opposed to sort of humans training and guiding them. And you find that with like, you know, Alpha Zero and R one, zero, whatever that was. Do you think this will hold for multi agents, like self play to improve better than humans?

• 54:33Speaker 02

  Yeah. So, okay. So this is a great question and I, I think this is like worth expanding on. So I think a lot of people today see self play as like the next step. And maybe the last step that we need for super intelligence. And I think if you're following, you know, you look at something like Alpha, Alpha Go and Alpha Zero. We seem to be following a very similar trend, right? Like. The first step in AlphaGo was you do large-scale pre-training. In that case, it was on human Go games. With LLMs, it's pre-training on, you know, tons of like internet data. And that gets you a strong model, but it doesn't get you, you know, an extremely strong model. You know, it doesn't get you a superhuman model. And then the next step in the AlphaGo paradigm is you do large-scale test time compute, or like large-scale inference compute. And in that case with MCTS.

• 55:18Speaker 02

  And now we have like reasoning models that also do like this large-scale inference compute. And again, that like boosts the capabilities a ton. Finally, with AlphaGo and AlphaZero, you have self-play, where the model plays against itself, learns from those games, gets better and better and better, and just like goes from something that's like around human-level performance to like way beyond human capability. It's like these Go policies now are so strong that it's just like incomprehensible. Like what they're doing is incomprehensible to humans. Same thing with chess. And we don't have that right now with language models. And so it's like, it's really tempting to look at that and say like, oh, well, we just need these like AI models to now interact with each other and learn from each other, and then they're just gonna like get to superintelligence. The challenge, and I kind of mentioned this like a little bit when I was talking about diplomacy, the challenge is that

• 56:02Speaker 02

  Go is this two-player zero-sum game. And two-player zero-sum games have this very nice property where when you do self-play, you are converging to a minimax equilibrium. And I guess I should take a step back and say like, in two-player zero-sum games, two-player zero-sum games are chess, Go, even two-player poker, all two-player zero-sum. What you typically want is what's called a minimax equilibrium. This is that GTO policy. This policy that you play, where you're guaranteeing that you're not going to lose to any opponent in expectation. I think in chess and Go, that's like pretty clearly what you want. Interestingly, when you look at poker, it's not as obvious. In a two-player zero-sum version of poker, you could play the GTO minimax

• 56:45Speaker 02

  policy, and that guarantees that you won't lose to any opponent on Earth. But again, I mentioned, you're not going to beat a weak player. You're not going to make as much money off of them as you could if you instead played an exploitative policy. So there's this question of like, what do you want? Do you want to make as much money as possible? Or do you want to guarantee that you're not going to lose to any human alive? What all the bots have decided is like, well, what all the AI developers in these games have decided is like, well, we're going to choose the minimax policy. And conveniently, that's exactly what self-play converges to. If you have these AIs play against each other, learn from their mistakes, they converge over time to this minimax policy, guaranteed. But once you go outside of two players, zero-sum games, like in the case of diplomacy, that's actually not a useful policy anymore.

• 57:29Speaker 02

  You don't want to just like have this very defensive policy. And you're going to end up with really weird behavior if you start doing the same kind of self-play in things like math. So for example, what does it mean to do self-play in math? You could fall into this trap of like, well, I just want one model to pose really difficult questions and the other model to solve those questions. You know, that's like a two-player zero-sum game. The problem is that like, well, you could just like pose really difficult questions that are not interesting. You know, you could just like get, ask it to do like 30-digit multiplication. It's a very difficult problem for the AI models. Is that really making progress in the dimension that we want? Like, not really. So self-play outside of these two-player zero-sum games

• 58:13Speaker 02

  becomes like a much more difficult nuanced question. So I think, and Tim kind of like basically said something similar in his talk, that there's a lot of challenges in really deciding what you're optimizing for when you start to talk about self-play outside of two-player zero-sum games. My point is that like, this is where the AlphaGo analogy breaks down. And not necessarily breaks down, but like, it's not going to be as easy as self-play was in AlphaGo.

• 58:39Speaker 01

  What is the objective function then for that? What is the new objective function?

• 58:44Speaker 02

  Yeah, it's a good question, yeah. And I think that that's something that, you know, a lot of people are thinking about.

• 58:49Speaker 01

  Yeah, I'm sure you are.

• 58:52Speaker 01

  One of the last podcasts that you did, you mentioned that you were very impressed by Sora. You don't, you don't work directly on Sora, but obviously it's part of OpenAI. I think the, the most recent new updates or in that sort of generative media space is autoregressive image gen. Is that interesting or surprising in any way that you wanna comment about?

• 59:10Speaker 02

  I don't work on image gen, so like my ability to comment on this is kind of limited, but I will say like, I, I love it. Like, I think it's super impressive. It's like one of those things where, you know, you work on these reasoning models and you think like, wow, we're gonna like, be able to do all sorts of crazy stuff, like advanced science and, um, you know, solve the gigantic tasks and, and software engineering. And then there's like this whole other like dimension of progress where you're like, oh, you're able to like make images and videos now. And it's like so much fun and that's getting a lot more of the attention, to be honest, especially in the general public. And it's probably driving a lot more of the like, you know, subscription plans for ChatGPT, which is, is great. But I think it's just kind of funny that like, yeah, we're also, I promise, we're also working on super intelligence.

• 59:50Speaker 01

  But you can make everything good, Lee. I, I think the, the delta for me was, um, I was actually harboring this thesis that diffusion was over because of autoregressive image and like there were rumors about this end of last year. And obviously now it's come, now it's come out. Then Gemini comes out with text diffusion and like diffusion is so back. And like this is two directions and it's very relevant for inference of autoregressive versus diffusion. Do we have both? Does one win?

• 1:00:19Speaker 02

  The beauty of research is like, you know, you gotta pursue different, different directions and it's not, it's always gonna be clear, like what is, um, you know, the promising path like? And I think it's great that people are looking into different directions and trying different things. I, I think that there's a lot of value in that exploration and I think we all benefit from seeing what works.

• 1:00:38Speaker 01

  Any potential in diffusion reasoning? Let's say you're.

• 1:00:42Speaker 02

  Probably can't answer that.

• 1:00:43Speaker 01

  Okay.

• 1:00:43Alessio Fanelli?

  So you did a masters in robotics too. Would love to get your thoughts on one, you know, OpenAI kind of started with the pen spinning trick and like the robotic arm they wanted to build. Is it right to work on this humanoid likes? Do you think that's kind of like the wrong embodiment of AI outside of the usual, you know, how long until we get? Robots, blah, blah, blah. Is there something that you think is like fundamentally not being explored right now that people should really be doing in robotics?

• 1:01:09Speaker 02

  I did a masters in robotics years ago and my takeaway from that experience, first of all, I didn't actually work with robots that much. I was like technically in a robotics program. I played around with some Lego robots my, my first week of the program, but then honestly, I just like pretty quickly shifted just working on AI for poker. And um, was kind of nominally in the robotics masters. But my takeaway from like interacting with all these roboticists and seeing their research was that I did not want to work on robots. Because the research cycle is so much slower and so much more painful when you're dealing with like physical hardware. Like software goes so much more quickly. And I think that's why we're seeing so much progress with language models and like all these like virtual coworker kind of tasks, but haven't seen as much progress in robotics that like

• 1:01:53Speaker 02

  physical hardware just is much more painful to iterate on. On the question of humanoids, I don't have very strong opinions here because this isn't what I'm working on. But I think there's a lot of value in non-humanoid robotics as well. I think drones are a perfect example where like, there's clearly a lot of value in that. Is that a humanoid? No. But in many ways, that's great. You know, like you don't want a humanoid for, for that kind of technology. I think weekly, um, I think that's non-humanoids provide a lot of value.

• 1:02:23Alessio Fanelli?

  I was reading, uh, Richard Heming's art of doing science and engineering and he talks about how when you have a new technological shift, people try and take the old workloads and like replicate them just in the new technology versus you actually have to change the way you do it. And you know, when I see this video of like, you know, your humanoid in the house, it's like, well, the human shape is kind of, has a lot of limitations that could actually be improved. But I think people, what's familiar, you know, it's like, would you put a robot with like 10 arms and like, you know, five legs in your house or would they be eerie? And like, when you get up and you see that thing walking around and is that why we use humanoid? So I, I think to me, there's almost like this local maximum of like, you know, we gotta make it look like a human. But I think like, what's like the, the best shape in-house?

• 1:03:08Speaker 02

  I am terrible at product design, so I, I am not the person to ask on this. I think there is a question of like, is it better to make humanoids because they're more familiar to us? Or is it worse to make humanoids because they're more similar to us, but not quite identical? Like, I, I don't know which one I would actually find creepier.

• 1:03:25Speaker 01

  Yeah. The thing that got me humanoid pilled a little bit was just the argument that most of the world is made for humans anyway. So if you want to replace human labor, you have to make a humanoid. I don't know if that's convincing.

• 1:03:39Speaker 02

  Again, I don't have very strong opinions in this field because like I don't work in it. I was like weakly in favor of humanoids and I think what really persuaded me to be weakly in favor of like non-humanoids was listening to the Physical Intelligence CEO and like some of his pitches about like why they're not pursuing, why they're pursuing like non-humanoid robotics. And conveniently their office is actually like very close to here. So if you wanted to.

• 1:04:00Speaker 01

  They're speaking at the conference I'm running.

• 1:04:02Speaker 02

  Okay. So I'm looking forward to that. I'd say like, listen to his pitch and maybe he can convince you that non-humanoid is the way to go.

• 1:04:08Speaker 01

  Awesome. The other one I refer people to is Jim Phan recently did a talk on the physical tearing tests, which he did at the Sequoia Conference, which was very, very good. He's such a great educator and explainer of things. It's very hard, especially in that field. Cool. What we're done asking you about things that you don't work on. So these are just more rapid fires to sort of explore some of your boundaries and get some quick hits. How do you or top industry labs keep on top of research? Like, what are your tools and practices?

• 1:04:39Speaker 02

  It's really hard. I think that a lot of people have this perception that like academic research is irrelevant and it's actually not the case. I think that we do, we look at academic research. I think one of the challenges is like a lot of academic research shows promise in their papers, but then actually doesn't work at scale or even doesn't replicate. I think if we find interesting papers, like we're going to try to reproduce that in-house and see if it like still holds up and then also does it scale well. But that is like a big source of inspiration for us.

• 1:05:10Speaker 01

  Whatever hits archive, literally you do the same as the rest of us. Or do you have like a special process?

• 1:05:15Speaker 02

  Especially if I get recommendations, like we have an internal.

• 1:05:17Speaker 01

  It's a word of mouth.

• 1:05:18Speaker 02

  Channel where people will post interesting papers. And like, I think that's a good source of like, okay, well, this person that is more familiar with this area thinks that this paper is interesting, so therefore I should read it. And similarly, like I'll keep track of things that are happening in my space that I think are interesting. And like, if I think it's really interesting, maybe I'll share it.

• 1:05:34Speaker 01

  For me, it's like WhatsApp and Signal group chats with researchers and that's it.

• 1:05:37Speaker 02

  Yeah. I think it is like, I mean, a lot of people look at things like Twitter and I think it's really unfortunate that we've reached this point where things need to get a lot of attention on social media for it to be paid attention to.

• 1:05:50Speaker 01

  That's what the grad students are trained, they're taking classes to do this.

• 1:05:54Speaker 02

  I do recommend to like, you know, I've worked with grad students, I work with fewer now because we don't publish as much. But when I was at FAIR, publishing papers, like I would tell the grad students I was working with that like, you need to post it on Twitter.

• 1:06:06Speaker 01

  Yeah.

• 1:06:06Speaker 02

  And you need to, and we go over like the Twitter thread about like how to present the work and everything and there's a real art to it and it does matter and it's kind of the sad truth.

• 1:06:15Alessio Fanelli?

  I know when you were doing the ACPC, like the AI poker competition, you mentioned that people were not doing search because they were limited to like two CPUs at inference. Do you see similar things today that are like keeping interesting research from being done? That might be, it's not as popular, it doesn't get you into the top conferences, like are there some environmental limiters?

• 1:06:37Speaker 02

  Absolutely, and I think one example is for benchmarks that you look at things like humanities last exam, like you have these incredibly difficult problems, but then are still very easily gradable. And I think that actually limits the scope of what you can evaluate these models on if you, if you stick to that paradigm. It's very convenient because, you know, it's very easy to like then score the models. But actually, a lot of the things that we want to, you know, evaluate these models on are kind of like more fuzzy tasks that are not multiple choice questions. And making benchmarks for those, for those kinds of things is so much harder. And probably also like a lot more expensive to evaluate. But I think that those are really valuable things to work on.

• 1:07:14Alessio Fanelli?

  And that would fit the same moment, GPT four point five is like a high taste model in a way, there's kind of like all this like non measurable things about a model that are really good that maybe people are not.

• 1:07:26Speaker 02

  Well, I think there are things that are measurable, but they're just like much more difficult to measure. And I think that a lot of benchmarks have kind of stuck to this paradigm of posing really difficult problems that are really easy to measure.

• 1:07:37Alessio Fanelli?

  Yep.

• 1:07:37Speaker 01

  So let's say that the pre-training scaling paradigm took about five years from like discovery of GPT to scaling it up to GPT four. And then we give you, we give test time compute five years as well. So if test time compute hit a wall by 2030, what would be the probable cause?

• 1:07:54Speaker 02

  It's very similar to pre-training where like you can push pre-training a lot further. It just becomes more expensive with each iteration. I think we're going to see something similar with test time compute. We're like, okay, we're going to get them thinking instead of three minutes, they're for three hours and then three days and then three weeks.

• 1:08:09Speaker 01

  You run out of human life.

• 1:08:10Speaker 02

  Well, there's two, there's two, there's two concerns. One is that it becomes much more expensive to get the models to like think for that long or like scale up test time compute. Like as you scale up test time compute, you're spending more on test time compute, which means that like there's a limit to how much you could spend. That's one potential ceiling. Now, obviously, but not obviously, but like I should say that we're also becoming more efficient. These models are becoming more efficient in the way they're thinking is they're able to do more with the same amount of test time compute. And I think that's a very underappreciated point that it's not just that we're getting these models to think for longer. In fact, if you look at oh, three, it's thinking for longer than oh, one preview for some questions, but it's not like a radical difference, but it's way better. Why? Because it's just like becoming better at thinking. Anyway, yeah, these models, um, you're going to scale up test and compute. You can only scale it up so much like that becomes a soft barrier in the same way that pre-training.

• 1:08:55Speaker 02

  It's becoming more and more expensive to train better and better pre-trained models or bigger pre-trained models. The second point is that like, as you have these models think for longer, you kind of get bottlenecked by walk clock time. Like if you want to iterate on experiments, it is really easy to iterate on experiments when these models would respond instantly. It's actually much harder when they take three hours to respond. And what happens when they have three weeks? It takes you at least three weeks to do those evaluations and to then iterate on that. And a lot of this, you can paralyze experiments to some extent, but a lot of it, you have to run the experiment, complete it, and then see the results in order to decide on the next set of experiments.

• 1:09:31Speaker 02

  I think this is actually the strongest case for, for long timelines that the models, because they just have to like do so much in serial time, we can only iterate so quickly.

• 1:09:42Speaker 01

  How would you overcome that wall?

• 1:09:43Speaker 02

  It's, it's a challenge and I think, I think it depends on the domain. So drug discovery, I think is one domain where this could be a real bottleneck. I mean, if you want to see if something like extends human life, it's going to take you a long time to figure out if like this new drug they developed like actually extends human life and doesn't have like terrible side effects along the way.

• 1:10:00Speaker 01

  Side note, do we not have perfect models of human chemistry and biology by now?

• 1:10:04Speaker 02

  Well, so this, this is, I think the thing, and again, I want to be cautious here because I'm not actually a biologist or chemist. Like I don't, I know very little about, about these fields. Last time I took a biology class was 10th grade in high school. I don't think that there's a perfect simulator of human biology right now. And I think that that's something that. Could potentially help address this problem.

• 1:10:22Speaker 01

  Like it's like the number one thing that we should all work on.

• 1:10:25Speaker 02

  Well, that's one of the things that we're hoping that these recent models will help us with.

• 1:10:27Speaker 01

  Yeah. How would you classify mid-training versus post-training today?

• 1:10:32Speaker 02

  It's such, all these definitions are so fuzzy. So I, I don't have, I don't have a great answer there.

• 1:10:38Speaker 01

  It's a question people have and you're like opening eyes like now explicitly hiring for mid-training. And everyone is like, what the hell is mid-training?

• 1:10:46Speaker 02

  I think mid-training is between pre-training and post-training. It's like, it's like, uh, it's not, it's not post-training. It's not pre-training. It's like adding more to the models. But like after pre-training, like

• 1:10:59Speaker 01

  Interesting ways.

• 1:11:00Speaker 02

  Yeah.

• 1:11:01Speaker 01

  Okay. All right. Well, you know, I was trying to get some clarity.

• 1:11:06Alessio Fanelli?

  Is the pre-trained model now basically like a, just an artifact that then spawns other models? And it's almost like the core pre-training model is never really exposed anymore. And it's the mid-training, the new pre-training, and then there's the post-training, once you have the models branched out.

• 1:11:23Speaker 02

  You never interact with an actual, just like raw pre-trained model. Like if you're going to interact with the model, it's going to go through mid-training and post-training. So, um, so you're seeing the final product.

• 1:11:31Speaker 01

  Well, you don't let us do it, but you know, we used to.

• 1:11:33Speaker 02

  Well, yeah. I mean, I guess you, you know, there's open source models where you can just like interact with the raw pre-trained model. But for, for OpenAI models, like they go through a mid-training step and then they go through a post-training step and then, and then they're released. And they're a lot more useful. Like frankly, if you interact with the only pre-trained model, it'd be super difficult to work with and it would,

• 1:11:49Speaker 01

  Yeah.

• 1:11:49Speaker 02

  it would seem kind of dumb.

• 1:11:51Alessio Fanelli?

  Yeah.

• 1:11:51Speaker 01

  But it'd be, it'd be useful in weird ways, you know, because there's a mode collapse when you, when you post-train for it for like chat.

• 1:11:59Speaker 02

  Yeah. In some ways you want that mode collapse. Like you want, you want that collapse of like

• 1:12:02Speaker 01

  Yes, to be useful. Yeah, I get it. Yeah. We're interviewing Greg Brockman next. Uh, you've talked to him a lot. What would you ask him?

• 1:12:11Speaker 02

  What would I ask Greg? I mean, I mean, I get to ask Greg all the time. What should you ask Greg?

• 1:12:14Speaker 01

  Like to evoke an interesting response that like, uh, not he doesn't get asked enough about, but you know, like this is something that he's passionate about, or you just want his thoughts.

• 1:12:25Speaker 02

  I think in general, it's worth asking where this goes, you know, like, what does the world actually look like in five years? What does the world look like in 10 years? What does that distribution of outcomes look like? And what could the world or individuals do to help steer things towards like the good outcomes instead of the negative outcomes?

• 1:12:46Speaker 01

  Okay, like an alignment question.

• 1:12:48Speaker 02

  I think people get very focused on what's going to happen in like one or two years. And I think it's also worth spending some time thinking about like, well, what happens in five or 10 years? And what does that world look like?

• 1:12:59Speaker 01

  I mean, he doesn't have a crystal ball.

• 1:13:02Speaker 02

  But he certainly has, he certainly has thoughts. Yeah. So I think that's worth exploring.

• 1:13:08Speaker 01

  Okay.

• 1:13:08Alessio Fanelli?

  What are games that you recommend to people, especially socially?

• 1:13:12Speaker 02

  What are games that I recommend to people? I've been playing a lot of this game called Blood on the Clock Tower lately.

• 1:13:18Speaker 01

  What is it?

• 1:13:19Speaker 02

  It's kind of like Mafia or Werewolf. It's become very popular in San Francisco.

• 1:13:24Speaker 01

  Oh, that's the one played in your house?

• 1:13:26Speaker 02

  Yeah.

• 1:13:26Speaker 01

  Okay, got it.

• 1:13:27Speaker 02

  It's kind of funny because like I was talking to a couple of people now that had told me that it used to be that poker was the like way that like the VCs and tech founders and stuff would socialize with each other. And actually now it's shifting more towards Blood on the Clock Tower. Like that's the, that the thing that people use to like, um, you know, connect in the Bay Area. And I was actually told that a startup held a recruiting event that was a Blood on the Clock Tower game.

• 1:13:54Speaker 01

  Wow.

• 1:13:55Speaker 02

  Yeah. So, uh, I guess it's like, it's really catching on, but it's a fun game. And I guess you lose less money playing it than you do playing poker. So it's like better for people that are not very good at these things. Uh, I think it's kind of like a weird recruiting event, but certainly a fun game.

• 1:14:08Speaker 01

  What qualities make a winner here that is interesting to hire for?

• 1:14:13Speaker 02

  That's the thing is like, okay, I guess you get

• 1:14:15Speaker 01

  Ability to lie.

• 1:14:16Speaker 02

  Deception and like picking up on deception. Like, is that the best employee? I don't know.

• 1:14:23Alessio Fanelli?

  So my slight final pet topic is Magic the Gathering. So you have, we talked about some of these games, Chessco, and they have perfect information. Then you have poker, which is imperfect information in a pretty limited universe. You only have a 52 card deck. And then you have these other games that have imperfect information, like a huge pool of possible options. Do you have any idea of like, how much harder that is? Like, how does the difficulty of this problem scale?

• 1:14:49Speaker 02

  I love that you asked that because I have this like huge store of knowledge on AI for imperfect information games. Like, this is my area of research for so long. And I know all these things, but I don't get to talk about it very often. We've made superhuman poker AIs for No Limit Texas Hold'em. One of the interesting things about that is that like, the amount of hidden information is actually pretty limited. Because you have two hidden cards when you're playing Texas Hold'em. And so the number of possible states that you could be in is 1,326, when you're playing heads up at least. And, you know, that's multiplied by the number of other players that there are at the table, but it's still like, not a massive number. And so the way these AI models work is that you enumerate all the different states that you could be in.

• 1:15:32Speaker 02

  So if you're playing like six handed poker, there's five other players, five times 1,326. That's the number of states that you'd be in, and then you assign a probability to each one. And then you feed those probabilities into your neural net and you get actions back for each of those states. The problem is that as you scale the number of hidden possibilities, like the number of possible states you could be in, that approach breaks down. And there's still this very interesting unanswered question of what do you do when the number of hidden states becomes extremely large? You know, so if you go to Omaha poker, where you have four hidden cards, there are things you could do that's kind of like, that are kind of heuristic that you could do to reduce the number of states. But actually, it's still a very difficult question. And then if you go to a game like Stratego, where you have 40 pieces, so there's like,

• 1:16:16Speaker 02

  close to 40 factorial different states you could be in, then all these like existing approaches that we used for poker kind of break down. And you need different approaches. And there's a lot of active research going on about like, how do you cope with that? So for something like Magic the Gathering, the techniques that we used in poker would not out of the box work. And it's still an interesting research question of like, what do you do? Now, I should say this becomes a problem when you're doing the kinds of search techniques that we used in poker. If you're just doing Model free RL, it's not a problem. And my guess is that if somebody put in the effort, they could probably make a superhuman bot for Magic the Gathering now. Yeah, there's still some unanswered research questions in that space. Now, are they the most important unanswered research questions? Right.

• 1:16:58Speaker 02

  I'm inclined to say no. I think there's like, the problem is that like, the techniques that we used in poker to do this kind of search stuff, were pretty limited. And like, if you expand, if you expand those techniques, maybe you get them to work on things like Stratego and Magic the Gathering, but they're still going to be limited. They're not going to get you like, superhuman encode forces with language models. So I think it's more valuable to just focus on the very general reasoning techniques. And one day, as we improve those, I think we'll have a model that just out of the box, one day plays Magic the Gathering at a superhuman level. And I think that's the more important and more impressive research direction. Cool. Amazing.

• 1:17:31Alessio Fanelli?

  Yeah. Thanks so much for coming on, Noam.

• 1:17:34Speaker 02

  Yeah, thanks for your time. Yeah, thanks. Thanks for having me.