Ep 49: OpenAI Researcher Noam Brown Unpacks the Full Release of o1 and the Path to AGI

Noam Brown, renowned AI researcher and key figure at OpenAI, joins us for a deep dive into the o1 release. Recorded just one day before o1’s full public debut, this episode explores the groundbreaking advancements and challenges behind this innovative test-time compute model. We discuss the technical breakthroughs that set o1 apart, its unique capabilities compared to previous models, and how it disrupts traditional paradigms in AI development. Noam also shares insights into OpenAI’s approach to innovation, the economic realities of scaling AI, and what the future holds for the field.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Noam Brown is a research scientist at OpenAI, where he was a key part of their work on O1. Noam is at the forefront of reasoning in LLMs, had a really interesting past track record at FAIR, where he worked on problems in diplomacy and poker. And we hit on the biggest questions in LLMs today on unsupervised learning. We talked about whether these models are hitting a wall, how far test time compute can scale. We hit on how Noam defines AGI and what he's changed his mind on in the last few years in AI research. This was a really fun one to do right after the general release of O1, and I think folks are really going to enjoy it. Without further ado, here's Noam.

• 0:30Jacob Effron?

  Well, Noam, thanks so much for coming on the podcast.

• 0:32Speaker 01

  Of course, great to be here.

• 0:33Jacob Effron?

  I've been looking forward to this one for a while and certainly well timed with some exciting launches going on with Chitmish.

• 0:39Speaker 01

  Uh huh, yeah, I'm looking forward to it. We're going to be releasing O1 tomorrow, which I guess by the time this podcast is out, it's already going to be out there. I'm pretty excited for it. I think the community is going to love it, but I guess we'll see.

• 0:50Jacob Effron?

  Well, I'd be remiss not to start around like what I feel like has been the question of the past month around, you know, have we hit a wall with model capabilities? And I think, you know, there's obviously different parts of that question. And so maybe to start would just be the extent to which you feel like there's still more juice to squeeze on scaling pre-training.

• 1:07Speaker 01

  So my view on this, and I've been pretty public about this. I think there's more room to push across the board, and I think that includes pre-training. I think the right way to think about it is that every time you want to scale these models further, there's like, there's a cost to that. And so you look at GPT-2, you know, it costs like between $5,000 and $50,000, depending on how you like measure it. You look at GPT-4, like obviously there's a lot of improvements, but the fundamental thing, the most important thing that's changed is the amount of resources that have gone into it. And so you go from spending for frontier models, like thousands to tens of thousands of dollars to, you know, hundreds of thousands to millions to tens of millions to, you know, for some labs, possibly hundreds of millions of dollars today.

• 1:52Speaker 01

  And the models keep getting better. And I think that that will continue to be true, that if you throw more money, more resources, um, more data, all this stuff into that, you're gonna get a better model. Um, the problem is that, okay, well, if you wanna like 10x it each time, then at some point that becomes like an intractable cost. And so, okay, the next, if you wanna make it, you know, better, you wanna do another 10x, now you're talking about billions of dollars. You wanna do another 10x, you're talking tens of billions of dollars. And at some points, it's no longer economically um, worth it to, to push that further. So you're not gonna spend, presumably, you're not gonna spend trillions of dollars on a model.

• 2:31Speaker 01

  Um, and so, there's no hard wall, it's more of like a soft wall, that eventually, the economics just don't work out for it.

• 2:39Jacob Effron?

  Right, and it seems like, obviously, there's, you know, uh, in many ways, you're able to push this forward with test time compute in like a, you know, like there's lower hanging fruit there from a cost perspective to push that forward.

• 2:48Speaker 01

  Exactly, and so this, this is why I'm really excited about test time compute, and I think, um, why, like, you know, a lot of people are excited about it, is that we're still, it's kind of like, we're back in the GPT-2 days, like when GPT-2 was figured out, and like the scaling laws were figured out, it was pretty obvious that like, oh, you just scale this up by 1,000x, and you're gonna get a better model. Um, and you could do that. It's a little harder now to scale things up by 1,000x when pre-training, but with test time compute, we're still pretty early, and so we have a lot of room, a lot of runway to scale that further. There's a lot more low hanging fruit for algorithmic improvements, so I think there's just a lot of exciting stuff to be done in that direction. That's not to say that, that pre-training is done, but um, it's just that there's so much more uh, headway to just like, push the test time compute paradigm further.

• 3:30Speaker 01

  And I should also say, like, you know, it's not like, even with, even going back to pre-training for a second, it's not like, um, you know, there's, uh, you know, two orders of magnitude or something that you can push it, and then you're done. You're still gonna be Moore's law, the costs are gonna continue to come down, it's just like a question of like, um, how quickly can you, can you scale? Like, there is this huge overhang, where it's very easy to scale it very quickly, um, and that's becoming like, a little bit less true.

• 3:56Jacob Effron?

  I realize this is probably an overly broad question, but like, how high is the ceiling on test time compute? Like, how do you think about, you know, where that could go?

• 4:04Speaker 01

  Again, I think about it in terms of dollar value. So, you know, how much does a ChatGPT query cost today? Um, ballpark, a penny.

• 4:13Speaker 01

  What cost could you spend on a query that you care a lot about? And like, what cost would you be willing to pay? I think there are some problems out there that people would be willing to pay a lot of money for. And I'm not talking about like a dollar or five dollars. I'm talking like a million dollars for some of the most important problems that society cares about. So how many orders of magnitude is that? Like, that's what eight orders of magnitude? So I think there's a lot of room to push it further. And I also think there's a lot of room for algorithmic improvements. So it's not just like, oh, we're just gonna like dump more money into the query and then you get a better output. It's like, no, we actually, we can improve this paradigm further and make this scaling a lot better.

• 4:51Jacob Effron?

  You know, one thing I thought was interesting is, I guess, maybe a month ago, you know, Sam Altman had tweeted, you know, we basically know what we've got to do to build AGI. And I think you tweeted like his view matches like the median view of of OpenAI researchers today. Can you say more about that? Because obviously, you know, there's so many people now talking like, oh, we've hit a wall. Like, what do you think they're missing?

• 5:09Speaker 01

  I feel like we've been pretty open about this, that we see things continue to progress pretty rapidly. I think that that's my opinion. I think that Sam expresses his opinion. And I think, you know, I've heard some people say that like, oh, Sam is just trying to like, you know, create hype or something. And I'm kind of surprised by that because like, we're saying the same thing. And, you know, yeah, I think it's a common opinion in the company that things are going to progress quickly.

• 5:37Jacob Effron?

  And do you think like pre-training and test time compute alone kind of gets you most of the way there? Or is there also, it seems like this algorithmic bucket as well?

• 5:44Speaker 01

  I it's not by any means that we've like, we're done. It's like we've cracked the code day to superintelligence and now we just have to like, you know, scale.

• 5:51Jacob Effron?

  That'd be pretty cool if you came on the pod and announced that you had that.

• 5:54Speaker 01

  But I think, okay, so the way the way that I think about it, back in 2021, like late 2021, I had coffee with Ilya Sutskever. And, you know, he was asking me about my AGI timelines. And I told him like, you know, to be honest, like, I think it's going to take a very long time. I'm pretty skeptical that we'll get there within the next 10 years. And the main reason that I gave him was that we don't have a general way of scaling inference compute, a scaling test on compute. Like, I saw how much of a difference that made when it came to games and the fact that it wasn't there in language models in a very general way.

• 6:32Speaker 01

  Just to me, it seemed kind of like silly that we're gonna get to superintelligence just by scaling pre-training. Because you know, you look at these models and like, yeah, they're doing pretty smart things, but also, you know, back then, they couldn't even draw a tic-tac-toe board, you know? And like, yes, you get to GPT-4, and sort of they can draw the board and like make mostly legal moves, but sometimes they still make illegal moves, they make suboptimal decisions in tic-tac-toe. And like, I have no doubt that if we scale pre-training another order of magnitude or two, it's gonna start playing tic-tac-toe really well. But like, if that's the state of things, that we're spending like, tens of billions of dollars to train a model, and it can barely play tic-tac-toe, you know, that's like, pretty far from superintelligence. So, and I told him, like, "Look, we're not gonna get to, uh, superintelligence until we can figure out how to scale inference computer very general way."

• 7:17Speaker 01

  And I think that that's an extremely hard research problem, and it's gonna take, you know, probably at least a decade to figure it out.

• 7:24Speaker 01

  To my surprise, by the way, he agreed with me. Like, he agreed that, um, scaling pre-training alone would not get to superintelligence. And I think, uh, I didn't realize it at the time, but he was also like, thinking about, um, very, very carefully, like, this scaling, uh, test-time compute direction.

• 7:42Speaker 01

  So, I thought it would take at least a decade. It took like, two or three years.

• 7:49Speaker 01

  So, and I thought that was the hardest unsolved research question at the time. I have no doubt that there are others, and in fact, I know that there are other, um, problems that aren't solved, research questions that aren't solved. But I don't think that any of them are gonna be harder than the problems that we've already solved.

• 8:07Jacob Effron?

  Yeah.

• 8:08Speaker 01

  And, uh, for that reason, I think that things will continue to progress.

• 8:12Jacob Effron?

  Obviously, you've had just a massive impact in, uh, in this test-time compete work. And, you know, your research career had obviously been in search and planning, you know, games like poker and diplomacy. And, I mean, from others' accounts, it seems like when you joined OpenAI, you were pretty clear, like, this was the direction to push in. Um, it seems to have really paid off. I'm curious, like, how consensus was that approach when you joined, like, you know, uh, you know, maybe talk about, like, getting the kind of research organization oriented behind that?

• 8:38Speaker 01

  Yeah, it's interesting. When I went on the job market and I was like, you know, interviewing at a bunch of places, people in general were quite receptive to the idea that like, okay, like the research labs, I think everybody actually, for the most part, was among the frontier research labs, believed that pre-training alone would not get us, the current paradigm would not get us to superintelligence and that there was something else that was needed. So there was a lot of reception to this idea of, okay, yeah, maybe we need to figure out how to scale, test on compute. Some labs were more bought into it than others, and I was actually kind of surprised that OpenAI was really, really on board with this. Because, you know, they're the ones that pioneered large-scale pre-training and scaled it farther than anybody else.

• 9:19Speaker 01

  And but they were very on board with it, and I didn't know it at the time when I was talking with them, that they had also been thinking about this for a while before I joined. So when I did join, you know, it's interesting because I think the motivation was was different.

• 9:39Speaker 01

  The motivation that they had in mind was more about like overcoming the data wall, not so much about we need to figure out how to scale, test on compute. It's more about like, how do we like get over the data wall?

• 9:52Speaker 01

  But the techniques ended up being, you know, the agendas ended up being pretty compatible. And yeah, so it actually wasn't too hard to get, you know, look, when we started, it was still this like

• 10:08Speaker 01

  exploratory research direction.

• 10:11Speaker 01

  And there were some people working on it, but it wasn't like, you know, half the company is dedicated towards this like large-scale effort by any means.

• 10:19Speaker 01

  But a few months after I joined, you know, I and various other people were trying things that some, many of which didn't work. But, you know, there was one thing that one person tried that like ended up getting some signs of life. And people were like, oh, that seems interesting. Maybe we should like, you know, try some more things and like, you know, get more and more signs of life. And And eventually, like, I think the leadership recognized that, like, okay, there's actually something here that seems different and valuable, and we should really scale this up.

• 10:50Jacob Effron?

  Yeah.

• 10:50Speaker 01

  And I was supportive of that, but I think others were, and I think it was like a testament to actually OpenAI and its organizational excellence that it was able to recognize that there was a lot of potential here and was willing to invest a lot to scale it up. I mean, I think people were like, really under, I think it's like an underappreciated point, that in many ways, it's really surprising that something like O1 came out of OpenAI. It's disruptive to the paradigm that OpenAI, you know, it's disruptive to the paradigm that OpenAI pioneered. And I think it's like a really good sign that OpenAI isn't getting trapped in the, you

• 11:34Speaker 01

  know, innovator's dilemma, and is willing to invest in a risky direction. And I think in this case, it's going to pay off.

• 11:42Jacob Effron?

  Yeah, no, it's really interesting, because obviously, if the, you know, script had continued to play out of just scaling, pre-training, you know, continuously and raising more money to do that, OpenAI has a great position to do that. And so, any sort of like orthogonal approach, you know, yeah, is different. And so, it's cool that it came out of the same place. Obviously, your original timeline was, hey, it's going to take 10 years to do this. You did it in two. What was the first thing you saw that kind of were like, yeah, okay, actually, this might be way faster than I thought?

• 12:06Speaker 01

  So first of all, it was not just me, like, a lot of, it was me and a lot of other people that managed to do it in a shorter period of time than I predicted. What's the first thing that I saw? I think when I joined, we had a lot of discussions about the kinds of behavior that we would like a, we would like the model to do. And that included things like, we want to be able to see it, try different strategies to solve a problem if like, a strategy isn't working out. We want to see it take a hard problem that involves many steps and break that steps, break that problem down into smaller, into smaller pieces that it can like tackle one by one. We want to see it recognize when it's making a mistake and correct those mistakes or avoid making it in the first place.

• 12:47Speaker 01

  And there were a lot of discussions around how do we get those individual things? And that kind of bothered me, the fact that we would like even try to tackle them individually because it just seems like, okay, well, ideally, we just get something that like figures out all this stuff on its own. And, you know, we got the initial signs of life. And, and then, um, you know, one of the things that, that we tried, that I was a big fan of, uh, the, uh, yeah, that I advocated for was like, why don't we just have it like think for longer?

• 13:15Jacob Effron?

  Yeah.

• 13:16Speaker 01

  And when we had it think for longer, it would just do these kinds of things emergently. It, it could, it wasn't like, um, it wasn't like, oh, suddenly we have like a one, but it was like, oh, there's indications here that it is doing things that we wanted, that we were strategizing about how to enable to do these things. Um, and it's just figuring out on its own that it should be doing these things. And it was also clear that we could scale it a lot further. So I think for me, that was, that was the big moment where we just like had it think for longer. And suddenly you see a qualitative difference, like you see this qualitative behavior

• 13:58Speaker 01

  um, that we thought we would have to somehow add to the model and it figures it out on its own. And of course, the performance was better, um, but the performance wasn't that much better. It was, it was really seeing the qualitative change, seeing, um, the, those behaviors um, that really gave me the conviction that like, okay, this is, this is going to be a big deal.

• 14:18Jacob Effron?

  Yeah.

• 14:18Speaker 01

  I think that was like probably October 2023.

• 14:22Jacob Effron?

  Wow, and it got out pretty fast after.

• 14:24Speaker 01

  Could have been faster.

• 14:26Jacob Effron?

  I guess, how, how would you kind of contextualize for our listeners today where planning in an O1 type model is helpful and where it's like, you know, you should stick with G P four O or, you know, it's not as helpful? And how do you expect that, I guess, to, you know, obviously you're constantly working on improving this, how does that change kind of going forward?

• 14:40Speaker 01

  I, I think eventually. There is a single model. I think, right now, we're in this, we're in this state where, you know, GPT-4O is better for, for many things. And O1 is better for, for many things. Certainly, O1 is more intelligent, so if you have like a very hard problem, O1 is extremely good for that. I have talked with researchers at universities, like, you know, a friend that's a professor who loves O1, like, is a real power user, because it just like can tackle these hard research questions that normally you would need somebody with a PhD to be able to handle.

• 15:21Speaker 01

  I think for some tasks, I think like creative writing might be one of them, though actually, I'm not sure if, I know that for, for something like creative writing, 4O is better than O1 preview, I'm not sure what the comparison is for O1. But certainly, like, the big benefit of 4O is that you just get a faster response. So if you just want a response immediately and it's not a very hard reasoning task, you know, I think 4O is a reasonable thing to try.

• 15:49Jacob Effron?

  Yeah.

• 15:50Speaker 01

  But I, I should say that eventually where we want to end up is like, there's a single model, and it can, you just ask it everything, and if it requires like a lot of deep thinking, then it can, it can do that. If it, if it doesn't, and it can respond immediately with a quite good response, it does that as well.

• 16:08Jacob Effron?

  What does the intersection of like multimodal models and these models look like going forward?

• 16:13Speaker 01

  So, O1 is, takes as input, images.

• 16:16Jacob Effron?

  Yeah.

• 16:17Speaker 01

  I think that's gonna be pretty exciting, it's gonna be exciting to see what people do with that. Yeah, I don't, I don't see any blockers to this, like, having them be as multimodal as like 4O and these other models.

• 16:27Jacob Effron?

  One of the fascinating parts of O1 is I feel like a lot of the previous work that you had done in reasoning was built, you know, on reasoning that was kind of specific to that problem, like I, you know, as I understand it, like, Go was, you know, Monte Carlo search that maybe wasn't as relevant for poker, and like, you know. Obviously, one of the things that is so impressive about what you built is, you know, you scaled kind of inference compute generally. Could you talk a little bit about like, what's required to do that versus maybe some of the more specific work that had been done in the past toward like specific types of problems?

• 16:52Speaker 01

  Well, I think it requires, I mean, I can't go into details about like the, you know, the actual technique, but I think the important thing is that it requires like, maybe a change in, in mindset, that I think when I was a PhD student and afterwards, once I saw how much of a difference scaling test and compute made in poker, I was like, okay, this is great, but unfortunately it only works in poker. So how do we extend this algorithm to be able to do more and more domains? And so, you know, there's a question about like, how do you get this technique to like work for both poker and Go? Or poker and diplomacy or something like that? And so, you know, we developed techniques that work in Hanabi. We developed techniques that worked in diplomacy.

• 17:34Speaker 01

  And one of the things that I was considering doing is like just trying to get this algorithm to play as many games as possible. Like try to come up with an algorithm that would work for as is a similar is similar to what was done in poker, but be able to like more broadly work. And I think the diplomacy work actually convinced me that that's kind of the wrong way to think about it, that you really need to start from the endpoint, which is like, okay, we have this extremely general domain. And language is actually a really good example of this, where you have such breadth. And instead of trying to extend a technique that worked in one domain to do more and more domains and eventually do everything, we should, we should instead start from everything and figure out,

• 18:17Speaker 01

  you know, figure out some way to scale test and compute. And my guess, and like initially, of course, it's not going to scale very well. It's not going to be like a very good technique to scale it test and compute. But then can you have it scale better and better? And I think that change in mindset, I mean, the diplomacy work is really what convinced me to have that change in mindset, because trying to take the techniques that we developed for poker and Go and apply them to diplomacy, we to apply to like the actual general, full general game of diplomacy. Like we managed to apply it to diplomacy with some constraints on like what it could actually do. And there was a ceiling to how, how much it could achieve. And we actually only got to human level, like strong human level performance in diplomacy. And it was pretty clear that if we pushed that paradigm a lot further, we weren't going to get to superhuman performance.

• 19:03Speaker 01

  So to actually tackle. The full game diplomacy and reach, super intelligent, like, superhuman performance in diplomacy, it was clear that we needed something that would actually just, like, work for pretty much anything.

• 19:14Jacob Effron?

  Yeah.

• 19:14Speaker 01

  And so I thought, like, okay, let's just, you just gotta jump to the endpoint and try to tackle everything.

• 19:19Jacob Effron?

  So interesting. I mean, you kind of mentioned that, you know, you kind of expect everything to converge on, you know, kind of a single model. I guess in what timeframe, like, in the medium term, do you think that, like, we have one model that rules them all, or, you know, obviously, there's lots of folks out there building specialized models for different use cases. Like, do you think building your own model, like, makes sense? I guess there's folks building legal models, or healthcare models, or some of these things.

• 19:42Speaker 01

  So, it's a good question. I get asked this a lot. I don't have a great answer for this, but, like, one thing I have been thinking about is, you know, you can ask, you can ask O1 to multiply two large numbers, and it can do it. Like, it'll work through the arithmetic to figure out how to, like, you know, carry the digits and all that, and then, like, actually multiply two large numbers and tell you the answer. It doesn't make any sense for it to do that. Like, the optimal, like, really what it should do is just call a calculator tool, or write a Python script that multiplies the two numbers, runs the script, and then tells you the output.

• 20:14Speaker 01

  So, I think that, that calculator tool is, like, one extreme end of the spectrum, of, like, very specialized, very simple, but very fast and cheap. And on the other end of the spectrum, you have something like O1 that is very general, very capable, but also pretty expensive. And I think it's quite possible that you'll see a lot of things that essentially act as tools in between those two extremes, and that O1, or a model like O1, can use to save itself and save the user a lot of cost.

• 20:45Jacob Effron?

  Yeah. It's really interesting. The tools don't end up being capability-enhancing. They're more just, like, to not require massive amount of compute costs to solve something that could be much more easily solved.

• 20:53Speaker 01

  Yeah. It's also entirely possible that some of these tools just do, like, a flat-out better job than O1. So I think the way I think about it is like kind of the same way that I would think about how a human would act like, you know, you could ask a human to do something, but like maybe they're better off just using a calculator or, you know, um, do you doing some other kind of like specialized, using some other kind of specialized machine or uh, something.

• 21:14Jacob Effron?

  Well, I guess on the O1 side, uh, any like you, you mentioned kind of your friend who's a professor or using it like, any other kind of unexpected use cases that you've seen in the wild or personal favorites?

• 21:23Speaker 01

  I think one thing I'm really excited for is to see how O1 is used for coding. Um, I think O1 preview, like people were pretty impressed, uh, its coding ability, but it was good in some ways for coding and, and not as great for others. And so, you know, it wasn't like strictly dominant in terms of among models for coding. Um, I think that O1 is going to do a lot better and I'm pretty excited to see, um, how, how that changes the field.

• 21:49Jacob Effron?

  Yeah.

• 21:49Speaker 01

  Um, if that changes the field and, um, yeah, I, I, I'm, yeah, I'm just really curious to see like, you know, we, we, I use O1 internally, um, other people do. We've had some people play around with it, give us feedback, but I don't think we really know how it gets used until we actually deploy it in the wild.

• 22:07Jacob Effron?

  Yeah.

• 22:07Speaker 01

  Yeah.

• 22:07Jacob Effron?

  How do you use it?

• 22:09Speaker 01

  I use it for a lot of coding tasks or like, you know, if I, yeah, have something and, and frequently what I'll do is like, if I have like something that's pretty easy, I'll like give it to 4O, but if I have something that I know is really hard or that I need to write a lot of code, I'll just give it to O1 and like have it just do the whole thing on its own. Um, and also frequently if like, if I have a tough problem that like, for whatever reason, 4O isn't getting, um, I'll just give it to O1 and it'll usually give me an answer.

• 22:31Jacob Effron?

  It's not doing core AI research yet.

• 22:33Speaker 01

  O1 is not doing core AI research.

• 22:35Jacob Effron?

  You know, you mentioned on the path to O1, obviously there were some things that you saw, you know, um, milestones that were really meaningful around, uh, you know, kind of the ability to reason through things. As you think about, obviously you're continuing to work on this class of models, like, what are the milestones that are meaningful to you going forward? Things that if you saw as, as you guys continue to scale up, that would be, you know, important to you.

• 22:52Speaker 01

  Like milestones is in like among, among benchmarks or something or?

• 22:55Jacob Effron?

  I mean, it could be specific benchmarks or like, you know, even just how you think about like the next set of capabilities that are important in, you know, that you, that you'd hope that like an O2 would have.

• 23:03Speaker 01

  Uh, I'm, I'm really excited to see these models, um, become more agentic. I think, I think a lot of people are, so.

• 23:12Speaker 01

  I think one of the major challenges, one of the major barriers to actually achieving agents. People have been talking about agents for a while, ever since ChatGPT came out. People were always talking about agents and like, you know, they would come to me and like, ask like, oh, why are you working on agents? And like, my feeling was that they were just, the models were too brittle. That if you have a, a long horizon task and there's a lot of intermediate steps, you need the reliability and you need the coherence to be able to like, have the model figure out that it needs to do these individual steps and then also like, execute on them. And, you know, yes, people tried to like, prompt the models to be able to do that and you could kind of do it, but it was always like, kind of fragile and not general enough.

• 23:54Speaker 01

  And the cool thing about O1 is that I think it's a real proof of concept that like, you can give it a really hard problem and it can figure out the intermediate steps on its own and it can figure out how to tackle those intermediate steps on its own. So the fact that it's able to do things that were completely, are completely outside the realm of what something like 4O can do without like, really excessive prompting. I think is, I think it's a good proof of concept that it, it can start doing things that are agentic. So I'm, yeah, I'm excited for that direction.

• 24:22Jacob Effron?

  There's obviously a lot of folks today that are working on agents and I think they take basically, you know, the current limitations of models and find ways around them, right? Whether they have, you know, they'll, they'll chain, you know, six model calls together to check outputs or they'll, you know, find some smaller fine tuned model that just checks, you know, whether something ties exactly back to, you know, the original data source. Or it feels like there's all these kind of like orchestration and scaffolding that's built to make this work. Does that feel like some of that stuff persists or is that eventually all just become part of like, the underlying model?

• 24:48Speaker 01

  You know, okay, so this is great essay called The Bitter Lesson.

• 24:50Jacob Effron?

  I knew we couldn't get through this podcast without The Bitter Lesson coming up.

• 24:54Speaker 01

  You know, because I'm surprised, like, whenever, whenever I like, give talks at various events, like AI events, you know, sometimes I'll poll people and I ask them how many have read The Bitter Lesson and surprisingly few have. I think people that have been in the field for a while.

• 25:06Jacob Effron?

  If anyone's listened to a podcast with you or follows you on Twitter, they would have been exposed to The Bitter Lesson many times.

• 25:10Speaker 01

  Perfect, great, okay. So for those that haven't, I mean, I think it's a great essay. I highly encourage people to read it. It was written by the creator of The Field of RL, Richard Sutton. And he talks about this and he says that every, you know, basically every time. There's, you know, you look at the history of chess, for example. The way people tried to tackle chess was to like, kind of code things up, like in a very, like code their knowledge into the models and try to get them to do human-like things. And the technique that ended up working really well was like techniques that scaled really well with more compute and more data. And I think the same is true now with these language models, that, okay, we've reached a certain level of capability and, you know, it's really tempting to like try to push it.

• 25:55Speaker 01

  Okay, there's things that they're just unable to do and you'd like them to be able to do those things. And so there's a big incentive to then add a bunch of scaffolding and add all these prompting tricks to push it a little bit further to be able to do those things. And you encode a lot of like your human knowledge in that in order to get the models to go a little bit further. What's ultimately going to work in the long run is a technique that scales well with more data and more compute. And there's a question about like, are those scaffolding techniques, do they scale well with more data and more compute? I think the answer is no. I think something like O1 is something that scales really well with more data and more compute. And so I think that in the long run, I think we're going to see a lot of those scaffolding techniques that push the frontier a little bit further.

• 26:40Speaker 01

  I think that they're going to go away.

• 26:42Jacob Effron?

  And I think it's an interesting question for builders today of like, you could solve a here and now problem with that and then evolve over time with what's required.

• 26:49Speaker 01

  Yeah, it's a tricky thing, especially for startups, because I know that they probably face a lot of demand for some task and, you know, there's something that's just out of reach of the models and they think like, okay, well, if I invest a lot into the scaffolding and customization to make it be able to do those things, then I'll like, you know, then I'll have a company that's able to do that thing that nobody else can do. But I think it's important, and this is actually one of the reasons why we're telling people like, look, these models are going to progress and they're going to progress quickly, is that you don't want to be in a position where The model capabilities improve and suddenly the models can just do that thing out of the box. And now you've wasted six months building scaffolding or some specialized, you know, agentic workflow that now the models can just do out of the box.

• 27:32Jacob Effron?

  Talking about kind of what's happening in the broader LM space, I mean, beyond, you know, test time compute, like what are other research areas you're paying attention to?

• 27:39Speaker 01

  I was really excited by Sora. I think a lot of people were. I thought it was really cool. I wasn't really keeping too up to date on like the state of video models. And so I was like, when I, when I saw it, I was pretty surprised at like how capable it was.

• 27:53Jacob Effron?

  You obviously, you know, cut your teeth in, in academia. You know, I think there's a question now that a lot of folks are thinking about, about the role of academia in AI research today, given obviously access to a completely different level of compute. Like, how do you think about like the, the role of academia today?

• 28:08Speaker 01

  Yeah, it's a real tough question. I've talked to a bunch of PhD students and they, you know, it's, they're in a tough situation where they, they want to help push the frontier further. And it's hard to do in a world where so much is dependent on data and compute. If you don't have those resources, then it's hard, it's hard to push the frontier forward. There's a temptation, I think, among some PhD students to try to do, you know, what I said shouldn't be done and like add their human domain knowledge, add these like little tricks to try to push the frontier a little bit further. So you take, you take a frontier model, you add some clever prompting or something, you push it a little bit further and you get like 0.1% higher than everybody else on some eval.

• 28:49Speaker 01

  And, you know, it's, the problem is that it's actually not, I don't, I don't blame the students so much as like, I think academia incentivizes this. I mean, it's, it's prestigious to have a paper accepted to a prestigious conference and it's much easier to get a paper accepted to a conference if you're able to show that you're like at least slightly better than everybody else on some eval. So the incentive structure is set up in a way that encourages that behavior, at least in the short term. But in the long term, that ends up not really being the most impactful research. So my suggestion is don't try to compete with the frontier industry research labs on frontier capabilities. I think that there's a lot of other research that can be done and I've seen really impactful

• 29:33Speaker 01

  research that, that can be done. So one example is just like investigating novel architectures or novel, novel approaches that

• 29:43Speaker 01

  Scale well. And if you can just show that like, okay, well, you can show the scaling trends and show that it's showing a promising path as you throw more data and more compute into it. Then that is good research, even if it's not getting to the performance in some eval. And the, you know, people are going to pay attention to that. I mean, it might not be that the people that casually pay attention to the field are going to like pick up on that. It might not like, you know, make it into the new cycle or something, but the people, it will have an impact if it's showing promising trends. And I guarantee you that industry research labs look at those kinds of papers. And if they see something that is showing a promising trend line,

• 30:26Speaker 01

  they're willing to like put in the resources to see if it actually like pays off at large scale.

• 30:31Jacob Effron?

  What evals are like still meaningful to you or like, you know, when you're playing around with a new model, like what are you looking at?

• 30:38Speaker 01

  I think there's like a lot of vibes questions that I ask and I'm sure everybody has.

• 30:42Jacob Effron?

  You have a go to vibes question?

• 30:43Speaker 01

  I mean, my go to is really tic tac toe.

• 30:46Jacob Effron?

  Always games, I guess that makes sense.

• 30:47Speaker 01

  Yeah, it's like, you know, it's shocking to see how challenging it is for some of these models to play tic tac toe. I joke that it's, I think it's just because there's not enough five year olds on the internet.

• 31:00Jacob Effron?

  We haven't populated the world with tons of tic tac toe data.

• 31:03Speaker 01

  Yeah. And I just like see how these models do with like the kinds of data, data, data questions that I have. And it's, it's pretty cool to see the progress in things like going from 400 to 01 preview to 01.

• 31:17Jacob Effron?

  Yeah. I mean, you mentioned, obviously, you know, it sounds like since 21, you changed your mind and then showed it with what was possible at test time compute. Anything like in the last year that you've changed your mind on in the AI research world?

• 31:28Speaker 01

  And I shouldn't say it wasn't like I changed my mind in 2021. I was like pretty bought into this. Even basically when we got the poker results in like early 2017.

• 31:38Speaker 01

  I think for language models, I think the shift of language models, like I think I started to think about that like more like 2020, 2021.

• 31:45Jacob Effron?

  Yeah. No, sorry. I meant more like the, you had in 2021, thought it would take 10 years to. And now I think it's two. Anything in the last year that you like, you've, you know, kind of done a 180 on of something you thought?

• 31:57Speaker 01

  You know, I think the main thing that I've changed my perspective on is like how quickly I think things would progress. So like I said, I I remember, I've been in the AI field for a pretty pretty long time is, you know, by today's standards. So I started grad school in 2012. I saw the deep learning revolution happen and I saw people like talking very seriously about AGI and superintelligence back in like 2015, 2016, 2017. And, you know, my my view at the time was that, you know, just because AlphaGo is superhuman to go, it doesn't mean that we're going to get to superintelligence anytime soon. And I think that was actually the correct assessment. Like, I think

• 32:40Speaker 01

  people didn't look at the limitations of AlphaGo enough and the fact that like, okay, it can play Go, it can even play chess and Chogi, but it can't play poker. And nobody actually has a good idea about how to actually get it to be more general than that. And two players here, some games are these like very ideal situations where you can do this like unlimited self-play and keep hill climbing in some direction that gets you to superhuman performance. And that's not true for for the real world. So I was I was pre I was on the more skeptical end. And it's probably like, you know, actually more optimistic than the average AI researcher that we could like progress towards very, very intelligent models that would change the world. But I think compared to, you know, people like at OpenAI or some of these other places, I was on the more skeptical

• 33:23Speaker 01

  end. And I think my perspective on that has changed quite a bit. I think seeing the seeing

• 33:34Speaker 01

  the ability to scale test on computer very general way like that changed my mind and like, and I kind of became like increasingly optimistic. Actually, I think like the conversation that I had with Ilya back in 2021 was the start of that, that he like, kind of convinced me that like, yes, we don't have the entire paradigm figured out, but maybe it's not as far away as like 10 years. Maybe we can get there sooner. Yeah. And I think Seeing that actually happen changed my perspective and I think things are going to happen faster than I originally thought.

• 34:04Jacob Effron?

  I mean, obviously, there's a bunch of folks out there that are trying to compete with Nvidia. You know, I think Amazon recently has been pretty aggressively investing in Tranium, having Anthropic use it. Like, what do you think about some of these other hardware efforts?

• 34:14Speaker 01

  Uh, I'm pretty excited for, um, you know, seeing the investment in hardware. I mean, I think one of the cool things about O1 is that I think it really changes the way that people should be thinking about hardware. So I think before people had this mindset that like, okay, there's going to be these like massive pre-training runs, but then actually like the inference cost is going to be like pretty cheap and, you know, very scalable. Um, I don't think that's going to be the case. I think that we're going to see a major shift towards inference compute. And if there are ways to optimize around inference compute, I think that's going to be a big win. So I think there's like an opportunity for a lot of creativity on the hardware side now to adapt to this new paradigm.

• 34:47Jacob Effron?

  Kind of hitting on some questions outside of LLMs, you know, I feel like your work with diplomacy is like incredibly interesting. Obviously, it's like this game that involves negotiation, predicting how others will act, et cetera. Um, it's hard not to think about the implications of that for like, you know, simulating society to test policies or like even having AI like as part of a government in some way. How have you kind of thought about this and what are your kind of intuitions as these models get better and better about like the role they play and those parts of society at large?

• 35:14Speaker 01

  Well, I think, um, there's, I guess, two questions there, but like kind of answering like one of them. I think one of the directions that I'm pretty excited about for these models is like using them, uh, for a lot of socia-, uh, like social science experiments and, um, and also things like neuroscience that I think you can learn a lot about humans by, by looking at these models that were trained on vast amounts of human data and are able to like imitate humans quite well. So, um, and of course, the great thing about them is that they're much more scalable and cheaper than like hiring a bunch of humans to run these experiments. So I, I am curious to see how, um, the social sciences use these models to do cool research in their fields.

• 35:54Jacob Effron?

  Yeah. What are some ways you could imagine that happening?

• 35:56Speaker 01

  Um, I think.

• 35:59Speaker 01

  You know, normally you would do a bunch of, like, if you want to do, um, I mean, I'm not a social scientist, so it's, you know, I haven't, I haven't thought about this, like, that well. Um, but I think like economics, for example, um, there's a lot of, um,

• 36:13Jacob Effron?

  You did work at the Fed before, right?

• 36:14Speaker 01

  I did work at the Fed, yeah, I guess it's social sciences, that's true. So I guess game theory is actually a good one, where, you know, I've been in these, uh, I, I, when I was in undergrad, I did some of these experiments where, like, you know, they would hire, um, they would bring in a few undergrads, pay them a small amount of money, and like, have them do these, like, small game theory experiments to see, like, oh, how rational are they? Um, how do they respond to incentives? Like, how do they, how much do they care about, like, um, how do they, how much do they care about, like, making money versus, like, getting revenge on people that, like, you know, wrong them? And a lot of these things you can do now with, uh, with AI models, and I, it's not obvious that it would translate, it would be like a match for human performance, but that's something that can be quantified, like, you could actually see, like, okay, in general, do these models

• 36:56Speaker 01

  do things that humans would do, and then if you have a much more expensive experiment, then you could, you know, maybe extrapolate and say, like, okay, well, we are, this is, like, not cost-effective to do with human subjects, but we could use this AI model. Or things that are, like, ethical concerns as well, you know, maybe you can't do this experiment because it's not ethical to do with humans, but you could do it with AI models. So I guess one example is, um, the ultimatum game, are you familiar with that?

• 37:17Jacob Effron?

  No.

• 37:18Speaker 01

  Okay, so the ultimatum game is that you have two participants, let's call them A and B. A has, like, a thousand dollars, and they have to give some percentage of that to B. And then B can decide, um, whether to accept that split, or to just say that neither player gets anything. So, you know, if A has a thousand dollars, they give, like, you know, two hundred dollars to B. If B accepts, then B gets two hundred, A gets eight hundred, if B rejects, then both of them get zero. And, you know, there's, um, experiments showing that, like, if people, um,

• 37:49Speaker 01

  Get offered less than roughly thirty percent, then they'll reject. And of course, there's the question about like, okay, well, if it's a small amount of money, then that's pretty understandable that, you know, if it's ten dollars and you're only offered three dollars, then you would just be like annoyed at that person and reject despite them. Are you still going to do that if it's ten thousand dollars and you're being offered three thousand dollars? It's a kind of a different question. And so the only way that you, of course, it's like super expensive to actually run that experiment. And so the way they've done it historically is they would go to a very poor community in like a different country and, you know, offer them what to them would be a very large amount of money and see how they would act differently. But even then, you can only like push that so far.

• 38:33Speaker 01

  So with AI models, like now, maybe you could actually like get some insights in how people would react to these kinds of situations that are like cost prohibitive.

• 38:40Jacob Effron?

  It's interesting, I mean, and also for, you know, neuroscience and other things, I always think, you know, I think a complaint of other social sciences has been all those experiments are done on like, you know, college kids that need to get credit in their, in their intro psych class or something. And so also, you know, getting exposure to a broader, you know, the internet at least is probably a broader swath of society that's been trained on than most of these experiments, which are basically like nineteen year olds at, at top institutions.

• 39:00Speaker 01

  Yeah, that's a great, yeah, that's a great point. And I should also say that like, look, if you're doing these experiments like GPT three point five, like GPT three point five is not going to do a great job of like imitating how an actual human would do in a lot of these settings. But this is a very quantifiable thing that you can actually just measure how closely these models are matching what humans would do. And I, I suspect, I haven't actually like looked at these experiments myself, but I suspect that as the models become more capable, they do a better job of how, of imitating how actual humans would do in these settings.

• 39:27Jacob Effron?

  Yeah, you know, and then obviously, you know, your work in diplomacy was focused on kind of an AI player among a bunch of humans. How if at all does that change? Like, I feel like we're about to enter some world where we have AI agents interacting with other AI agents and negotiating and whatnot, like, how does that change, uh, change things, if at all?

• 39:43Speaker 01

  I think one of the things that I'm really excited for about

• 39:47Speaker 01

  L L Ms is that, you know, there was always this question in A I about how do you get A Is to even communicate with each other? Um, so there's this whole field of A I called emergent communication, where people would try to like, teach A Is to like be able to communicate with each other.

• 40:01Jacob Effron?

  Yeah.

• 40:01Speaker 01

  And that problem is now effectively solved because you you have um, a language built in that conveniently also humans use. And so a lot of these problems are just like kind of conveniently like out of the box, just like answered. So I I it's quite possible that maybe you don't need to change that much.

• 40:18Jacob Effron?

  What do you think what's uh, happening in the A I robotics space? Like, where do you think that space goes in the next few years?

• 40:22Speaker 01

  I think in the long term, it makes a lot of sense. I, you know, I did a masters in robotics. Um, I didn't actually like work with robots that much, but like I was in the program and like I had a lot of friends and that that were working in robotics. And one of the main takeaways that I got is that hardware is hard. And it takes longer to iterate with hardware compared to software. So I suspect that robotics is going to take um, take a little while to progress, just because iterating on actual physical robots is is hard and expensive.

• 40:57Speaker 01

  But I think that there's going to be progress.

• 40:59Jacob Effron?

  Obviously, you're about to release one into the wild and people are going to build all sorts of things on top of it uh, that that neither of us could possibly imagine. Um, but are there some areas generally that you feel like are underexplored applications today or places where you wish there were more builders messing around with these models?

• 41:13Speaker 01

  Um, I think I'm really excited to see these models advancing scientific research. I think we've been kind of a weird state up until now where the models were like broadly very capable. But they weren't necessarily surpassing uh, expert humans in hardly any domains. And I think increasingly as time goes on, that's it's going to stop being true. We're going to start seeing the models like surpassing what expert humans can do. In first, just like a few narrow domains, but then increasingly more and more domains. And That opens up the possibility that you can actually like advance

• 41:54Speaker 01

  the frontier of human knowledge and use these models like not as replacements for for researchers, but as a partner that you can use to do things that were not otherwise possible or do them a lot faster. So I think that's the the application that I'm most excited about, and it's not something that like has really been possible yet, but I think that we're pretty, we're going to start seeing happening.

• 42:17Jacob Effron?

  You think it's possible this current set of models?

• 42:19Speaker 01

  I don't know. And this is actually one of the reasons why I'm excited to see O1 release, because, you know, I'm not a researcher in, I mean, a researcher in one domain, but I'm not a researcher in like all these different domains. And I'm, I don't know if it will be able to improve chemistry research, the state of chemistry research, or the state of biology research, or theoretical mathematics. And getting the model into the hands of these people and seeing what they can do with it, I think will

• 42:44Speaker 01

  give us some feedback on like what, what, where, where it's at on those domains.

• 42:49Jacob Effron?

  You mentioned that it might start, you know, more narrowly first before expanding out. Any intuitions on like the narrow subset of things that might be particularly well suited to it? Or is that for the community to find out as they, as they mess around with it?

• 42:59Speaker 01

  I think it's, I think it's for the community to find out. It for O1 preview, it looks like it does particularly well on math encoding.

• 43:06Jacob Effron?

  Yeah, those were very impressive results.

• 43:08Speaker 01

  Yeah. And I mean, it's like improving things pretty broadly, but we're seeing like quite noticeable progress on those two. I, I wouldn't be surprised if that like continues to be true and like we see the performance improving very broadly, but like because math encoding is already ahead, that it will continue to progress more quickly on those two.

• 43:27Speaker 01

  But I think it's, it's going to be a broad improvement across the board.

• 43:31Jacob Effron?

  Well, no, this has been a fascinating conversation. We always like to end with a quick fire round where we get your quick take on things. Maybe to start, what's one thing that's overhyped and one thing that's underhyped in the AI world today?

• 43:40Speaker 01

  I mean, I guess like overhyped, I would say a lot of these

• 43:45Speaker 01

  kind of like prompting techniques and like scaffolding techniques that, you know, like I said, I think are going to be done away with in long term. Underhyped? I I think, I mean, I'm a huge fan of O1. I gotta say O1. I think that for people that are paying attention to the field, it has been a big update. I think that for the broader world, I don't know if people have recognized what it means yet, to the extent that they should. Yeah, I think I'll go

• 44:14Jacob Effron?

  with those two. Hopefully, the release tomorrow starts getting to that.

• 44:16Speaker 01

  Yeah, well, yeah, we'll see.

• 44:18Jacob Effron?

  Do you think model progress in 2025 will be more or less are the same as 24?

• 44:23Speaker 01

  I think that we will see progress accelerate.

• 44:26Jacob Effron?

  How do you define AGI?

• 44:28Speaker 01

  I don't.

• 44:32Speaker 01

  I've been trying to shift away from using that term as much as possible. I think an AI that can do, I mean, I think that there's going to be a lot of things that an AI will not be able to do that humans can do for a long time. And I think that that's the ideal scenario, especially things like physical tasks. I think that humans will have an edge for a very long time. And so I think an AI that can do, that can like accelerate human productivity and make our lives easier, I think is like the more important term than AGI.

• 45:05Jacob Effron?

  Well, no, I always like to leave the last word to our guests. And I feel like there's a million places you could point people to your work, what's going on at OpenAI, but the floor is yours. Anything you want to say to our listeners or things you want to call out?

• 45:17Speaker 01

  Yeah, I mean, I guess the main thing is that, like, I, you know, to skeptics out there, like, I get it. Like, I, I've been in this field for a long time. I was very skeptical about like the state of things and where and the hype around the progress in AI. I, you know, I kind of recognized that AI was going to progress, but I thought that it would take us much longer to even reach this point. Like, I think it's really important to recognize that, like, where we are right now is complete science fiction compared to like even five years ago, let alone 10 years ago. So the progress has been astounding. And I think that there are

• 45:58Speaker 01

  Reasonable concerns about like, oh, are we going to hit a wall? Is progress going to stop? But I think it's important to recognize that the test on compute paradigm, in my opinion, really addresses a lot of those concerns. And so I think for people that are still skeptical of like the progress in AI, I would just recommend like, take a look for yourself. We've been pretty transparent with the blog post and our results about like where things are, where we see things going. And I think the evidence is pretty clear.

• 46:29Jacob Effron?

  Well, no, this has been absolutely fascinating. A real pleasure of this job to get to sit down with you. Thanks so much for taking the time.

• 46:35Speaker 01

  Of course. Thanks.

• 46:36Jacob Effron?

  A huge thanks again to Noam for just fascinating conversation. If you enjoyed that, please consider subscribing, sharing with a friend. We're always trying to get the word out about the podcast. We have a bunch of great conversations coming up with leading AI researchers and founders. 2025 is going to be an incredible guest lineup. Thanks so much for listening and I'll see you next week.