Ep 81: Ex-OpenAI Researcher On Why He Left, His Honest AGI Timeline, & The Limits of Scaling RL

This episode features Jerry Tworek, a key architect behind OpenAI's breakthrough reasoning models (o1, o3) and Codex, discussing the current state and future of AI. Jerry explores the real limits and promise of scaling pre-training and reinforcement learning, arguing that while these paradigms deliver predictable improvements, they're fundamentally constrained by data availability and struggle with generalization beyond their training objectives. He reveals his updated belief that continual learning—the ability for models to update themselves based on failure and work through problems autonomously—is necessary for AGI, as current models hit walls and become "hopeless" when stuck. Jerry discusses the convergence of major labs toward similar approaches driven by economic forces, the tension between exploration and exploitation in research, and why he left OpenAI to pursue new research directions. He offers candid insights on the competitive dynamics between labs, the focus required to win in specific domains like coding, what makes great AI researchers, and his surprisingly near-term predictions for robotics (2-3 years) while warning about the societal implications of widespread work automation that we're not adequately preparing for.

• 0:00Speaker 00

  As VP of Research at OpenAI, Jerry Tork was a part of many of the biggest advances in AI these past years. Reasoning models like 0.103, Codex, he recently left OpenAI, citing that he wanted to pursue research areas that were harder to pursue within a big lab. I'm Jacob Effron, and today on Unsupervised Learning, I got to sit down with Jerry to talk about everything that's happening in AI right now and where the space is headed. We talked about reinforcement learning, what's required to further scale it up, and the origin of the ideas for 0.103. We talked about continual learning and other research approaches, and how Jerry is thinking about their promise and the problems to go solve there. We talked about his reflections on seven years at OpenAI, and some of the key decisions that had to be made, as well as how he sees the race between the foundation models playing out from here. And we talked about what makes good researchers, and ultimately, why researchers choose to work at different labs. It's just a fascinating conversation to get to ask everything that's top of mind in the space today,

• 0:46Speaker 00

  with someone who's been so close to building these models and is really zooming out to think about what's next. I think folks will really enjoy this conversation. Without further ado, here's Jerry.

• 0:55Speaker 00

  Well, Jerry, thanks so much for coming on. I've been super excited for this episode for a while. I feel like you've been behind some of the biggest breakthroughs in AI over the past years, you know, 0.1, 0.3, Codex, and now have some really exciting next steps. I know you've left OpenAI to pursue something new. So I really can't think of a better person to talk about the state of where we are with AI today and kind of all the potential future directions.

• 1:16Speaker 01

  Thank you very much. Very happy to be here. And AI is one of my favorite topics to talk about, so let's do that.

• 1:25Speaker 00

  Amazing. Well, you know, I think I'll start with, you obviously played a key part at OpenAI in introducing reasoning models and then scaling reinforcement learning. And so maybe we'll start with the existing scaling paradigms, and I'd love to get your temperature. How far do you think we have scaling the current vectors we have of pre-training in RL? Like, where does that get us from model capabilities?

• 1:45Speaker 01

  It definitely gets us somewhere. The question is, how do we call that place? How do we name it?

• 1:51Speaker 00

  I fear you could tell me.

• 1:53Speaker 01

  But it is a very, very real and for for for most practitioners, pretty striking thing that the benefits to scaling are real, predictable and nice. And whenever we scale up pre-training, we get better pre-trained models, which fundamentally know more about the world, which fundamentally understand language better and fundamentally build the linguistic world model of everything around them. And in the same way, scaling up reinforcement learning makes the model better into acquiring skills that we want to do. And in many ways, in both those cases, you get what you train for.

• 2:36Speaker 01

  If you if you want to do next token prediction, you can pre-train models very heavily and get really, really great models in next token prediction. If you want a specific set of skills, you train reinforcement learning models and then you get them really, really great at whatever you are training for. Like there are basically no limits. Everyone, everyone knows these days that if you care about a skill, you just just do reinforcement learning on that skill and you get them all, that that is excellent. It's it's it's kind of that simple and works. What people hesitate sometimes where are the moment of holdups are how do those models generalize? How do those models perform outside of what they've been trained for? How do how do models do? Do models do knowledge that is not in a pre-training corpus?

• 3:21Speaker 01

  Probably not. How do models do tasks that you don't? Reinforcement learning on? And probably not that great. And this is this, these are basically the remaining questions in the in the world of AI, because what we train for, we can, we are getting really, really good at.

• 3:36Speaker 00

  And it feels like there's kind of two schools of thought here. One is, you know, look, we're still early in this reinforcement learning paradigm. You know, as we scale this up, we'll start to see, you know, more signs of generalization. And so maybe these two scaling vectors alone take us to to most of what we want from AI models. And then the second, you know, being, you know, really, we need something new and different to to kind of continue. Uh, where do you kind of fall on that?

• 3:57Speaker 01

  And something I almost think it's a question of the economics. It is pretty clear. That if you again, if you like scaling means in many ways adding data. Uh, that scaling doesn't work that much without data. So if you add more data that you want, then you get better models doing, doing what you want. It's what you see right now, every quarter, every lab releasing a better model. It mostly means they've both together with scaling or compute. More importantly, they added more data to the model. And most importantly, it was data targeted to what the previous model was, was, was bad. It's a, it's a really powerful, really powerful methodology of training better and better models.

• 4:41Speaker 01

  And in that case, obviously, the rating for it will give you a model that to do everything that you want. If you keep adding the data to do the skills that you want, that you want the model to do. But this, this loop is slow in some aspects. And then the, the, the fundamental question can be, can be faster. Because I, I, I, I believe it is very natural that, that with, with the paradigms of the methods of training models today. If we keep adding the data that we want the models to be good at, they will be good at it. And there, there, there is some amount of generalization within it. But the, the main question is there, is there, is there more? Is there, is there some kind of research that would give us more results with less data?

• 5:24Speaker 01

  And in, in, in some ways are more fundamentally better, uh, better ways of, of, of most generalizing. From what they've seen so far and what they, what they've learned so far.

• 5:34Speaker 00

  And I know we'll hit on some of those, you know, future potential directions later. Maybe in the reinforcement learning world, just to kind of set the context for our listeners. How do you kind of characterize today, like what works and what doesn't? You know, what's your, what's your own mental model? Is it literally anything we have data for? I know a lot of people talk about the difference between spaces that have kind of are easy to verify versus those that are a bit more difficult. What's your own mental model for what we can do reinforcement learning on today effectively?

• 5:59Speaker 01

  The question about easy and hard to verify often comes very close to what is easy to even get a signal on the quality of, if you'd like. And at some moment, we've made a pretty good progress at OpenAI, training all that to become great writers. It is possible. Reinforcement learning can really be done on a lot of great things. Sometimes, meaningfully, it is a very hard thing to know what is good or what is not good, or you need to wait a lot of time. If you write a book, there are some easy ways to tell if it's a good book or not.

• 6:41Speaker 01

  But most likely, getting a signal on that, you have to try to sell this book and see how many people would like to read it, how many people would want to buy it. And sometimes even that is not a good signal, because a lot of critics says, "Oh, this book is so good, just no one did want to buy it because marketing didn't work well." So how can we do reinforcement learning on writing good book? It's hard to say. How do people learn how to write good book? It is a very, very hard thing to say. Similarly, with starting companies, there are lots of companies started early on. And how do we know which ones are good or not? Only five to 10 years down the line, we'll see that some entrepreneurs have succeeded and started really successful companies, and some of them failed.

• 7:24Speaker 01

  Was it because of some actions started early on were good or bad? Or maybe it was a stroke of luck? Doing reinforcement learning on that very directly is a very, very hard thing to do. Everything where you get feedback of any kind, you can use that to do reinforcement learning on.

• 7:43Speaker 00

  People have been blown away by the results that the models you worked on have gotten in coding competitions and math competitions and other things. And I think a lot of people are still trying to figure out their intuition for, do most tasks look like coding and math? Or do most tasks look like books or starting companies or things that are actually very hard to build rewards and also just test tons of times? You know, maybe we'll take things like accounting or medicine or legal. Do you have a gut instinct on if those are more like the former or the latter?

• 8:11Speaker 01

  I think fundamental question is how easy to tell if you did a good job or not. Because arguably even for humans, it's a very hard thing to tell if you did write a good book or not. With a lot of other work, if you can be, let's say, a manager of accountants and be able to tell which accountant is doing a good job or not if there are rules, then with those rules, you can train whatever you want, really. For medicine, I've been thinking a lot about surgeons. And there clearly is a number of rules and there are clearly

• 8:56Speaker 01

  sense of feedback of humans have survived the operation. And that is a success criteria that is very good. And that is a very interesting way where really experts, really skilled humans go against the rules to do something differently that has ever been done before. And they succeed through that. If a surgeon really for their experience sees that they need to do this particular operation differently than all the others before, they go against the established practice and through that they do something completely new that suddenly succeeds and makes an easy operation successful.

• 9:39Speaker 01

  I think the models would be able to do that too with some time and with enough ability to try. The question is, how much time would it get the today's models to really get to something like that?

• 9:55Speaker 00

  As you think about the problems that need to be solved to kind of make RL more and more generalizable for tasks that people care about, like maybe help our listeners understand what are the next frontiers for these other domains of RL?

• 10:06Speaker 01

  Generalization, I think fundamentally is a property of a model. So the story is, whenever you train, you really affect your training objective. And that's kind of it. With training objective, you get what you train for. And the question is, how many other things you are getting for free? There are clearly methods of learning, various bits, even next token prediction, that generalize poorly. There is like nearest neighbor classification, very classical ML algorithm. Theoretically, you can use it for any ML problem there is.

• 10:48Speaker 01

  It just generalizes poorly because it has a very, very simple representation it builds of the world. Neural networks, how they work magically is that through large-scale training, through large-scale pre-training, they manage to learn very interesting, very useful representations of the world around it. And it sometimes may seem like we get it for free. How has it happened that through training large transformer on the internet, it learns really, really to understand well a lot of the concepts around the world? Where does it come from? It's a magical power that comes from a transformer

• 11:30Speaker 01

  and a lot of parameters that we hammer over with gradient descent repeatedly. And this is the type of generalization we get from this model. There's a question, is there a different model that will generalize better? And almost surely, there must be. The question is, how does it look like?

• 11:48Speaker 00

  I've heard you talk before about, you know, say that you kind of updated your timelines maybe to be a bit longer for some of the, you know, different aspects of AGI after working on scaling RL. Why was that?

• 11:59Speaker 01

  I was definitely a very optimistic person in a sense of thinking we do reinforcement learning on our model and we'll get to AGI. And maybe we did, maybe it already is AGI. The definition of AGI is very personal. Exactly, it's a very personal thing and in some ways it is everything we still don't have. So the models that can solve basically any Ollie problem, any competitive problem, the models that are meaningfully right now proving new, solving new mathematical problems that no one has solved before. We get with the latest GPT 5.2 examples of that every week of someone achieving that. Is it an AGI? Like, many people would say yes.

• 12:44Speaker 01

  At the same time, I'm big fan of using coding models and they still make mistakes. In some places, they do things that could take me really, really long time to do well. So they can be extreme force multiplier on doing programming work, but at the same time, there are clearly places where they fail. And I would say the biggest limitation of the models today is that if they fail, you get kind of hopeless pretty quickly. Sometimes you can do a bit of back and forth between pasting an error message, "Hey dear model, this didn't work, like try harder." Sometimes we need to do words of encouragement, but fundamentally, there isn't a very good

• 13:28Speaker 01

  mechanism for a model to update. It's beliefs and its internal knowledge based on failure, which, like, this is probably the biggest update on me, unless we get models that can work themselves through difficulties and get unstuck on solving a problem. I don't think I would call it AGI, because of this feeling of hopelessness that if you hit a wall with existing model and they can't solve it, they just can't solve it. Either try a different model or do it yourself. The intelligence always finds a way. Intelligence works on the problem and probes it until it solves it, which the current models

• 14:13Speaker 01

  do not really.

• 14:15Speaker 00

  Well, I mean, it's kind of a great transition to some of the other research areas, maybe beyond kind of the pure scaling of pre-training in RL. And a lot of what you're talking about sounds, you know, in a similar vein to continual learning, a lot of topics that people have been discussing more and more, you know, in public these days. I'm wondering, like, how do you, maybe at the highest level for our listeners, think about the set of problems that need to be solved, like, to make something like continual learning, you know, actually possible?

• 14:40Speaker 01

  Yeah, and the very core thing, being able to continuously train a model means being able to have the model not collapse and not go into the weird mode or error. Like, there are many ways in which training a deep learning model fails horribly, and a lot of the work that's happening in the big labs these days is about keeping those models so-called on the rails and keeping the training healthy, and fundamentally a fragile process. It is a process that you have to make effort to go well, and if you don't make that effort, it explodes.

• 15:24Speaker 01

  You just don't get a good model in the end. And that seems fundamentally different to how humans learn. I think human learning is much more anti-fragile in a way. It can get itself again. It is fundamentally robust, can get itself unstuck. Throughout learning models with reinforcement learning, I've often really marveled at how infrequent it is for humans to crash out and then start talking gibberish and then the brain to, and then the human brain after getting some new information to spiral into some weird state, while the AI models do that. They do that pretty frequently.

• 16:09Speaker 01

  And it's something that researchers try to find both theoretical and practical solutions of how to fight it. And I think this fundamental robustness of a training process. Is something that is necessary for continual learning.

• 16:28Speaker 00

  How much of the ideas for, you know, the maybe some of the interesting ideas for continual learning? How much of it does it feel like has been around maybe for a bit or has been discussed versus, you know, it's entirely net new research problem?

• 16:40Speaker 01

  The main question is worth asking yourself, specifically as a researcher. That's something that I am asking a lot. Why hasn't it been solved yet? And that is, that is, that must be the number one. Like when someone starts working on problems like continual learning, which I think pretty clearly hasn't been solved yet. The main, the main question is, why, why not? What, what was the particular path that no one has taken so far and why? There are many researchers in the world that are very smart, are having a lot of brilliant ideas. And so far, no one has really cracked continual learning. And there are, there are many, many hypotheses for it.

• 17:21Speaker 01

  But one fundamental one, I think, is that most likely it is a research that needs to happen at scale, at least at a certain scale. And there are only so many well-funded research labs in the world right now that can only do so much research and so many few research projects. That, that, that is most likely one of the, one of the really big reason. If there is a research that you could do at a small scale and fundamentally discover something, you know, it's probably there were a few, but either it would be something very complex or very theoretically difficult to do, or, or, or just requires already models and levels of compute that are available to very few.

• 18:06Speaker 01

  And it's very likely that the very few labs didn't go in a, in a particular direction yet because, because they were, they were busy doing other things.

• 18:14Speaker 00

  I mean, I've heard you talk about before this idea that like there's ideas in AI whose time isn't right, but they're still good ideas. And certainly we saw this with reinforcement learning and, you know, it, it became much more effective after having large pre-trained models to be on top of. So it sounds like, you know, your, your maybe intuition is that there are some really good ideas out there that maybe if actually applied at scale would, uh, would be really helpful toward this domain of problems.

• 18:35Speaker 01

  I definitely think so.

• 18:36Speaker 00

  I've heard you talk before about the labs really converging on working on pretty similar stuff, right? And I don't know how, if that feels like that's been common over your past two, three years. But it seems like when you were leading a lot of the work in O1, that was a genuinely new thing that a lot of the labs were caught maybe flat footed on. Talk a little bit about this convergence that's happened maybe over the last year. And was that surprising to you?

• 19:01Speaker 01

  Yeah, even when training malls with reinforcement learning, there is this well-understood and well-documented trade-off of exploration versus exploitation. And you wonder, when is the right time to try different things than you've been doing so far? And when is the right time to try to optimize very, very well what you already know well how to do? And it's a trade-off that has no real solution because you don't know what is the unknown. Like whether exploration is successful or not fundamentally is, is there any way that is different from the current one that would give me a lot? And unless you know the landscape of what you are doing, it's fundamentally a very hard thing to do. So I don't think there's a fundamental question about it.

• 19:43Speaker 01

  But I remember someone telling me, it's someone in my life, "Why do all the planes, all the commercial planes look the same?" Even though there are a few companies building those. Because in the end, this is the most economically efficient design. In a way, what all the labs are doing today, that the forces of economics are fundamentally very strong in that. And if you want to compete, you need to have the best models of the lowest price. And the competition is pretty efficient there in terms of customers can switch whenever they want. And it's really the customers that are winning for that in many, many ways.

• 20:25Speaker 01

  But that is one thing that drives the labs to go for higher and higher efficiency and to go for produce better and better models in a pretty predictable way. And then there is a question of exploration versus exploitation. Should we try to go, should we try to sail over the sea and see what is out there? Should we try to train a model that is completely different? It would probably be... It would probably walk away and lose some focus on trying to get the current thing better. You lose some focus in trying to get the current thing more efficient. But maybe there's something 10 times better. Maybe there is something 100 times better there.

• 21:04Speaker 01

  There's a question of a belief and conviction at the heart of it of how much do we want to try those other things versus not.

• 21:12Speaker 00

  And to your point, I mean, obviously, there's such a clear path forward on adding more and more data to RL in different domains and that improving models for economically valuable tasks. There's kind of a clean roadmap to how each of the labs can continue to improve their underlying models. That maybe makes it harder to go out and make that big bet. Whereas when it felt like maybe pre-training was slowing down, it's easier to go out and explore a bunch of different things.

• 21:37Speaker 01

  Yeah, I think there are also just different times in history. Sometimes there is more appetite for it and a little bit of more of a freedom to explore various dimensions. The more competitive the landscape becomes, it becomes harder because it is almost something like a prisoner's dilemma.

• 22:02Speaker 00

  Yeah.

• 22:03Speaker 01

  Situation where we're trying to do things differently. Like exposes you very heavily to losing market share to other players.

• 22:16Speaker 00

  Yeah. Do you think it actually even matters for the labs if the next big breakthrough is discovered there? I mean, one thing I've been struck by is just the dissemination of a lot of these advances. Obviously, oh, one, you were kind of the pioneer on the reasoning side. There's a few labs now that have great reasoning models. And I almost wonder if the labs would be just fine if the breakthrough happened somewhere else. Because these ideas diffuse and eventually they'll be able to plug it into the existing business.

• 22:45Speaker 01

  Ideas diffuse, and that's a good thing. But at the same time, the lead it gives you to be the first, I don't think it's something to really discount.

• 23:00Speaker 01

  In many ways, we have seen the fact that, if you believe that fundamentally there would be no place in the world for OpenAI to succeed, and it succeeded because it went into pre-training transformer, large-scale, much better than anyone else. And that lead made it one of the largest and most successful company in the history of the world. And in the same way, because of OpenAI being the first one to figure out how to do large-scale reinforcement learning. I think for a long time, and still until today, I believe that it has the best reinforcement learning research program out of all labs, which allows it to do things better and more ambitious than most other labs that had to catch up to this much later.

• 23:45Speaker 01

  And while the ideas diffuse, the lead can be a very, very powerful thing that if maintained, it will stay with you for, maybe potentially, forever. In many ways, I've been reading a book about semiconductor manufacturing, which many of the core initial parts of the invention were done in the United States. And through that, they slightly disseminated through the world through two various places. But at the same time, there has been moments and space of lead that a lot of other countries couldn't ever match.

• 24:27Speaker 01

  The continued compounding of advantages over time for some of the countries that bet on it early and really started hard to build it. And it's not that there is only one country building, having a successful semiconductor business, but also not every country. It doesn't exist everywhere.

• 24:47Speaker 01

  There always is a place whenever there is a business shift. There are some newcomers that will be successful. Some newcomers that will be unsuccessful and some old companies that will that will stay and manage to turn themselves around and some old companies that will die. That's the that's the direction part of of progress.

• 25:05Speaker 00

  No, and I feel like consumers and enterprises always remember the first company that that introduces them to some, you know, pretty magical experience. I mean, certainly you guys experienced those with ChatGPT. One thing that's fascinating about you is you obviously made all this incredible progress in in RL and, you know, were help pioneering a bunch of this. And RL progress is still alive and well and going along and we're making lots of progress across domains. And you decided to leave OpenAI and I think you cited kind of different research areas you wanted to explore. I'm curious, like, when when did you kind of begin to know that might be something you wanted to do? And how did you ultimately make the decision?

• 25:38Speaker 01

  It's it's it's definitely not something that happens very quickly. It's a it's just something that slowly grows in a human. And OpenAI is not not an easy place to leave because I have many friends there, a lot of all short history. And a lot of I I I've built all of all of my life there. And for a long time, really, really tried hard to make it to make it work and try to try to see what are what are the various place to do it. But at some moment, if you specifically as a researcher, if you if you wake up and for any any reason figure out you don't love your work anymore, you are not incredibly incredibly excited about what you are doing.

• 26:21Speaker 01

  There's a good moment to try to explore and try to do something something else. Like, it is basically impossible as a researcher to do your your best work if you are you're not hundred percent excited. And if you don't go with full enthusiasm, there were many, many days at OpenAI when I was when I when I had basically infinite enthusiasm for the work I was doing. And was believing I was doing exactly the right things. But somewhere somewhere around the D and D was was getting it was getting harder and harder. And so, so like, you know, that's that's that's kind of kind of long story short.

• 26:56Speaker 00

  Yeah. What are some of the things that are giving you energy today?

• 26:59Speaker 01

  I think on a most fundamental level, what I did and what I've done, when I started OpenAI, I believed reinforcement learning as a necessary element of a path to AGI. And I really wanted to make it happen and I really, really did. And introducing reasoning to the world, reasoning malls to the world has been like, you know, to me, a tectonic shift and a paradigm of how we train the malls. And in some ways, I want to chase that high again and try to do something, something similar. Try to find something that is missing in how the world is training models so far. And try to make it mainstream again in one way or another.

• 27:44Speaker 01

  But once you did something like that, you know, it's hard to get another similar dopamine hits doing something else. So that's, that's kind of what I would like to do. And I would like to have a bit of freedom thinking about how to explore it and try to attack the most core, the most important problems there are.

• 28:05Speaker 00

  How much are you feeling like, hey, I've got dozens of hypotheses? Or how much are you zooming out and being like, you know, I've been so heads down in OpenAI, let me zoom out and kind of see what else has been going on?

• 28:14Speaker 01

  You know, in general, the real important hypotheses and the real important problems are most likely not something that will be new and that will appear to you. If you worked on machine learning for seven years, it's very likely you know what the important problems are. The main question is what I alluded to before, which means how do you solve it differently than everyone else? Because it means no one has solved it yet. So what is, what is different? And what is, what can be done, what can be done differently than people in the past?

• 28:52Speaker 00

  Well, I mean, I definitely want to hit on, you know, OpenAI. And obviously you had such an incredible run there and time. You know, one thing I've heard you say before was you, you know, I think you've been at OpenAI since 2019, and you said, every year felt like a different company in some ways. And so I'd love if you could just walk through that evolution and kind of, you know, the, how you kind of described the narrative of the past six, seven years there.

• 29:13Speaker 01

  Starting with a small lab, that is like thirty, forty people, where we were always, OpenAI from the very beginning was extremely ambitious, believing this is the place that will build AGI and that will create benefits of digital intelligence for the world. But starting from a few people trying to do a few kind of full projects that were incredibly ambitious to where it is today, which is again, one of the largest companies in the world, with a product that everyone knows and everyone uses. And it's almost hard to imagine not using it. It's been a wild ride. And as you also are aware,

• 29:55Speaker 01

  the execs at OpenAI have shifted throughout the years pretty significantly. So the type of people you worked with every day has changed a little bit. The size of the company has changed. The themes of research have changed. At some moment, in the old days, there was no pre-training at all. Then for a while, it became kind of the pre-training company. Then for a while, I think it became pretty much largely our own company. And now it's a little bit more in a balanced way of a mix of those two, with many, many people living OpenAI and building both businesses. And then there are their life outside.

• 30:39Speaker 01

  And many new, fresh, great people coming in and doing incredible, incredible research inside. It's just a company that manages to reinvent itself and manages to grow through that, through all the stages. Trying to imagine, I always had this weird hope, thinking of all those big, successful companies, how incredible it would be to live through such a story of being there for those stages. And I feel like I've been through quite a bit of that at OpenAI. And it's an experience that I think is very hard to compare with something else.

• 31:18Speaker 00

  You know, I think everyone's eagerly awaiting for the definitive stories to be written about this chapter of OpenAI. When those stories do get written, I feel like people always like to focus on the kind of, you know, the difficult fifty-one, forty-nine decisions that could have gone either way, that really like moved the company forward. Are there any kind of those pivotal ones that stick out to you?

• 31:36Speaker 01

  Good question.

• 31:38Speaker 01

  I was only central to some of them. There were probably many that I didn't, that I only was a background character. And probably even this discussion about releasing ChatGPT to the world or not, as you may have heard, and many people did. This popularity and its virality total didn't. Like, wasn't expected internally by at least no one I heard about it. I think in the end with ChatGPT and GPT-4 released soon thereafter, we created a bit of a moment and a bit of a momentum that was incredibly hard to predict, but made OpenAI largely what it is today.

• 32:21Speaker 01

  And that was definitely a very, very important call on many axes and many dimensions.

• 32:32Speaker 01

  Like, the decision to pool a lot of resources to train GPT-4 at the moment it is, and with a lot of trade-offs that went there, again, remained very, very important and very critical in the history of OpenAI. And it turned out to be a really good decision. And in the same way, betting and saying Resonant Mols are our future in a world where it was completely unsure of. And just a bit of first principles thinking and a bit of intuition that this is the right thing to do, allow the OpenAI to completely reinvent itself and say, "We are doing Resonant Mols right now, even though there is no product-market fit." And though they seem to be kind of cool with puzzles.

• 33:15Speaker 01

  If you look at a one, like, it was a smart model, but it wasn't really good for anything practical, except for just showing around, "Hey, we have a model that is kind of smart." Really, only with O3 and with a little bit of more investments into tool use with those models, we were able to start building something that started being incredibly, incredibly useful for research, for coding. And from there, once you have the first signs of real product-market fit, then humans are really, really good at optimizing something that exists and that they can see it works. But getting to that moment has been... Great, great journey and something, something to study because it was not an easy thing in OpenAI. At that moment, really, really passed the exam.

• 33:59Speaker 00

  Yeah, no, I mean, I think what you describe is so interesting, this idea of you, you know, have to keep scaling and investing in something, you know, not quite knowing if it's really going to work or it's working. It's obviously very relevant to some of the things you're thinking about in the, in the future, too. You know, with the, with the reasoning models, like, you know, was it clear after O1 that this was going to be more than kind of fun and games? Or what was like the, the moment, the spark for you that you're like, oh, this is really going to work and we can really scale this?

• 34:24Speaker 01

  I kind of believed in it from the very beginning, just, just because I believed in reinforcement learning. Again, my, my core belief from my very first days of OpenAI is that reinforcement learning is a necessary part of getting to AGI. And that was, the main way was how, rather than, rather than if. The question is, when are you ready to do it and how exactly do it, do we do it? And I fundamentally, but just simply I started, I know, I know this is, this is, this is what we, what we need. And over time and over, over research, there, there came various experimental results that have informed us. This is, this is the right way of doing it.

• 35:06Speaker 00

  One thing that's so interesting about OpenAI is obviously, I think people liked, I think Ben Thompson coined like this phrase, like the accidental consumer business, the idea that like, obviously you were a research lab pursuing AGI and then kind of almost accidentally stumbled upon this, this incredible consumer product that was, you know, immediately picked up by the broader world. And I'm wondering, you talked about some of the kind of different chapters of, of being there. I think a question everyone always asks about OpenAI is the company's doing so many different things, you know, they got the consumer product, core research, you know, Sora, the enterprise product, like, how does it actually work internally? And, you know, how do you kind of, do you feel that tension at all on the research side of getting pulled in a bunch of different directions?

• 35:43Speaker 01

  One thing pretty clear is that OpenAI research operates very separately from the product and has from the very beginning, almost like you said, which is that OpenAI's goal and mission is to build intelligence. And that is, I think, the main goal and motivation of majority of research. There is like one team specifically built and directed towards product research. And that is the part of research that optimizes for whatever product metrics it is. And the rest of the research mainly focuses on how do we make our models more intelligent? And that tension really doesn't exist there.

• 36:23Speaker 01

  What I think is real and interesting is that OpenAI is at the center of probably the biggest technological shift of our lifetime, which means there is so much opportunity to do various things. It is almost, it feels wasteful and it feels imprudent to not try to do all those things because basically everything in the world will be disrupted by AI. But it has a downside, which is very real and very problematic here of focus. Companies are very bad at doing multiple hard things successfully.

• 37:06Speaker 01

  Companies are well-known for succeeding in one very hard thing and then doing others similarly well to others. There are very, very few places in the world that can do a few of those things. And that is a very, very hard thing to do. And I think this is a very, very, very big risk for OpenAI. Try to do everything and then not succeed at it. But we will see if OpenAI in a half-focused state can...

• 37:39Speaker 01

  Execute well on all those side bets or there will be other companies. Famously, and I think it's a little bit sad, OpenAI really lost focus on coding for quite a while. I want to focus on the consumer product and that has costed a bunch of markets, sure, that is working very hard on regaining right now. And I think OpenAI's coding malls are already great these days. Again, but lost focus and lost lead definitely has a cost here. So there are, in some way, like when you are a company

• 38:19Speaker 01

  doing AI in the world right now, you feel like a kid in a candy store because there's so much potential of extremely valuable things that can be built for the world. And it's hard to prevent yourself from doing all of it. But for everything, there is competition and there is question who will do each one of those things in exactly the right way.

• 38:42Speaker 00

  Yeah, well, I think that's a great point to transition to just like the general ecosystem today. And you kind of alluded to coding, which I think has been a really fascinating space to watch play out. Why do you think Anthropic's been so successful at coding?

• 38:54Speaker 01

  It's focus. I think focus can explain 95% of things. What are companies, where companies succeeding in things? I know Anthropic's founders from even the time when they were at OpenAI and they were always extremely fond on coding and they always believed that it's a necessary and critical part of AGI. And I think I can only imagine how focused they have been on it over the years. And they definitely managed to get their vision very far with the latest mall with Cloud Code and coding agents. And I believe they are saying truth when they say very few people in Anthropic type code themselves these days.

• 39:39Speaker 00

  And do you think that's kind of a foreshadowing almost of, you know, pick the major labs that are out there, like each can kind of focus on different things. And so you end up with, you know, models that are particularly good at some things in different labs?

• 39:52Speaker 01

  It's a good question. And I think There are multiple worlds in which we can be. There are, there's a world where data matters. In that case, data is a very zero-sum game where like you put the data into the skills that you want and your model is better at that skills. And in that case, we see splintering of the, of the market.

• 40:14Speaker 00

  Because it's basically you can shift the data mix of what you're putting in, but ultimately it's at the cost of some other skill.

• 40:19Speaker 01

  Exactly, and, and it's a little bit about, about shifting the data mix, but mostly shifting the work. Data is a, is a labor-intensive and slightly more, less money-intensive, but, but it's, it's largely very much labor-intensive work. In terms of, in terms of research engineering, let's, let's call it that way. And so, so with that, just, just you have so many researchers that can work parallel, in parallel, doing, doing, doing useful, useful work on preparing the next dataset. So if, if, if data drives the improvements, then we will see different labs being, being better, different things in natural specialization trade-offs. But if research is king, I think, I think the research has this magical property that is, is hard. It is, is high-risk, high-reward.

• 41:04Speaker 01

  But at the same time, if you have a good idea in the research, it could improve your model in all domains at the same time. And you can, you could leapfrog everyone in all domains, uh, naturally by just, by just training better models. And, and like which one, which of the world, which of the futures we are in, it's, it's very hard to say right now.

• 41:25Speaker 00

  I mean, this current paradigm feels almost like very anti-bitter lesson, right? Where everyone's going off into specific domains and like, you know, specializing and, and really putting in those. And you're, you're right, that it feels that there's this intuitive feeling almost that there must be something else that is, that is a bit more generalizable.

• 41:38Speaker 01

  I'm, I'm pretty sure there is. The question is like how easy it is to find. There, there is one, one version of the world where, which is not completely impossible, uh, although, although slightly pessimistic on humans, that says coding agents are so good right now. Let's first get them to the moment where they can automate AI research and then have the models, like, like research, better models. Because, because maybe we are at the, at the last mall that human has. Humans could have figured out. And it's not a completely impossible framing. It's maybe it makes sense with all those GPUs we have and all those really, really capable models and their tenacity, maybe they should be researching future models.

• 42:20Speaker 01

  But maybe there are still a few things humans can do that we can, that we can put our, put our heads to work.

• 42:28Speaker 00

  I feel like a lot of the top AI researchers are working on coding for that very reason, right? This belief that we'll, we'll speed things up. You've obviously worked on coding too. I mean, you spent time, you know, a lot of time on Codex. How do you think about like the next frontiers for these AI coding products? And it seems like we're on this like just crazy exponential hill climb right now. How do you even conceptualize like what these products will be able to do in a year?

• 42:49Speaker 01

  Fundamentally, the whole, the whole story of coding is that we are able to program computers at a higher and higher level of abstraction in many ways, and we need to know less details and need to track less details. And coding agents in some ways can be thought of as a higher level programming language that has very different semantics from all the, all the, all the other programming languages. And I think it's a trend that will, I think it's pretty unlikely that in the future we will be typing code ourselves.

• 43:31Speaker 01

  Very few people already do and it's, it's a, it's a one-way ticket. But at the same time, software is important. Software needs to be reliable. And there will be more and more progress in how do we get certainty about software doing the right things if we are not the ones typing it? And maybe not even the ones reading it. Fundamentally, I think those are all solvable problems. And I am very excited about them.

• 43:56Speaker 00

  It'll be interesting whether the core skill set to work with these agents, you know, looks somewhat similar to a software engineer or really everyone just becomes a PM. And it's really just about knowing, you know, having some idea of what to go do. And, and the model's going for it.

• 44:09Speaker 01

  Yeah, I think it's an interesting question because I think almost the most important skill right now is being a skill of a good manager of junior software engineers. A lot of software engineers have been a little bit reluctant of going to management.

• 44:28Speaker 00

  Of course.

• 44:29Speaker 01

  They really, really like to specialize in narrow domains. And I think for a long time, it was really, really the right thing to do. And understanding deeply systems is incredibly important. You need to combine deep understanding of computer systems with being able to give back at least a little bit of control, which is what I think really the best managers are. The best managers understand extremely deeply the work of people on their team, but at the same time are able to give back some control for the people to drive their own destiny. And that's probably the best way how to work with malls these days. You kind of everything that the malls do, you understand the trade-offs and you understand

• 45:12Speaker 01

  what they are doing, but you allow the malls to make their own choices and live with the consequences of them one way or another.

• 45:19Speaker 00

  You know, around coding, one thing people are also trying to figure out is just where the various applications kind of fit in. And I think, you know, there's, I guess, a few questions, like one being, you know, hey, like you have the Codex, the Claude Code team sitting, you know, right by the research teams. They build these great harnesses and products. Like, you know, how do you think about the, you know, the opportunity for companies like Cursor and Cognition? And to what extent is it like a disadvantage to not be, you know, sitting next to the researchers at the labs?

• 45:46Speaker 01

  It definitely is a disadvantage. I think the fact that the most successful AI companies of the world are training their own models is telling about something. And I kind of think, just like the future of big AI companies is to become hyperscaler and run their own data centers instead of renting compute, the future of successful AI application companies to start training the malls themselves. This is just how the stack works, but you've got to start somewhere. So it may be the path that you start. With with the AI application that is successful and then you first start post training your own models. And then you start pre training your own models if you are if you are more and more successful and then you start building your own data centers.

• 46:30Speaker 01

  And those are those those could be just just the natural natural paths and stages of of success of being a good being a good AI business.

• 46:39Speaker 00

  So you think it makes sense for these companies to kind of do reinforcement learning on their own user data, you know, post train a bunch and. I mean, do they have any hope of catching you have, you know, endless compute going toward the big labs, you know, endless collection of talent like. If you if you focus on a very, you know, specific domain, do you kind of have a hope of building a better model there or. Is it kind of a almost a hopeless task in some ways?

• 46:59Speaker 01

  Nothing ever is hopeless task, the future, the future is not determined, but it's some of that is what I what I hinted before, which is. Is the data important or is the research important? If the data is important, you can always try to differentiate yourself with the data, but it is it's not clear we are we are really in that world. Maybe there is a there is a world where research is important, but that also allows the smaller companies to do some research that they think is better. And maybe maybe win through win for research in the market.

• 47:26Speaker 00

  But it seems like it almost requires a world where, you know, you have specific types of models that are focused on things being better, right? And not generalizing, right? So we get into a world of generalized models, it feels like it becomes hard for. An application company focused on any specific task to have like the better pre trained or or large model.

• 47:42Speaker 01

  Sometimes innovation comes from constraints. I think it is possible for a company focused on a specific domain. Through seeing the deficiencies of the malls on this domain to create a mall that is generally better. I I don't think it's possible and that could be like the next layer of of success of this company. If you try to do really, really good model for X and suddenly for that you make. The best mall for everything else and then you grow and then you become another another big, successful company.

• 48:14Speaker 00

  So like in the past, the problem has been you get better for like one second and then the next generation of models come out and you're like, oh, man, we're way behind again.

• 48:21Speaker 01

  Competition is difficult and definitely we have seen for a while in the US tech landscape, how big companies have tons of advantages, and this is true. But at the same time, there are new big, successful companies coming up. So it's not hopeless, it's just hard.

• 48:39Speaker 00

  Well, I want to shift gears to the talent ecosystem and maybe research itself, because obviously you both are an incredible researcher and have worked with amazing researchers. Maybe to start, obviously researcher hiring is very competitive today. I know you were probably on the forefront of bringing folks into OpenAI. What determines what companies researchers join today?

• 48:56Speaker 01

  That's a good question. In the end, people are very complex, definitely more complex than the models these days, which means everyone's incentives are different, what they want. And I honestly cannot really generalize. Whoever is hiring people should not think about how do I convince the most of the people to join me and how do I become the most appealing place for researchers to do. It's probably a good question to be asking yourself. But I think there is a second one, which I think is much more important. What type of researchers would really want to work here and then find those?

• 49:39Speaker 01

  Because it's kind of impossible and very difficult to try to appeal to everyone just because of diverse preferences and diverse viewpoints and diverse ways of working. So it's much better to try to build a team that has some shared values, some shared approaches, because it's pretty clear that the teams that are aligned and have the same goal move faster and work better than the teams that are not. So it's really, it should be kind of filtering on both sides and trying to just find the right

• 50:24Speaker 01

  people for the right group. And that makes everyone happy, that makes the group successful, and that makes the group more attractive over time.

• 50:33Speaker 00

  Yeah, but there's been some interesting experiments around this, right? I think you had Meta famously offering the first offer of these mega packages. What's your reaction to that?

• 50:42Speaker 01

  There are different strategies how to build a research group. And Meta, at some moment, I think there are supply and demand curves. But they needed to make offers really attractive to start bringing people back after a few missteps in that world. And the momentum is a very hard thing to stop. If at any moment there is a perception in the industry that you are not doing very well, then you will not hire people and then it can roll itself. So in many ways, I think it was a really, really good strategy to try to change that

• 51:21Speaker 01

  dynamic and change the negative momentum in a place where AI is so important for every large-scale business. And Meta has a very much new team built out that is training a new model these days. And a lot of people in the industry are watching. Will it be successful and how it will be successful? And that will be determined in the future of this lab. But it was definitely a good moment to bring in some new life into their Meta AI efforts.

• 51:54Speaker 00

  I mean, you've obviously done a ton of groundbreaking AI research. You've worked with other great AI researchers. What makes a great AI researcher?

• 52:01Speaker 01

  It's a good question, and it's a difficult one. Because in many ways, being a successful AI researcher, in many ways, it is about just being in the right place at the right time.

• 52:17Speaker 00

  That's very humble of you.

• 52:19Speaker 01

  But there is something to that. I think the fundamentals of being a great AI researcher these days, but in the span of my career, are one thing, being extremely good at both systems and engineering levels, understanding how computers work and how neural networks are trained, together with the theory of neural networks and optimization.

• 52:43Speaker 01

  It's very, very hard to be very successful doing only one of those things well. And suddenly, if you are at least okay in both of those things, it makes you easily 10 times more productive in any research endeavors. And I think this is an important part.

• 53:02Speaker 01

  The other one is, if you want to be a successful researcher, you very necessarily need to have some ability to think independently,

• 53:13Speaker 01

  be able to get away from group think. People have kind of some natural tendency of converging on a median viewpoint of a group, which is, which kind of kills research. I have a saying that you have 100 researchers that think the same thing. You essentially have one researcher. It being a researcher means being slightly contrarian all the time, because you want to work on something that is not working yet and that, by default, people don't really believe in. And that's, being contrarian means, there's something that a lot of researchers that are extremely brilliant and extremely hardworking are unfortunately missing is something you can call it courage.

• 53:59Speaker 01

  And some like it is about standing up and saying, "Let's try to do something different. Let's do something in a way that most people don't believe in yet." And it's an extremely hard thing to do, especially in a world where experiments are so expensive as they are and they involve so many things. In a world of machine learning, experiments being the cost of Hollywood movies,

• 54:27Speaker 01

  directing one is, you know, it does require a lot and you don't know. Similar in the movies, you don't know if the movie will be successful, but with a big budget, you'd risk it buying stars and doing the best CGI you can. In a world of ML, you also try to, try to the risk and bring as many, as many things as you can, stack as many advantages on your side. But in the end, experiments are experiments and they are meant to be going to the unknown and either will be, will be successful or not. But I think, I think like if I were to say this is like knowing both systems and theory very well. It is about...

• 55:07Speaker 01

  Not following group thing that much and then having the courage to state that to other people.

• 55:13Speaker 00

  Well, we always like to end our interviews with a quickfire round where I stuff like all the other questions I couldn't fit in elsewhere into the end here and get your rapid reactions. And so maybe to start, what's one thing you've changed your mind on in AI in the last year?

• 55:27Speaker 01

  Probably the last thing I meaningfully updated on is that I don't think a static model can ever be AGI. Like that continual learning is a necessary element of what we are pursuing.

• 55:45Speaker 00

  Is that just because of something it won't be able to accomplish? Or is it more that, you know, that it doesn't just meet the definition of AGI needs to have a continual learning aspect to it?

• 55:55Speaker 01

  It's mostly about uncovering what our models are still missing. In many ways, you go layers and layers deep because our models are already good into so many things. But them being as brilliant as they are, it is clear that without it, it will never feel to me that intelligent. It will still be a tool that needs to be supervised by someone who has the ability to continuously learn.

• 56:21Speaker 00

  Yeah, there's obviously a lot of AI progress happening in spaces we didn't talk about today. Like, what's your kind of timeline for, I don't know, a chat GPT-like moment in the robotics space?

• 56:32Speaker 01

  Probably around two, three years from now.

• 56:34Speaker 00

  That's pretty bullish. I feel like everyone's still trying to figure out if there are scaling laws in robotics or if this stuff will actually, you know, if there's enough data.

• 56:40Speaker 01

  I think, like, you know, honestly, between you and me, I think things are slightly better than most people realize. There are tons of companies making tons of progress, but always the progress needs some time to play out and some more investments to happen. But I think robotics will be doing pretty well over the next few years.

• 56:59Speaker 00

  Yeah. And a similar sense on biology?

• 57:01Speaker 01

  I think biology will take longer.

• 57:04Speaker 00

  Why longer than robotics?

• 57:08Speaker 01

  Just thinking from perspective of how much intelligence there is needed, how much precision to really manipulate biology successfully. It's a harder problem and require more fundamental investments to start to work.

• 57:25Speaker 00

  Yeah, I guess most like three, four year olds figure out how to manipulate things in the world, but they're not like world-class biologists.

• 57:30Speaker 01

  Something like that, yes.

• 57:35Speaker 00

  What's one impact of like this continued model improvement we're on that you think maybe we're underestimating or not talking about enough as a society?

• 57:42Speaker 01

  It's hard to say, but fundamentally widely deployed work automation will be reality over the coming decades. And in one hand, we are talking about it, but on the other, I think we are not talking about it enough and seriously enough because the world will change. The world will change very drastically from where it is today. If there are some people who it is still not obvious today, like it is obvious to me at least. And like societal changes are slow and I think it will be extremely weird. I think it will be probably painful in some ways and we should try to figure out how to make it the least painful.

• 58:24Speaker 01

  But we need to think about how does the world look like or the job market is looking very, very differently than it is today.

• 58:33Speaker 00

  I mean, I guess related to that, one thing I'm interested in, obviously been thinking about is, you know, whether, you know, your kind of thinking about this has impacted the way at all you, you know, you act as a parent, right? And how you think about, I guess, you know, raising kids and then if AI has changed that at all?

• 58:48Speaker 01

  The interesting thing, and I think I differ here to most parents, but I don't know, I'm not, I'm not, my daughters are very, very young and very, very, very small. So there isn't, there isn't really that much, but I'm definitely not pushing them very, very hard to study these days. It's kind of hard to imagine how hard you can push seven-year-old to, to study, but I don't think I will be the parent saying, oh, you have to be the best in maths in your class. I don't think you will be like, oh, you have to win all those competitions. I don't think you have to be like, you have to read all those books because like. You know, being a specialist, like when they will be grown up,

• 59:28Speaker 01

  probably will like look very different and it doesn't make sense to really, really bet on like trying to fight for your place in a job market anymore in that, in that way. I like there are clearly the ability to think critically will be always important, because if you don't, if you don't think for yourself, no one else will, even, even, even AI will not, will not fully advocate for you in the way you want. But hmm, I, given, given how many, how many unknowns there are, I like, at the moment, just, just want them to have happy childhood and have, have a good time as much as they can.

• 1:00:10Speaker 01

  And I hope, I hope they also have, have happy adulthood if we don't, if we don't screw up this, this, this AI deployment thing.

• 1:00:18Speaker 00

  Yeah, I guess that, you know, everyone likes to talk about the existential risk side. Have like, has your worries about that gone, gone up or down these past years?

• 1:00:24Speaker 01

  I fundamentally think and hope that no human wants the, the, the humanity to go extinct, which is a pretty good, like the incentives are aligned for everyone in the world for their existential risk not materialize, which, which I really, hmm, I'm not like subscribing to that, that it is that easy to just make a mole that will, that will, that will paper clip all of us. I think there are enough humans in the loop through all, throughout all the process that just because it is in no one's interest that we will, we will manage to be, to be successful enough and the people controlling the biggest clusters in the world will be, will be responsible

• 1:01:09Speaker 01

  enough, because it's also not a good business to kill everyone. So capitalism is also, is also fully, fully aligned here. I think it's much more dystopian and much more worrisome if we... Like push entertainment so far that is more interesting in the real world than if humans will want to live in VR and only play virtual games, which is like not impossible. I think much more, much more realistic than all of us going extinct, but probably similarly bad.

• 1:01:40Speaker 00

  Yeah, now it's like Ready Player One and Wally and all these kind of like depictions of that world before.

• 1:01:45Speaker 01

  Yeah, so that to me is much more worrying, but this one we can only solve ourselves really. And this is not an AI problem, this is a human problem. And what should our preferences be? So that's kind of how I've been thinking about it.

• 1:02:00Speaker 00

  Yeah, I love that. I guess anything you're comfortable sharing at this point about what's next for you?

• 1:02:05Speaker 01

  It's still very early, but I'm thinking.

• 1:02:09Speaker 00

  Awesome. Well, Jerry, this has been a fascinating conversation.

• 1:02:12Speaker 01

  It's been nice chatting with you and thank you very much for inviting me here.

• 1:02:16Speaker 00

  I'm Jacob Baffron and you've been listening to Unsupervised Learning, a show where we probe AI's sharpest minds on what's true in AI today, where the space is going, and what it means for businesses in the world. I love doing this podcast alongside my day job as the Managing Director at Redpoint, where I've led investments in companies like Legora, or Bridge, and Fiscal Intelligence. If you enjoyed it too, and found today's episode valuable, please subscribe on YouTube or follow us on whatever platform you're listening on. It's the best way to support the show, which helps us to continue to grow and get access to the best possible guests. Thank you so much for your support and listening. We'll see you next episode.