Ep 51: Former Chief Research Officer of OpenAI Bob McGrew - What Comes Next for AI?

In our new world of AI, few minds shine as brightly as Bob McGrew's. Until November Bob was the Chief Research Officer at OpenAI, and before that led Palantir’s engineering and product management for the first decade of its existence. He’s seen it all and we were fortunate to get his insights and vision for the future in one of my favorite episodes of Unsupervised Learning to date.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Bob Agrew was the chief research officer at OpenAI for six and a half years. He recently left a few months ago, and we had the privilege at unsupervised learning of being one of the first podcasts he's come on. So it was an opportunity to ask him literally everything about the future of AI. We talked about whether models have hit a wall. We talked about robotics models, video models, computer use models, and the timeline and capabilities that Bob envisions in the future. We talked about OpenAI's unique culture and what made research so effective there, and also some of the key decision points and what it was like to live through them. We hit on why AGI might not feel that different than how things feel today. And Bob also shared why he left OpenAI and what's next. I think folks are really going to enjoy this episode. Without further ado, here's Bob.

• 0:38Jacob Effron?

  Well, Bob, thanks so much for coming on the podcast.

• 0:40Speaker 01

  Thanks for having me here. This will be fun.

• 0:41Jacob Effron?

  Really excited to have you. I know we'll hit on a bunch of different topics. I figured we'd just start with a question that I feel like is top of everyone's mind right now, which is the kind of big debate on if we've hit a wall with model capabilities. And so we'd love your thoughts on that and the extent to which you feel like there's still more juice to squeeze on the pre-training side.

• 0:57Speaker 01

  Yeah, I think this is probably the place where the outside view of people who are watching and the inside view of people who are at the big labs diverges the most. I think if you're coming at this from the outside, a lot of people first start paying attention to AI with ChatGPT. And then six months later, boom, GPT-4 comes out. And it feels like everything's accelerating very fast and progress is happening. And then GPT-4 came out a year and a half ago. Everybody knows it had been trained beforehand. And so what's happening? Why is nothing coming out? Right? And the view from inside is very different. And on the outside, people, are we hitting a data wall? What's going on?

• 1:40Speaker 01

  But you have to remember that to make pre-training progress in particular, it involves increasing the amount of compute by really a huge amount. So to go from GPT-2 to GPT-3 or GPT-3 to GPT-4, that's 100x increase in effective compute. That's what that increment means. And so you get that by a combination of just adding more flop, so more chips, bigger data centers, and algorithmic improvements. And the the algorithmic improvements, you know, you can get you can get some out of them, you know, 50%, 2X, 3X would be amazing. But fundamentally, you got to wait to build a new data center. And so there is no shortage of new data centers coming up. You can just read the news.

• 2:23Speaker 01

  You can see, you know, Meta, X, you know, you know, other other frontier labs are also doing new data centers and maybe not making the news. But, you know, fundamentally, this is a very slow multi-year process. And in fact, you know, before you see, you know, you know, a full generation of GPT-4 to a GPT-5 lift, you're going to see something that's just a 10X lift. Because, you know, you know, people often forget that we went from GPT-3 to GPT-3.5 to GPT-4. The the thing that's interesting right now is that, you know, pre-training is going, you know, I think we'll we're going to have to wait and see when those this next generation of models gets released.

• 3:08Speaker 01

  But if you look at something like O1, where we've been able to make progress using reinforcement learning, by a variety of metrics, O1 is a 100X compute increase over GPT-4. And, you know, I think some some people don't realize this because the decision was made to name it O1 instead of GPT-5. But actually, like, this is effectively a new generation. And when that next gen, when that GPT-4.5, you know, that next generation is trained, the interesting question is going to be, how does this pre-training progress stack with the reinforcement learning process? And I think that's something that we're going to just have to wait and see what gets announced.

• 3:48Speaker 02

  That begs the question, you know, with the multi-year process going into 2025, do you think there's going to be as much progress this coming year in AI as the last year? Or do you think that things will begin to get slower?

• 3:58Speaker 01

  Well, I think I think it there is going to be progress. I think it's going to be different progress. You know, one thing that happens is when you go to this next gen or any next generation, you always run into problems that you didn't see at the previous generation. And so, you know, even after the data centers are out there, it takes time for people to to work through the problems and finish the models. So the reinforcement Learning process that we use to train O1, OpenAI use to train O1. It's, you know, what it does is it creates this longer coherent chain of thought that, you know, effectively is like packing more compute into the answer. So, you know, if you have a model that takes a few seconds to generate an answer versus a model that takes, let's say, a few hours to generate an answer, that's 10,000 times more compute.

• 4:44Speaker 01

  If you can actually leverage it, right? And so, honestly, we've been thinking about how to use test time compute for, since, I don't know, let's say, 2020. And, you know, finally, I think this is actually the real answer for how to do this, how to do it without wasting a lot of compute. And what's great about this is it doesn't require a new data center. So here, there's a lot of room for, you know, this is a new technique that's just started. There's a lot of room for algorithmic improvements. And there's no reason in theory why the same sort of fundamental principles and ideas that are used to get O1 to go from a few seconds, you know, what GPT-4O can do a few seconds, to O1 spending,

• 5:27Speaker 01

  you know, 30 seconds or a minute or a few minutes to think. You can extend those same techniques to go to hours or even days, in theory. Now, just like with, you know, going from GPT-3 to GPT-4, you know, there were no foundational new techniques. The both were trained, you know, roughly the same way, but scaling is very hard.

• 5:46Jacob Effron?

  Yeah.

• 5:47Speaker 01

  And so that's actually the meat and potatoes of it is like, can you actually do that scaling? And I think, I think that's the kind of progress that we're going to see that's going to be most exciting in 2025.

• 5:58Jacob Effron?

  With kind of the focus on test time compute and, you know, the, you know, O1 being out there in the wild now. I think it's really interesting to think about how folks are actually going to use these models, right? And I think you had a recent tweet that was, I thought was really interesting about how you're going to kind of need these new form factors to unlock the capabilities of some of these models. And so maybe just expand on that a little bit. Like, have you seen kind of any early form factors that you thought were interesting working with these models?

• 6:19Speaker 01

  Well, yeah, and just to sort of explain the problem, chatbots have been out for a while and most of the interactions that people have today with chatbots, GPT-4 class models do a great job with those. You know, if you're asking ChatGPT, who is the fourth Roman emperor? Or how do I reheat basmati rice? Or, you know, most, most of sort of the, the bread and butter conversations that we have. You know, you don't have any pain. It works just fine. And, you know, when we were thinking about releasing O1 Preview, there were a lot of questions about will people use this? Will they find things to do with it? And, and I think those were kind of the right questions. It's, you know, what is it that I need to do with this model? That where I can really get the value out of it?

• 7:04Speaker 01

  You know, programming is a great use case for this because with programming, you have, you know, a structured problem where you're trying to make progress on something, again, over a long period of time and in a, in a way that, that is highly leveraged for reasoning. Another example is if you're writing a policy brief, right? You're writing a long document. It needs to make sense. It needs to all tie together. You know, the truth is most, I mean, there's a lot of programmers out there, but most people who aren't programmers don't have something like this that they need to do every day. But going back to the, the underlying breakthrough here, it's that you have a coherent chain of thought, a coherent, you know, way of making progress on a problem. And that doesn't just have to be thinking about the problem.

• 7:48Speaker 01

  That can also be taking action, making a plan of action. And so the, the thing that I'm most excited about with models like O1, and I'm sure there will be others from other labs soon, is using them to really enable long-term actions, you know, basically agents, you know, although I think the word is so overused that it's, it's, doesn't really tell you much about what you're trying to do. But I have all sorts of tasks in my life where I would like the model to, you know, book things for me, shop for me, solve problems for me in ways that involve it interacting with the rest of the world. And so I think that's the form factor that we really need to nail.

• 8:28Jacob Effron?

  Yeah.

• 8:28Speaker 01

  Is, is what is this? How do we do this? I don't think anybody's figured it out yet.

• 8:31Jacob Effron?

  It's so interesting. I mean, it makes total sense. I feel like everyone, you know, minds run wild with what, you know, these, these agents can do and all the problems they can solve for people, for businesses. What do you, what are the big gaps to that happening today? Obviously, you've seen some of the early, you know, models like Anthropic released computer use and there's, you know, I'm sure other labs are working on this as well. But as you think about kind of what's stopping us from being there today. What are like the the hard problems still to solve?

• 8:54Speaker 01

  Yeah, there's a there's a whole host of them. I think the most immediate one is reliability. So, you know, if I'm asking something, forget actions for a second, right? If I'm asking an agent to do something on my behalf, even if it's just thinking or writing some code for me, and I need to go away for five minutes or an hour and let it work. If it, you know, goes off task and makes a mistake and I come back and it hasn't done anything, I've just wasted an hour. That's a big deal. Now add into the fact that this agent is now going to be taking actions in the world. Maybe it's buying something for me. Maybe it's, you know, pushing a PR. Maybe it's sending, you know, a note, an email, a Slack message on my behalf. Well, if it does, if it does a bad job of those, well, now there's a consequence.

• 9:39Speaker 01

  I'm going to be at least embarrassed. You know, maybe I'm going to be out some money. And so reliability just becomes a much, much more important thing than in the past. I think there's a, there's a rule of thumb when you think about reliability that to go from 90% to 99% reliability, that's maybe an order of magnitude increase in compute, you know, that's one of these 10x generations. And to go from 99 to 99.9% reliability, that's another order of magnitude. So every nine is requires a huge leap in model performance. You know, that 10x, that's, that's a really big improvement. That's a year, two years worth of work.

• 10:18Speaker 01

  So I think that's, that's really the first problem that's going to be faced. The second thing that I think is interesting is so far, everything we've talked about is for consumer, right? You know, you're not embedded in enterprise. But when you're talking about something that's doing, taking, working on a task, for a lot of us, that will be something we do in our jobs, something that's embedded in enterprise. And so that, I think, brings in, you know, a whole host of other considerations.

• 10:47Speaker 02

  That's an interesting point. You know, we're seeing in enterprises today that a lot of consulting firms are actually doing really well because there's a lot of handholding to deploy to enterprises at this point. Do you think that handholding and sort of the need for help on the enterprise is going to persist for a while? Or do you think it's going to get more sort of turnkey? And enterprise will just be able to deploy these LLMs really easily in the future?

• 11:08Speaker 01

  Yeah, I think that's a really interesting question. And like, I mean, even, even like, so what is the problem of deploying an LLM in the enterprise, right? Well, Probably the thing that something needs to do if it's going to automate a task for you or do your job, it needs context. Because in consumer, there's not a lot of context. Okay, you like the color red. Fine. Not very interesting.

• 11:29Jacob Effron?

  Appreciate you choosing red for your example.

• 11:31Speaker 01

  Yeah.

• 11:33Speaker 01

  But you know, in enterprise, well, you know, who are your coworkers? What project are you working on? What is your code base? You know, what have people tried? What do people like and dislike? And all that information is out there sort of ambiently in the enterprise. It's in your Slack. It's in your docs. You know, maybe it's in your Figma or whatever. So how do you get access to that? Well, you know, you have to sort of either build something one off. I think there's definitely an approach where, you know, people build a library of these connectors. And you know, you're able to sort of like come in and do this. This is, you know, very similar to the work we did at Palantir,

• 12:12Speaker 01

  where you know, the fundamental problem that Palantir was solving was to integrate the data in an enterprise.

• 12:21Speaker 01

  You know, that I think that's one reason why something like Palantir's AI platform, AIP, is so interesting.

• 12:28Speaker 01

  So I think that that's like the first path where you're sort of like building a library of these things. There's a whole platforms that can be built off of this.

• 12:36Speaker 01

  The other thing around is the opportunity to do something like computer use. So, you know, now, instead of having this, this sort of very specific and potentially bespoke set of ways to do it, you have, you know, one hammer that can go hit everything. And the funny, and so Anthropic has rolled this out. You know, it's really funny. We were actually talking about these kinds of computer use agents before the Anthropic folks left OpenAI way back in 2020. And, you know, you know, Google DeepMind has published papers on this. Every, every lab has, has thought about this and worked on this problem.

• 13:18Speaker 01

  The, the thing that is different about a computer use agent versus one of these programmatic API integrations is that now, since you're controlling a mouse and a keyboard.

• 13:30Speaker 01

  The actions you're taking now involve a lot more steps. You know, maybe you need, you know, 10x or 100x the number of tokens that you would need if you were working with one of these programmatic integrations. And so now, again, what are we back to? You need a model that has a very long, you know, coherent chain of thought, the ability to make progress on a problem consistently over a long period of time. The exact kind of problem that one has solved. And I'm sure there are other ways that you can solve this. But I think that's going to be one of the breakthroughs that I think we're going to see used over the next year.

• 14:05Jacob Effron?

  How do you think it ends up playing out? Because I imagine, on the one hand, obviously, a general purpose model that could use computer in any context seems quite compelling. I imagine, you know, getting to 99.999% reliability might prove difficult. And also, you know, there's so many steps that could go off at different points. You know, one other vision of how this could work is I'm sure, you know, some of these problems could be simplified if like the underlying application APIs were opened up in some way, right? Or other ways to, or you could have specific models for like using Salesforce, or I don't know, some of these specific tools. You know, do integrations end up being a big advantage if you have access to the under the hood experience so you can just do things in a quick split second versus like sit there while you watch a computer, do you think, on the screen?

• 14:45Speaker 01

  Yeah, well, I mean, I think you definitely see probably a mix of these approaches where some use these integrations and others, you know, you know, the computer use becomes sort of this like backup that you can employ if you don't have something bespoke. And then maybe you see which things people use and you come up with the more detailed integration if it works. I think to the question of, you know, will you see a Salesforce specific computer use agent? That doesn't make a ton of sense to me technically. Because I think, you know, fundamentally, what you're leveraging there is the data, right? There's someone went out and collected a huge data set of how to use Salesforce. And you know, you could throw that into and sales for it would be in Salesforce's

• 15:26Speaker 01

  advantage to share that data set with Anthropic and with OpenAI and with Google. And and you know, they train their own models. And you know, I think every application provider would want that to be public and part of every foundation model. So I don't think this, you know, to me, this doesn't seem like a reason to have sort of specialized models in that way.

• 15:47Jacob Effron?

  No, it's a really compelling point because I guess also at the point where you're in a competitive space and your competitors are making their data available and their products become easier to use, you certainly want yours to yours to be as well.

• 15:56Speaker 01

  Yeah, I it's a bit of a mystery to me why, you know, that kind of ecosystem hasn't happened where people are, you know, shoving their data into the L L Ms, you know, it's the the equivalent of Google S E O.

• 16:07Jacob Effron?

  Yeah.

• 16:08Speaker 01

  Effectively.

• 16:08Jacob Effron?

  That's a really interesting point. How how far away from like, you know, widespread computer use, do you think we are?

• 16:14Speaker 01

  Well, I mean, I think there's a good rule of thumb for these things, which is you see a demo and it's super compelling and doesn't doesn't quite work yet. It's like too painful to actually use. And then, you know, give it a year and it's 10x better. And again, scaling is log linear, so 10x better, you know, it's one increment of improvement, but one increment of improvement is is pretty big is a pretty big deal. And you start seeing it being used for limited use cases. And then give it a second year. And at that point, you know, it's surprisingly effective, but you can't rely on it every single time, right? Where we are now with chatbots, where you still have to worry that they've hallucinated. Then it's the question of adoption is really about what level of reliability do you require?

• 16:57Speaker 01

  And anything where you can tolerate mistakes will get automated a lot faster than places that you can't.

• 17:04Jacob Effron?

  So I guess to go back to Jordan's original question, it makes total sense, you know, basically that today you need a lot of kind of handholding to integrate into the right data and to kind of, you know, define bespoke guardrails and workflow. What will the layer in between like, hey, great computer use model and enterprise ready to to sign up like, what's the layer in between look like?

• 17:21Speaker 01

  Man, I think there should be startups out there defining that. You know, I think we don't quite know the answer to that yet. I think one of the interesting things that you see on this is when you have these general tools like something like computer use, the problems it solves are sort of at fractal levels of difficulty where, you know, it can solve a lot of problems, but then you will see a problem that really matters and you can't quite solve it. And then you'll say, okay, well, now we're gonna do a very specific, you know, maybe we'll make a programmatic approach for something like this. So I think we're gonna see a mix of approaches for a while.

• 17:59Speaker 02

  I'm curious, you know, you've obviously been on the research side and responsible for, you know, really cutting edge research. We talked a little bit about test time compute. Are there other areas that you're particularly excited about?

• 18:09Speaker 01

  Well, I think that, you know, we've talked about pre-training, we've talked about test time compute. The other really exciting thing is multimodal.

• 18:16Speaker 02

  Big day for multimodal.

• 18:18Speaker 01

  Big day, yeah. Today is the Sora release. And really, this is, in some ways, the culmination of this really long arc. LLMs, you know, were invented, you know, let's say 2018. And one of the, it became sort of obvious that you could apply transformers and some of the same techniques to, you know, adapt to other modalities. So you include vision, you have, you know, images out, audio in and audio out. First off, these things start off as sort of side models, like a Dolly or a Whisper. And ultimately, they get integrated into the main model. And the modality that has resisted this for so long is video.

• 18:59Speaker 01

  And, you know, Sora, I think, was sort of first to demo. Other companies, you know, Runway, you know, some other models have come out in between. And now Sora itself has released. I think there's really two things that are interesting and different about video than about the other modalities. When you are creating an image, really, you probably wanna just have a prompt. It creates an image. Maybe you try this a few times. If you are a professional graphic designer, you might edit some of the details in this image. But let's be honest, none of us are, you know, a lot of the usage here is, did you need some slide ware? Did you want a picture to go along with your tweet or your presentation?

• 19:44Speaker 01

  But it's a very straightforward process. But with video, wow. I mean, it's an extended sequence of events. It is not one prompt. And so now you actually need a whole user interface. You need to think about how to make this a story that unfolds over time. And so that's, I think, one of the things that we see with the Sora release. And I think Sora has spent a little more time thinking about this. The product team has spent more time thinking about this than some of the other platforms out there. And the other thing that you always need to think about with video is just that it is very expensive. It's very expensive to train these models. It's very expensive to run these models.

• 20:27Speaker 01

  And so I think as interesting as seeing Sora quality video, and Sora, I think, is better quality. But you have to pay attention a little bit to see that it's better quality, at least if you're looking at a short snapshot. But now Sora usage is available to anyone with a Plus account. We have, OpenAI released the Pro account for $200 a month. Now that includes unlimited slow generations of Sora. And so when you have this level of quality and this level of distribution, that is actually, there's two hard problems that have been solved there. And I think that's going to be a really high bar for other competitors to face.

• 21:08Jacob Effron?

  What does the progress of video models look like over the next few years? I mean, obviously, in the LLM world, we've seen just like massive, it feels like whatever the model was last year is now 10 times cheaper and much faster. Like, do you expect a similar level of improvement on the video side?

• 21:23Speaker 01

  Actually, I think the analogies are very direct. So if I think about the differences between video models today and video models in two years, the first thing is the quality will be better. Now, the instantaneous quality is already really good. You can see the reflections. If you share things, all the, they're going to be, all the hard things that are hard to solve, you can point out, oh, look, it did a reflection there. There's some smoke. What is hard is extended coherent generations.

• 21:55Speaker 01

  So the Sora product team has a storyboard capability that allows you to put checkpoints at various, every five seconds or every 10 seconds to help be guideposts for the generation. Fundamentally, if you want to go from a few seconds of video to an hour of video, that is a very hard problem. And that's, I think, something you'll see in next generations of models. The other side, again, the other analogy is I actually think it will be very much like LLMs where if you want a GPT-3 quality token, that's 100 times cheaper than it was when GPT-3 came out. And the same thing is going to be true for Sora, where

• 22:35Speaker 01

  you're going to be able to see these very beautiful, realistic-looking videos, and they're going to cost practically nothing.

• 22:42Jacob Effron?

  I feel like the dream is like a full-length AI-generated movie that wins some award or something. To ask a shameless podcast question, like, when do you think we'll have that? What year, if you had to make a guess?

• 22:52Speaker 01

  Oh, man. Yeah. Honestly, winning an award is somehow too low of a bar, right? There are a lot of award shows, I guess. Really, is this a movie that you actually want to watch?

• 23:06Jacob Effron?

  Yeah.

• 23:07Speaker 01

  I think we will see this in two years, but it will actually be less impressive than I just said, because the reason you wanted to watch it is not because of the video, but because there was a director who had a creative vision and used the video model to exercise their creative vision. And the reason they would do this, I think, is they could do something in the medium that they couldn't film. And we can all imagine, none of us are directors here, but we can all imagine lots of possibilities.

• 23:38Jacob Effron?

  Not graphic designers, not directors, but yeah.

• 23:39Speaker 01

  That's right. That's right. Yeah, we have some very specific skills here.

• 23:42Speaker 02

  Yeah, we're seeing a bunch of companies pop up trying to be like the Pixar for AI. And it's always a question that we ask is, when is that actually feasible? So it sounds like a lot sooner than at least we would have thought.

• 23:53Speaker 01

  That's my guess. That's my guess. Once things get to the stage where they can be demoed, progress is very fast after that. And before that, progress is very slow, or at least it's invisible.

• 24:05Speaker 02

  I guess, switching gears from video to robotics, you joined OpenAI to work on a lot of robotics stuff at the beginning. Would love to just to understand your perspective on the space and where we are today and where you think it's going.

• 24:17Speaker 01

  And this is a very personal question. When I left Palantir, one of my ideas was that robotics was going to be the domain in which deep learning became real, became more than like a button on somebody's website. And so I spent a year between Palantir and OpenAI, really deeply understanding robotics, writing some of the first code I did with deep learning around vision. And it was a very challenging space. And at the time, I thought, well, maybe it would be five years away. This was 2015. And that was very wrong. But I think it's correct

• 24:56Speaker 01

  now. I think robotics will see widespread, if somewhat limited, adoption five years from now. And so I think this is a very good time to be starting a robotics company. And I think, yes, for the fairly obvious point that that foundation models are just a huge breakthrough in your ability to get robotics up and running quickly and for it to generalize in important ways. And there's a few different aspects to that. The sort of obvious one is that the ability to use vision and to translate that into plans of action. A lot of that comes

• 25:40Speaker 01

  for free with the foundation models. The slightly less obvious, maybe more fun thing is just the fact that the whole ecosystem is developed. Now that I left OpenAI, I've spent some time hanging out with founders, and I've talked to a couple robotics founders. And one robotics founder was telling me about how they can literally have set it up to talk to the robot. And that it's really cool and much easier. You can tell the robot what to do. It sort of has the gist. It uses some specialized models to go, you know, actually do the operations. But it was just a pain and a hassle to have to type out what you wanted. And, you know, you have to like sit there in front of the computer instead of watching the robot. Now you can just talk to it.

• 26:26Speaker 01

  I think one of the big differentiators that we still don't know where it will go is this question of do you learn in simulation?

• 26:33Jacob Effron?

  Yeah.

• 26:33Speaker 01

  Or do you learn in the real world? And, you know, what our, you know, major contribution, those, those two years of robotics was showing that you can train in the simulator and have it generalized to the real world. But the, and there's a lot of reasons to use a simulator if you can, you know, for all the same reasons you would do that if you're programming.

• 26:55Jacob Effron?

  Right?

• 26:55Speaker 01

  It's, it's just, it's a pain to run against production systems or against the real world. You get free testing, all that stuff. But simulators are, simulators are good at simulating rigid bodies. You know, if you're doing pick and place with things that are hard, like hard bodies, that's great. But a lot of the world is floppy. You have to deal with cloth or, you know, thinking about warehouses, you have to deal with cardboard. And your simulators just do not do a particularly good job of dealing with that. And so for anything that wants to be really general, the, the only approach we have now is to use real world demonstrations.

• 27:37Speaker 01

  But I think as you can see from some of the work that's been coming out recently, that, that can actually work really well.

• 27:43Jacob Effron?

  And I guess like, obviously it's somewhat unknowable, like, you know, when one finds scaling laws in robotics. And then also the extent of, you know, tell operated data one might need. But do you have like, uh, does it feel like we're, we're close to that? Or, I mean, obviously, you know, I guess in 2015, you thought you were five years away. How far away do you think we are from people would cheaply call it like a chat GPT like moment for robotics? Where people are like, oh, that really, that, that, that's something visceral that, that seems like it's different and works.

• 28:06Speaker 01

  Any kind of prediction, especially about robotics, you really have to think about the domain. So I am pretty bearish on a mass consumer adoption of robotics. Because having a robot in your home is scary. Robot arms are deadly. Like they can, they can kill you more to the point they can kill your kids. And you know, and you know, you can use different kinds of robot arms that don't have these drawbacks, but then they have other drawbacks. The home is a very unconstrained place.

• 28:37Speaker 01

  But I do think, you know, being in in various forms of retail or other forms of work environments. I think in five years, we will be seeing that. And you can even see this, you know, if you're, if you go in an Amazon warehouse. You know, they, they already have robots that have solved mobility for them. You know, they're working on pick and place. I think in, you know, you will see lots of robots rolled out in warehouse settings. And then, you know, it's going to sort of be domain by domain for a while. I won't offer a prediction on when it, when it's in the home, but I think you're going to see it widespread. I think we'll, we'll interact with them in our daily lives in five years in a way that, that would feel strange today.

• 29:17Jacob Effron?

  I mean, obviously, there's been these like separate robotics companies to some extent, obviously robotics leverages foundation. You know, the advances in L L Ms. I'm curious, like the, you know, does this all kind of converge? You know, obviously there's, there's companies that just do video models. There's companies that are focused on bio, on material sciences. Like, as you think about where this goes long term, you know, is there one massive model that just knows all of this?

• 29:38Speaker 01

  At the frontier model scale, I think you should continue to expect the companies are going to put out one model. It's going to do the best of everything on every axis for every form of data they have access to. So that's an important caveat.

• 29:55Speaker 01

  What specialization really gets you is price performance.

• 29:58Jacob Effron?

  Yeah.

• 29:59Speaker 01

  So over the last year, you've seen the frontier labs get much better at having very small models that have lots of intelligence that can do, you know, chatbot like use cases very cheaply. And, you know, if you're a company at this point, you know, a very common pattern is that you figure out what you want the A I to do for you and. You train that against your, you run that against the best frontier model that you like, and you generate a huge database, and then you fine tune some much smaller model to do that. And, you know, it's, this is such a common, OpenAI offers this as a service.

• 30:40Speaker 01

  I'm sure, you know, this is a common pattern on every platform. And, and you can say, you know, and this is just tremendously, tremendously cheaper. Now, if, you know, you've trained a chatbot like this, your customer service chatbot is trained like this, and someone goes off script, it's not going to be as good as it was if you would actually use the frontier model natively. But that's fine. That's, that's the price performance that, that people are willing to take.

• 31:06Jacob Effron?

  One thing I thought was really interesting is when we were talking before, you kind of mentioned this, this macro point about AI progress, about basically that in 2018, we'd said, hey, we're going to be in 2024, and we're going to have all these model capabilities that you would have kind of thought for first principles, like, okay, these things have just completely changed. Like, the world is almost unrecognizable relative to what it was in 2018. And, you know, while certainly you guys have had a massive impact on the, on the broader world, I wouldn't say, you know, yet AI adoption has, you know, completely changed the, the entire way the world works. Like, why don't you think that's been the case?

• 31:38Speaker 01

  Yeah, I mean, just, you know, to restate this a bit, I think, weird as it sounds, the, the right mental frame to have about AI is, is deeply pessimistic. Like, why is progress so slow? Like, why, you know, we're talking about a, if, you know, some people say that AI has led to a 0.1% increase in GDP growth, but that's not because of productivity from using AI. That's from capital expenditures of building out all the data centers to train the AI. So, you know, why is the AI not visible on the productivity statistics, right? Just like people said about the internet in the 1990s. And I think, you know, there's

• 32:21Speaker 01

  a few reasons for this. And, and for starters, you know, that, that view from 2018 that, you know, well, you know, once it, you can talk to it, and it can write code, then, you know, everybody's going to be automated right away. You know, it's the same kind of view that you have when, you know, you're an engineer and someone asks you to write a feature and you're like, oh yeah, I can totally knock this out in a couple of weeks. And then you you start writing the code and you're like, oh, well, you know, actually, there's a lot more to this feature than I realized. Uh, you know, and if it's a good engineer, you know, offhand, they think two weeks and they they sort of plan it out and they estimate it takes two months. And if it's a bad engineer, well, it's, you know, something that they couldn't write at all. And and I think that's what happened as we try, as we really dug into like,

• 33:05Speaker 01

  what is it like for a human to do a job? Well, you know, yes, you may talk to them on the phone, but that doesn't mean that the thing they were doing was talking to you on the phone. You know, there was, there was, you know, real work that they were doing. And, you know, fundamentally, the thing that AI can automate is a task. And a job is composed of many tasks. And when you dig into real jobs, you find that most of them, you know, for most jobs, there are some tasks that cannot be automated. Even if you look at programming, you know, it's the boilerplate gets, you know, optimized first. And maybe the last thing is this sort of, well, what is it I'm even trying to do? Like the the giving direction part.

• 33:50Speaker 01

  And so I I think as we continue to roll out AI, we're going to find more and more of that.

• 33:59Speaker 02

  So I guess with that in mind, in terms of progress, what's an area that you think is underexplored today? That should be getting more attention than it is?

• 34:07Speaker 01

  Hmm, I, okay. One answer here, the the kind of startup that I'm really excited about are startups where people are taking AI and they're applying it to something really boring. So imagine that, you know, you were running a company and you could just hire all the smart people you wanted to do something super boring, like, look at all the places where you're spending money and make sure that you're comparison shopping appropriately. Like, you know, if if if your procurement org was full of like Elon Musk quality people who are who are really being careful about their spend, you know, you could probably save a lot of money. And no one does that because, you know, the kind of people that it would take to really

• 34:49Speaker 01

  save money, you know, uh, they they would get bored, you know, they would hate this job, right? But AI is infinitely patient. It doesn't have to be infinitely smart. And you know, I think anywhere where you, you know, if you're running your business, you could get value out of, out of people who are infinitely patient doing something. That's something that, that AI should be automating.

• 35:16Jacob Effron?

  It's interesting. I always thought of like consultants as this like arbitrage of getting smart people to work on like kind of boring problems or in boring industries. And obviously with, you know, top cutting edge AI models, it's like you can get these, you know, someone with a brilliant IQ to work on like a problem. You'd never be able to get a smart person to work on. It's a really interesting point.

• 35:33Speaker 01

  Yeah. Well, I mean, the, the, the first time I heard that people had done productivity studies where they show that, you know, AI really helps like 20, 20, 50% improvements. I was like, wow, that's amazing. I'm like, oh, it's consultants. Well, you know, AI is really good at bullshit and consultants are, you know, their job is to produce bullshit. So maybe we shouldn't be surprised that that's where the productivity gains are showing up first.

• 35:55Jacob Effron?

  Yeah, I think it was also the biggest in the, in the bottom half of performers. Right?

• 35:59Speaker 01

  That's right. Well, I mean, actually I think that's kind of hopeful because if you look at the bottom half of performers, you know, they are, you know, like the hopeful version of this is they had the skills that humans have that are hard to automate. You know, they knew what they were trying to do, but they couldn't figure out how to write the code to get there. And then the model comes along and it's like, oh, I know how to write the code, how to get there. I don't, you know, I don't know how to figure it out what it is that I should be doing. And so, um, you know, now the, these bottom performers can, can actually, you know, really improve in their jobs. So I see that as very hopeful.

• 36:38Speaker 02

  I guess in terms of performers, I mean, you are and have worked with some of the best researchers in the world. What do you think makes an AI researcher the best?

• 36:46Speaker 01

  There are many different kinds of researchers that do different things. So if you think about someone like Alec Radford, who invented the GPT series, but also Clip, I mean, you know, basically invented LLMs and then, you know, went on a tear through all forms of multimodal. Um, you know, Alec is someone who does his best work, you know, with a computer alone at odd hours of the night. You know, by comparison, you know, other brilliant people like Ilya Sutskever or Yanki Pachoki, who are the first and second chief scientists at OpenAI. You know, those guys have have sort of big ideas and big visions and help work a lot

• 37:27Speaker 01

  through other people and helping set out a whole roadmap for the company. But the thing that I think is really common for the very best scientists is that they have a certain amount of grit. I will always remember watching Aditya Ramash, who invented DALL-E, just just work through this problem where he, at the time, the the original idea for DALL-E was to see if we could generate a picture that was not in the training set, to prove that neural networks were were creative, they weren't just sort of memorizing and rehashing. And so he wanted to generate a picture of a pink panda skating on ice, which he was

• 38:10Speaker 01

  certain was not in the training set. And he worked on this for 18 months, maybe two years. And I remember, you know, a year in, Ilya came by and brought me a picture and he's like, look, this is the latest generation. It's really beginning to work. And it was just this blur. And Ilya's like, you can see up here, there's like pink up here and it's white down below. You can really see the pixels starting to come together. And I, I, you know, I couldn't really see anything, but but he just kept working at it. You know, this is this has every researcher that, you know, really succeeds at one of these foundational problems, they have to see this as their hill to die on and they

• 38:54Speaker 01

  are just going to go after it and keep working on it, you know, for years if necessary in order to make it work. What have you learned about putting together like a research organization with a bunch of folks like that?

• 39:06Speaker 01

  Well, you know, the funny thing is, the best analogy I have actually comes from Alex Carpenter Palantir, which is, you know, he always used to say that engineers were artists. And it makes a lot of sense, you know, when you, when you talk to a really good engineer, they just want to create, they have something in their heart, and code is, is their way of bringing that sculpture to life. And, you know, at Palantir, it was, well, you know, you know, you have to make them fix bugs, but every time you do, you know, the artist part of them is going to be sad. You have to have a process in order to make people, you know, you know, work together, but the artist part of them will be sad. And the truth is that, you know, an engineer is an artist, and a 10x engineer is 10x an artist,

• 39:48Speaker 01

  and a researcher is 100x the artist of any engineer. And so, you know, to, to build an organization with researchers, there's so much more, you, you cannot, you know, there's, there's, there's a version of, of engineering management where you say, it would be great if everybody were interchangeable parts, and you had a process, and then you can sort of make them work together. And working with researchers, it's very high touch, because the most critical thing is you cannot snuff out the artistry, which, which is what, you know, it's, it's because they care about the vision they have in their head, that makes them willing to do, go through all of the pain that it takes to actually produce that vision and make it a reality.

• 40:33Speaker 02

  You're fortunate to have worked at both Palantir and OpenAI, and there's been a lot of articles going around about how Palantir's culture was really special. When you think about OpenAI, I'm sure there'll be many articles in the future about its culture. What do you think those pieces will say?

• 40:48Speaker 01

  Yeah, I mean, I think, you know, one piece of it is, is working with researchers, like we talked about. The other thing that is just crazy about OpenAI is how many times it has pivoted, or even, I like to think of it as being refounded. So when I joined OpenAI, it was a nonprofit, and the vision of the company is that we would build AGI by writing papers. And we knew that that was wrong. It didn't like quite smell right. You know, a lot of the, the early people, Sam, Greg, you know, me, were startup people. And you know, this this path to AGI felt felt wrong. And so, you know, you know, after a couple of years, there was a transition from a nonprofit to a for-profit.

• 41:31Speaker 01

  And that was very controversial within the company. In part because we knew at some point we would we'd have to interact with products. We'd have to think about how to make money.

• 41:41Speaker 01

  The the partnership with Microsoft became another sort of refounding moment, another very controversial thing. I mean, maybe it's one thing to make money, but but then to give it to Microsoft. I mean, to big tech. I mean, wow, that's terrible. And then, you know, you know, and then again, of equal import was the decision to say, well, not only are we partnering with Microsoft, but we're going to build our own products with the API. And then finally, the move, you know, to add consumer to enterprise with ChatGPT. These are all, you know, the kinds of any one of these pivots would be definitive for a startup. And at OpenAI, it was, you know, every 18 months, every two years,

• 42:24Speaker 01

  we were fundamentally changing the purpose of the company, the identity of the people who worked there. Right? And we, again, you know, we went from writing papers was your job to, you know, building one model that everyone in the world can use. And the really crazy thing is, I think this is what we actually, you know, if you'd asked us back in 2017, you know, what is the right mission? You know, wouldn't have been getting to AGI by writing papers. It would have been, we want to build one model and everyone in the world can use it. But we didn't know how to get there. And so we just had to explore and and and find all these things out along the way.

• 43:01Jacob Effron?

  What do you think enabled you guys to be so successful in making these these big shifts?

• 43:05Speaker 01

  Well, I mean, necessity for one. None of these were, none of these were were sort of freely chosen. You know, you have a nonprofit, you run out of money. Maybe you need to find a way to raise money. Maybe to raise money, you have to be a for-profit. You know, your partnership with Microsoft, you know, maybe they can't, you know, they aren't seeing the value in the models you're creating. So you need to build an API because that might actually work. And then you could show them that people actually want these models.

• 43:35Speaker 01

  You know, ChatGPT, this this is one. I think we actually, you know, really did believe after we, after G P T three, that with the right sets of advances, the right form factor was not just an A P I, where, where people had to go through an intermediary to talk to the model, but that the model would be something you could just converse with directly. And so that was one that was, I think, very deliberate. But, you know, somewhat famously, the way that it happened was an accident. You know, we were working on this, we'd actually already trained G P T four. And the, the, we, we had wanted to release when

• 44:19Speaker 01

  the model was good enough that we all used it every day. And we all, we all looked at Chat G P T, you know, in November. And we're like, ah, did it pass the bar? Not really. And John Schulman, one of the co-founders who led this team, said, look, I really just want to release. I want to, I want to get some outside experience. I remember thinking if a thousand people used it, that would be, that would be success. You know, we had a pretty low bar for success here. And we made the fateful decision of not putting it behind a waitlist. And then, you know, again, the world first forced our hand. And, you know, suddenly everybody in the world wanted to use it. What were those first days like when you, when you released it? Oh, my gosh, they were, they were very intense.

• 45:01Speaker 01

  At first, there was a certain amount of disbelief that this was actually going to happen. There was, there was some anxiety. We were quickly trying to figure out, how do we get the G P U's? So we, you know, we temporarily repurposed a bunch of research compute over there. And then there was this, this question of, you know, when is it going to stop? Is this going to keep going or is it going to be a fad? Because, you know, we had almost gone through something similar with Dolly. The Dolly two model was, was a big internet sensation. And then it died back. And so there was this concern that, in fact, Chat G P T would disappear. And

• 45:39Speaker 01

  This was this was one place where I, you know, pretty strongly believed that that it wouldn't, that this was actually going to be bigger than the API. And in retrospect, I was right.

• 45:48Jacob Effron?

  I mean, what a fascinating experience. I guess, you know, one of the the cool things is you're just so close to this cutting edge AI research, you know, I'm curious, like, what's one thing you've changed your mind on in the AI world in the last year?

• 45:59Speaker 01

  You know, the really funny thing is, I don't think I have changed my mind. After G P T three and, you know, and into the, you know, 2020, 2021, if you were on the inside, a lot of the things that needed to happen over the next four or five years felt kind of obvious, you know, we were going to have these models, we were going to make the models bigger, that we're going to become multimodal. Even in 2021, we were talking about how we needed to use RL on the language models and try to figure out how to make that work. And there's a real truth that the difference between 2021 and 2024 is not, you know,

• 46:44Speaker 01

  what needed to happen, but just the fact that we were able to make it happen at all. And, you know, we, the whole field was able to make this happen. But in some sense, also, what where we are now feels a little bit predestined.

• 46:57Jacob Effron?

  I guess, looking forward, as you think about scaling pre-training and scaling test time compute, like, does it also feel like you're kind of predestined to reach AGI on those two alone? Or like, how do you think about that?

• 47:06Speaker 01

  I have a hard time conceptualizing what AGI is. And I think, you know, if anything, I think I have a deep critique of AGI, that there is no one moment that, in fact, you know, these problems are fractal. And, you know, we're going to see more and more things automated. But somehow, we're, I don't know, I have this belief that it's going to feel very banal, that somehow we're all going to be taking our self-driving cars to our offices where we boss around armies of AIs, and we're like, oh, it's kind of boring. It's because somehow this still feels like office space, and my boss is still in it. And that's what, you know, that's what our AGI future will look like. And, you know, we can't, we won't be able to wait until 5 P M rolls around or something,

• 47:50Speaker 01

  you know, but but on a more serious note. I have felt for a while, and again, I think this was a common view within OpenAI and probably within the other frontier labs, that solving reasoning was the last fundamental challenge that was needed in order to scale to human-level intelligence. That you need to solve pre-training, you need to solve full T-modal, you need to solve reasoning. And at this point, the remaining challenge is to scale. But that is a big deal. Scaling is very hard. There's actually very few foundational ideas at all. Almost all the work is in actually being able to scale them to accept larger and larger amounts of compute.

• 48:34Speaker 01

  And that's a systems problem. It's a hardware problem. It's an optimization problem. It's a data problem. It's a pre-training problem. All the problems really are just about scaling. And so, yeah, I think in some ways at this point, it is predestined. And the work here is to scale it, but that is hard. It's a lot of work.

• 48:58Jacob Effron?

  Obviously, I think people talk about the societal impacts of these models scaling their capabilities. And I think we're still early in that discourse, and there's probably lots of different conversations to be had there. But any aspects of that that you are particularly interested in or passionate about are things you think we should be talking about more?

• 49:16Speaker 01

  Yeah. The thing that I think is most interesting is that we're moving from a world where intelligence is probably the critical scarcity in society to a world where intelligence will be ubiquitous and free. And so, what then is the scarce factor of production? And I think we don't know. My guess is agency. That you can just go do things. What are the right questions to ask? What are the right projects to pursue? I think these kinds of problems will be very hard for AI to solve for us. I think those are going to be really core things that humans need to figure out. And not every human is good at this.

• 50:01Speaker 01

  And so, you know, that's the thing that I think we all need to think about is, is how do we develop that kind of agency that allows us to work with AI?

• 50:09Speaker 02

  Do you think that's now or how far in the future?

• 50:12Speaker 01

  I think it's going to feel very continuous. It's an exponential curve. And the thing about exponential curves is that they're memoryless. It's always feels like you're, you know, you're always, you know, moving at the same speed, at the same pace.

• 50:24Jacob Effron?

  The model's not like eventually also figure out, I mean, if you think about like the figuring out what to do or the project goal, which you've kind of referenced a few times, like, you know, you could imagine at the most fundamental level in the future, we're like to a model, hey, build a, build a good company or like, you know, create an interesting work of art or, you know, create a movie or whatever, like, you know, does that agency, I guess, maybe talk a little bit about that as these models get more powerful?

• 50:46Speaker 01

  Yeah, I mean, I think the, you know, can you just ask the, the AI to figure it all out? Well, I think you can, and you get something out the other end. But let's use Sora as an example here. So, you know, if you're creating a video and you give it a very vague prompt, it will totally create a video for you. Maybe it will be a very cool video. Maybe it would be better than the coolest video that you could have come up with, but it might not be the video you wanted. And so, you know, so, so you can also interact with it where you give it a very detailed prompt and you say, I'm making these specific choices about the video that I want to see. And, you know, that I'm able to make it the video that I wanted, you know, that satisfies,

• 51:30Speaker 01

  you know, me or perhaps whoever my audience is. And I think that kind of tension will remain, no matter how advanced the AI is, because, you know, how you color in the blank space will determine a lot about what the final product is.

• 51:46Jacob Effron?

  How do you kind of use the, the most advanced O1 model today?

• 51:50Speaker 01

  My go-to for, you know, understanding models for playing around with it is that I spend a lot of time with my eight-year-old son, teaching him how to code. And he loves to ask questions. And so I'm always trying to figure out how to connect the thing that he is interested in today with the lesson that I want to teach him. And so as an example, you know, the other day he said, you know, Dad, what is a web scraper? How does that even work? And, you know, that gave me the opportunity to say, okay, well, can I teach him how networking works, you know, with a short program that scrapes the web? And so I turned to O1 and, you know, played around with it and had it try to figure out

• 52:34Speaker 01

  how to create, you know, a program that was short enough that didn't introduce too many new concepts that I hadn't already taught him. But did teach him about networking, which was the core concept that I wanted to teach him. And, you know, that was accessible for, you know, an eight-year-old. And that, I think, is just a really... and it was able to do that and it took some playing around with it. But partly is, you know, playing around with it is an important part of testing it.

• 53:01Speaker 02

  I guess on the testing side, when you think from the research testing perspective, what are sort of the core evals that you tend to have when a new model comes out? And what do you sort of rely on from that perspective the most?

• 53:12Speaker 01

  Yeah, well, I mean, you know, the first point to make here is that it changes with every generation. You know, as we were developing O1, the right metric to look at was GPQA, so Google-Proof Question Answering. And, you know, by the time we were ready to release, it's not a very interesting metric anymore because we'd gone from basically, you know, when we started, you could almost do nothing, to, you know, it being completely saturated. And the last few remaining questions were maybe ill-posed or, you know, not very fun to answer. And so, you know, the metric like very much depends on the work you're trying to do in the research. And I think this is a very commonplace.

• 53:54Speaker 01

  The thing that has been consistently useful, though, for the last few years has been programming. Because programming, it's a structured task. It's something many people understand. It's something I and other researchers understand, which is very important. And, you know, it scales from finish this line of, finish completing this line of code to write an entire website. And, you know, and, and, and we are not yet at a point where programming is a solved problem by any means. And I think we have a long way to go. I think there's several orders of magnitude left before, you know, do the work of a real software engineer is actually solved.

• 54:32Jacob Effron?

  I mean, you know, one thing from your earlier career is obviously, you know, you were getting a PhD in computer science and I think had at least a partial focus on game theory. And obviously, I think there's lots of interesting implications for, you know, using these models and, you know, to explore subjects in game theory. I guess just generally, like, how do you think AI will change social sciences research going forward, you know, policymaking, some of these areas? And I guess if you were revisiting some of that work today with the, with the power of some of these models, anything you'd be trying?

• 54:59Speaker 01

  Yeah, well, you know, as, as a first point, I'm actually pretty down on academia. I think it has a terrible set of incentives. And, you know, in some ways, you know, I designed the organization at OpenAI as like the mirror image of, of, you know, academia. So it's a place where you can collaborate well. But a lot of the, one thing that I think is interesting is that in business, a lot of what product management is, is actually like an experimental social science. You have an idea, you want to test it out on humans, you want to see how it works, you want to use good methods while you do that. And, you know, something like A/B testing, it's actually, you know, you're really doing a form of social science when you do that.

• 55:38Speaker 01

  And, you know, this is one of the things that I'm really excited about is, you know, if you're doing A/B testing, why not take, you know, all the interactions you have with your users right now, fine tune a model on that. And suddenly you have a fake user that reacts like in distribution to how your users do it. And now you can do an A/B test without going to production. Or, and, and maybe after that, you can have a deep interview with one of them where they ask what they thought. Does that work today? I don't know, I haven't tried it, but it might work tomorrow. And, and just, you know, this, this I think is a good general principle is anywhere you

• 56:20Speaker 01

  find yourself wanting to ask a human to do something for you, can you ask an AI to do that instead? And, and, and can it do, you know, a hundred of that, where maybe you could only possibly have made a human do it like once painfully?

• 56:34Speaker 02

  Yeah, I asked Jacob to do a lot of tasks for me, so.

• 56:37Speaker 01

  Yeah, you should stop.

• 56:38Speaker 02

  You should start asking my model.

• 56:41Jacob Effron?

  Thank you for shipping that, because you're saving me a lot of time.

• 56:45Jacob Effron?

  You mentioned, I guess, designing the incentives that exist in academia and designing the OpenAI organization in contrast to that. Like, say a little bit more about that.

• 56:51Speaker 01

  Yeah, yeah, yeah. So, I mean, again, like, think back 2017, 2018, 2019. You know, AI research labs were not a big business. They were research labs. And a lot of the people who came from them came from academia. And if you look at how academia is structured, there's a set of incentives that I think, I think, work okay for how they were originally designed. But, you know, there's a, there's an extreme focus on credit. Who actually did this? And in which order are people's names listed on the paper? That's, you know, a very important thing for people with an academic background. Maybe you don't want to collaborate with people because it dilutes your contribution to the result.

• 57:37Speaker 01

  If there's two people working on a problem that feels like competition, rather than an opportunity to get the job done twice as fast. And so, you know, I think DeepMind, you know, in many ways, I think, I think DeepMind thought, well, let's, let's build a lab that is like academia, but inside a company, so we don't have to, you know, so we can all, so I can tell people what to do, and where it's all about deep learning. And I think Brain originally said, well, let's, let's bring a bunch of academics and have them do exploratory research, very academic style. I will not tell them what to do, but I'll station product managers on the outside and maybe catch these great ideas and they'll, they'll become products.

• 58:22Speaker 01

  And, you know, we said, you know, we, we were a bunch of startup people, more or less, along with a few very great researchers, people like Ilya. And we said, well, clearly, the natural modality for a research lab is that it should work like a startup. So we will have an opinion on the right place to go. We'll try to give people a lot of freedom, particularly, you know, these great, these, these researchers, great researchers, some of whom we didn't know were great researchers at the time, but give them the freedom to go, you know, find that hill that they're willing to die on to, to produce the amazing thing they want to produce. But we'll make it very collaborative and we will make sure that people are working together. And ultimately that we want to build one thing, not, you know, publish a lot of papers.

• 59:04Jacob Effron?

  I love that. I guess you, you kind of ticked through earlier some of the, you know, the kind of most famous decisions in OpenAI's history, the switch from nonprofit, the Microsoft partnership, you know, API release of ChatGPT. Anything that maybe sticks out as, as like a key decision point that, that isn't maybe, isn't as, as famous or something that you think was either a really hard decision to make or something that really tipped kind of the, the direction the company, you know, the organization went in?

• 59:26Speaker 01

  I think one decision that I didn't talk about earlier, but which was also quite controversial at the time, was the decision to double down on language modeling and make that really the central focus for OpenAI. And, you know, this was complicated for, for all reasons, you know, a change like this involves a reorg and a restructure and, you know, people have to change their jobs and, you know. Again, we had started with this culture that we were going to try a lot of different things and see what worked. You know, our first major significant effort was, you know, a, a joint effort to play the game Dota 2.

• 1:00:11Speaker 01

  And which, you know, followed in the great AI tradition of solving harder and harder games. You go from chess to go to Dota 2 and StarCraft, which somehow feels less cool, but like actually, you know, I can tell you like with the math, these games are genuinely harder than Go and chess. Even if they're, they're not as classy. And, you know, and, and that was a big success and it taught us a lot. Because out of that experience came the conviction that you can solve problems by increasing scale.

• 1:00:44Speaker 01

  And you know, and a set of technological tools for doing that. And so, you know, by deciding to shut down the sort of more exploratory

• 1:00:55Speaker 01

  projects like the robotics team, you know, games teams and really refocus around language models and generative modeling in general, so including, you know, the multimodal work. I think that was, you know, obviously a very critical choice, but one that was was very painful at the time.

• 1:01:14Jacob Effron?

  One thing I'm struck by earlier, you obviously mentioned, you know, testing these models out with your eight-year-old kid. And, you know, I imagine in the time that you've been parenting, obviously, the world eight years ago looked quite different than than it does now, you know, in large part due to the advances that you've, you know, helped drive it up AI. And I'm wondering, you know, either for your life, for the way you parent, like, have you changed anything based on kind of updating your beliefs on like, how soon the power of these models will manifest in the world?

• 1:01:41Speaker 01

  Yeah, I think the truth is I haven't. And I think this is probably a failure on my part. Right? Like, you know, who is better placed to figure out what the kids should be learning than, you know, than me? And yet I think I'm like, pretty much trying to teach them the same things that I would have tried to teach them eight years ago. Why am I teaching my eight-year-old son to code when ChatGPT can code for him? Right? I think it's a mystery. But, you know, the the future, you know, in some sense, the future is predestined, but the the actual contours of how that works are, are, I think, going to be pretty mysterious, and they're going to be revealed to us over time. And

• 1:02:21Speaker 01

  so I think the the sort of age-old truth of of try things that are just at the boundaries of your capabilities, you know, work on the math, work on the coding, write, learn to write well, learn to read a lot and widely. I think. You know, those are going to develop the skills in kids and frankly, adults that they're going to need, no matter what AI turns out to do. Because fundamentally, it's not about the coding. It's not about the math. It's about you learning to think in a structured way about problems.

• 1:02:52Speaker 02

  Well, this has all been awesome. I'm sure we could chat with you for hours more. But we like to finish things with some rapid fire questions. And so the first one is what's overhyped and underhyped in AI today?

• 1:03:05Speaker 01

  Yeah, wow, okay. Well, one easy answer for what's overhyped is I would say new architectures. You know, there's a lot of things out there. They're pretty fun to look at. They tend to fall apart at scale. And so if there's one that doesn't fall apart at scale, that will not be overhyped. Until then, they're overhyped. Underhyped, I'm going to go with O1. I think it is very hyped. But is it appropriately hyped? No, I think it's, I think it's underhyped.

• 1:03:34Speaker 02

  I know our listeners will all be curious to ask, but anything you can share around why you ended up leaving OpenAI at this time?

• 1:03:38Speaker 01

  Well, the, the, the, the truth is that I had been there for eight years. And I really felt that I had accomplished a lot of what I had set out to accomplish when I showed up. And it's not a coincidence that I announced my resignation after O1 preview had shipped. You know, there was a particular program, research program that we had developed. You know, again, pre-training, multimodal reasoning. And those pieces were solved. And, you know, frankly, it's a hard job. And, you know, when I felt like I had done what I needed to do,

• 1:04:23Speaker 01

  it was a good time to hand over to a next generation of people who, you know, were excited about the job and excited about solving the problems that remain. And I think there's, they are super exciting problems.

• 1:04:37Speaker 02

  Any idea what's next for you?

• 1:04:38Speaker 01

  When I left Palantir, I spent two years before I landed at OpenAI. And, you know, I, I, you know, began the process of starting a robotics company. I tried a lot of things. I got my hands dirty, you know, actually building things. I talked to a lot of people. You know, frankly, I made a lot of mistakes, but, you know, none that I, you know, none, none that really mattered. And in the process of doing that, I learned a lot and I developed my own theses about what was important in the world and what was important about technological progress. And, you know, all those things, the people I met, the ideas that I came up with, you know, helped land me at OpenAI.

• 1:05:23Speaker 01

  And that turned out to be, you know, a much, much better experience than anything I could have picked the first six months after leaving Palantir. And so I'm not in any hurry. I'm gonna, you know, continue meeting people, figuring things out. Just, I'm really enjoying the process of thinking and learning new things.

• 1:05:46Jacob Effron?

  Now that you have kind of more time, like any threads you're like excited to pull on or things that you always wanted to spend more time on, but we're pretty busy in your day-to-day job?

• 1:05:53Speaker 01

  Well, you know, the funny thing is, I feel like I've been, you know, stuck in a box for eight years. It's a very cool box. Yeah, very cool box to have been stuck in. But there's a lot of things that, you know, have been happening outside. And, you know, I've been, like I said, I've been talking to robotics founders and seeing a lot of cool things that have been happening in the time period when OpenAI wasn't working on robotics. And just generally connecting with, you know, founders, researchers, people doing interesting things has been just really fun and engaging.

• 1:06:27Jacob Effron?

  Well, this has been an absolutely fascinating conversation and I know it's for Jordan and I and our listeners. Yeah, thank you for coming on and sharing all this. I want to leave the last word to you. I guess, any, you know, where can folks go to learn more about you? Anything you want to leave our listeners with? Any threads you're excited to explore that you want to put out a call for or whatever, whatever the floor is yours?

• 1:06:46Speaker 01

  Yeah, well, if you want to, you know, follow what I'm thinking as I develop it, the best place is to follow me on Twitter at BobMagruAI. And I think the right last words here are just that progress in AI is going to continue and it's going to be very exciting. And it's not going to slow down, but it's going to change. And that's fun. So I encourage you all to keep working on it.

• 1:07:08Jacob Effron?

  Well, Bob, thanks so much. Seriously, this is fascinating.

• 1:07:11Speaker 01

  Anytime.