Ep 84: OpenAI’s Chief Scientist on Continual Learning Hype, RL Beyond Code, & Future Alignment Directions

Jakub Pachocki, OpenAI's Chief Scientist, sits down with Jacob to cover the full arc of where AI research stands today and where it's headed. The conversation spans the explosive growth of coding agents and what it signals about near-term AI capability, the use of math and physics benchmarks as proxies for general intelligence, how reinforcement learning is being extended beyond easily-verified domains toward longer-horizon tasks, and what it means to run a research organization at the precise moment the models themselves are starting to accelerate the research. Jakub shares a candid take on the competitive landscape, why chain-of-thought monitoring is one of the most promising tools in the alignment toolkit, and — with unusual directness — why the concentration of power enabled by highly automated AI organizations is a societal problem that doesn't yet have an obvious solution.

• 0:00Speaker 00

  I definitely agree that continual learning is really the thing. It's really the thing that we're building. But I don't really think this is like a problem that's ignored and off the path of what we're doing currently. I think it is what we're working towards.

• 0:09Speaker 01

  What are like the other research areas within alignment that you're paying attention to or that you think are promising?

• 0:13Speaker 00

  A lot of the like longer term challenge with alignment is about generalization. What are the values that the model falls back on?

• 0:20Speaker 01

  What are the things that you need to figure out to be able to really make models work well in some of these other spaces?

• 0:25Speaker 00

  I come back to this a lot.

• 0:26Speaker 01

  Akko Payoki is the chief scientist of OpenAI. I think literally one of the most important people on the planet. And today on unsupervised learning, I got to ask him literally everything that I've been thinking about. And I know a bunch of people in the ecosystem have too. We talked a lot about model progress, what's required to make long running agents work, as well as the really interesting work OpenAI has done in the AI for science world and the progress he sees in that over the next years. We talked a lot about how companies should be thinking about model building in this moment, when they should be doing reinforcement learning, how they should be thinking about the evolution of harnesses and the impact that will have. We hit on a lot of his really interesting research, including the work he's done around alignment, the work that OpenAI broadly has done around math competitions. And we also talked about this focusing moment at OpenAI

• 1:09Speaker 01

  and what it means for the research organization and how he runs his team. Literally just such an awesome opportunity to talk to someone who is driving so much of the change that has revolutionized this space in the world. I hope folks enjoy this wide ranging conversation as much as I did.

• 1:26Speaker 01

  I feel like you are the perfect person to talk to you about all the questions everyone has in the ecosystem. What's happening with model progress? A lot of companies are thinking about how they should be building things based on what's happening with the models. A lot of people at a societal level are thinking about the impact AI is going to have on science and broader society. And you've been at the forefront of the space for pretty much every generation of improvement these past years. And so really excited to have you on the podcast.

• 1:52Speaker 00

  Happy to be here.

• 1:53Speaker 01

  I think I'll start with one of the juiciest things you said, which is, you know, four months ago, I think you and the OpenAI team talked about aiming for a system with research level interim capabilities by September of this year. So coming up, I think that's what six months from now. And then a more fully automated AI researcher by March 2028. And so I guess, you know, checking in four months later, how are you feeling about those timelines?

• 2:16Speaker 00

  Yeah, I think, you know, over, I think over, over the last months, I think like the change that really happened is we've seen this explosive growth of coding tools.

• 2:27Speaker 01

  It's an understatement.

• 2:28Speaker 00

  Yeah, we definitely like really kind of gone to a place in OpenAI where we use Codex for the for the majority of actual coding. And so I think I think for most people like the kind of the act of programming has has has changed quite a bit. So I definitely see this as a signal that like, you know, something here is on track. The other kind of like very interesting update over the last few months to me has been the progress on the math research capabilities, also the results we've kind of seen in physics in other fields. I think I think this kind of level of capabilities, this level of like ability to provide insight

• 3:09Speaker 00

  when combined with ability to access infrastructure, ability to use maybe more computer test time than something that Codex isn't currently, and very strong improvement in general well intelligence, which I also expect over over the next couple of months. Yeah, it's something we're still very much planning for and very focused on.

• 3:29Speaker 01

  And how do you like know when you've you've gotten there? Like, what's like a workflow you might look to to say, hey, OK, I think we've got these, you know, research intern level capabilities?

• 3:37Speaker 00

  The the way I would distinguish, you know, a research intern from from full automated researcher is the kind of span of time that that we would have it work mostly autonomously or the kind of like specificity of the task that has to be given. So I don't expect, you know, we'll have systems where you kind of just tell them, oh, like, you know, go improve, you know, model capability, go solve alignment.

• 4:01Speaker 01

  That'd be nice.

• 4:03Speaker 00

  And they will do it, not this year, you know, I think we might get there at some point. But I think for like more specific technical ideas, like I have this particular idea, how to improve the models, how to like, you know, run this evaluation differently. I think I think we have the pieces that we mostly just need to put together.

• 4:19Speaker 01

  Andre Karpathy released, you know, a pretty viral version of of using some of these models to, you know, improve some of his, you know, obviously way less complex models than what you guys are building here. But did that feel like generally in this, you know, in the spirit of of some of what these tools might look like?

• 4:35Speaker 00

  Yeah, I think it's in this period. Yeah, I mean, I I I expect it to look like a pretty Continual evolution from kind of where Colleagues is now, I think, towards a bit more autonomy running for a longer time.

• 4:51Speaker 00

  But yeah, I think we'll see a lot of this sort of application. I think in general, we'll see like more autonomous and higher compute use of these models for different things.

• 4:59Speaker 01

  You mentioned kind of like the math and physics side, and obviously you've had these really impressive breakthroughs in math on, you know, some interesting like different kinds of competition, you know, problems. Maybe, you know, I think for our listeners, it like intuitively makes sense how progress in coding directly translates to something like, you know, helping with AI research. How does like math and physics progress like also tie into this?

• 5:21Speaker 00

  The biggest draw that like, you know, focusing on this math benchmarks has played for us as a general, yeah, like benchmark and north star for like how to improve this ecology, right? Math is very measurable, right? It's much easier to tell whether you've actually solved a math problem than whether you've even like produced a good, you know, piece of software. And also, it can get very hard, right? So you can have things where like it's very definite whether you solve them, but it can be like arbitrarily pretty much hard to actually solve them. You know, I would say like up until not too long ago, like, you know, my perspective has been like, well, okay, like we, you know, our models are not, you know, maybe able to solve like simple math problems. Okay, our models are able to solve simple math problems, but are not able to solve like IMO level problems.

• 6:03Speaker 00

  So clearly, there is just like a gap in just like this, you know, intelligence of these models of like that is very measurable, very, you know, very easy to run at. It's very clear what we need to do. And, you know, and this has been kind of our north star for like reasoning models and so far. Now, of course, that is changing quite a bit, right? And we are, you know, we have kind of reached these milestones that we've been working towards of like, yeah, IMO goals level, solving IMO problem six, and, you know, and making far as into research level mathematics. And you know, from this one, I think, I think there still is, you know, there's a definite still is utility, like continuing to measure progress on this.

• 6:45Speaker 00

  I think there's also like, you know, there's definitely like transfer that that you can get from like getting better at mathematical reasoning to getting better at AI research. You know, a lot of our best researchers are, you know, mathematicians we're training or from other kind of theoretical fields. But definitely, we are, you know, we are very much changing how we think about, you know, these now stars, and we are very focused on how the models, the next models that we're producing are actually useful in the real world, you know, useful, you know, especially for AI research, but also for other kind of economically valuable activities and for other fields of science, and especially maybe more applied sciences.

• 7:28Speaker 00

  And the reason for this shift is because we believe the models are now capable enough, not as smart as people and always, but capable enough to actually materially change the economy, change how things are done. And so, yeah, we feel a lot of urgency about that.

• 7:43Speaker 01

  In the early days, picking a domain like math that is so hard to solve, but then easily to verify whether you did it, like it's kind of the perfect place to get started. And I think code obviously shares a lot of attributes to that, you know, possible to check and verify and great for reinforcement learning. I think one question that a lot of people are thinking about is, okay, we've seen reinforcement learning work incredibly well in these domains where you can verify it rather easily. A lot of, you know, valuable tasks in the world, medicine, law, finance, you know, there's some level of the ability to do that, but it's certainly not to the same extent that math and code are. And so I think a lot of people are trying to figure out, you know, are we going to see similar improvements, you know, obviously code and math, the rates of improvement have been so astronomical and shocking.

• 8:26Speaker 00

  Yeah, I definitely expect so. I think an interesting duality that we think about a lot is, you know, for this more general task, for these tasks that are kind of harder to evaluate, they share a lot of commonalities with. Just longer horizon tasks, right? Because if you think about even like a very well specified math or coding problem, again, like if it's something that you need to work on for like a year, then even if it's very clear what the criteria of success are in the long term, like what to do on your first day of working on it is a pretty open ended problem. Yeah, and so I kind of believe this difficulty is calling side and they're very clearly the next frontier for how these systems develop.

• 9:10Speaker 00

  And I think we've definitely seen very encouraging signs, both on just like our ability to scale our own this more general domains. And I think also like we can scale our methods that that that that's our promise.

• 9:24Speaker 01

  In these other domains, it feels like one of the hardest things to know is just what was success in a task, right? And you can imagine, you know, there's going to be, you know, whatever the problems you are that are facing in coding math that are short term tasks and then longer term tasks, feels like it will be amplified in the space that is, you know, outside of those, right? Where a short term legal task or medical task may be harder to run thousands of iterations on, right? And figure out, you know, was that done correctly? And then those longer term tasks, like even harder. I'm curious like how you even conceptualize that research challenge? Like what are the things that need to be, that you need to figure out to be able to to really make models work well in some of these other spaces?

• 10:02Speaker 00

  Yeah, I think I come back to this reality of like, how do we make the models work for a very long time? And how do we teach them to evaluate kind of partial progress?

• 10:12Speaker 01

  Yeah.

• 10:13Speaker 00

  I mean, I think if you look at like even outside of RL, like where that sort of progress on longer horizons is coming from, right? Like, I mean, as the models kind of become more consistent from just like pure supervision and pre-training, they they gain some idea of like, you know, oh, what what does like a good partial artifact here look like? And so I think I think even if we weren't like. Scaling RL very meaningfully, we would see an elongation of these horizons over time. Yeah, it's definitely, you know, a risk challenge to like, to figure out how to like leverage these new ideas from RL and so forth to apply this to general domains, but I'm quite optimistic about that.

• 10:52Speaker 01

  Yeah, and it sounds like part of your mental model is like the models themselves being able to check progress with some, at some sort of cadence that is, you know, reliable enough. On the outside at least, it's not totally clear if we've seen like generalization in RL yet. It feels like we, yeah, clearly you seem to have some techniques that really optimize models around whatever we choose to focus on, but it's like, almost feels like an older school version of, of ML of like one, one thing at a time. Is that like, you know, I guess, would you agree with that characterization and like, you know, how do you kind of see this, this current climate?

• 11:21Speaker 00

  Well, we are buying a lot of compute, right? Because we, we don't, I mean, we still believe the bitter lesson, we believe, you know, modern average some degree. Yeah, we've seen, you know, new techniques and I think new ways to scale, but like that, that is kind of the, the lens through which we've been doing things. Yeah, I think there is a certain amount of complexity that we need to grapple with and kind of underground needs to grapple with because, you know, we're no longer really like purely building like, um, you know, brain, the sky is completely isolated from the real world, right?

• 12:03Speaker 00

  Like if you actually, you know, if you want this model to do like medical research, if you want it to cure cancer at some point, it needs to like learn about the real world in some meaningful way, you know, maybe conduct some experiment and learn from its results. And for that, you, you, you need to figure out how to actually connect it, right? And that is going to involve something that is, yeah, that goes in the direction you described, but I, I don't think that goes counter to actually scaling the, like finding and scaling the simple algorithms that, that we've been developing.

• 12:32Speaker 01

  I feel like I talked to a lot of companies and like, one of the main questions that everyone seems to be asking these days is like, should we be doing, you know, our own reinforcement learning, like take an open source model and like, we have some data on a task that people do, we have evals because we know our domain pretty well, like, is this something that makes sense for us to do or like, should we just wait for the models to continue to get better at, at some of these things? You know, what advice would you guess, would you give for like the many builders that listen to the podcast as they think through, you know, the extent to which they invest on the, on the reinforcement learning side?

• 13:03Speaker 00

  Referring to learning definitely can be a very data efficient way to really improve the model at some sort of task, right? There is a much more data efficient way of learning that we know, right? We're just like learning in context, right? And this is maybe the most fundamental way that people, you know, teach these models. You just prompt them with like examples with with with instructions for what you want. I expect that learning is going to get much better over time. And so I think it definitely really matters that the models can adapt to your context. They can adapt adapt to kind of the the kind of tasks you care about. So I think that will be very important. I'm not sure if like, you know, replicating the kind of current RL pipelines is going to be like the right way to go about it. But yeah, it's definitely a problem that that we're thinking about.

• 13:45Speaker 01

  Yeah, so it's almost like, yeah, you still have to do the work, like you still should, you know, figure out what the evals are that matter, gather the data, the examples. But like, it may just turn out in the future, you're far better off just feeding that into this context than trying to like do anything on on, you know, your own model.

• 13:58Speaker 00

  Yeah, I think I think that's quite plausible.

• 14:01Speaker 01

  And I think that like, you know, obviously, people have seen the success of of tools like Codex, which I know, you know, you've obviously been a key part of and and wondered like, you know, hey, do we need to build like our own kind of, you know, should we build our own harnesses or our own ways of of using these things or, you know, for for our own domains, whether it's like, you know, legal or finance or or healthcare, or do we kind of just like take the harnesses that the large models do and and kind of use them within, you know, with with the context that we have? Any any thoughts around like that?

• 14:32Speaker 00

  Like the implementation of the harness shouldn't really be a limitation for a very long time. I think we'll be able to get like much more general harnesses that people can use for for all sorts of other domains. I mean, I think Codex is pretty good, actually, if you try using it for things beyond coding.

• 14:46Speaker 01

  That's so interesting, like a much more general harness being something that's almost like adaptive to or like just works across whatever the, you know, specific set of tools you have in your domain or specific set of things you want to expose to the model.

• 14:59Speaker 00

  Yeah, I mean, I think, and you know, I think it's also worth thinking about like, you know, why, like, you know, what, what, what is kind of the kind of ultimate interface that we want to interact with the model with. So the model gives some, the models give some new affordances, right? They can build their own UIs. They can kind of do things that, you know, people would find very time consuming. But yeah, I definitely think there is also just like a lot of space to kind of enable the models to access like their current interfaces that we use for, for people, right? So I think like, we want to have, you know, AIs on Slack, for example, or that are kind of plugged into our, our context and, and yeah, and are able to, to learn from it and able

• 15:41Speaker 00

  to kind of, yeah, to utilize these existing things, right? So definitely like, there is some meat in the middle here, but definitely I believe like long-term, like, you know, like by default, the AI should kind of meet you where you are. And if not, that would be because it kind of, it has new abilities, not because it has limitations.

• 16:01Speaker 01

  Yeah, that's an interesting point that basically today it feels like these harnesses are so bespoke to certain environments, but like, over time, as you add more and more skills and tools and models can navigate across those effectively, it's like, there'd just be a general, like, you know, the way humans have, that, that makes a tremendous amount of sense. I guess I'm curious, like, you know, you, obviously I'm sure like every day you see kind of crazy stuff on the research side. At this point, like, what are the milestones that are like, still meaningful to you as you think about like, it would be pretty crazy if I, you know, did a run one day and saw like, X or Y, like, what are the things you're paying most attention to?

• 16:38Speaker 00

  Yeah. I mean, at this point, it really is about research, right? Like, is it about, it is about can the model discover new things, can it execute on like a longer? Horizon research problem.

• 16:54Speaker 01

  It's almost like looking for some sort of insight that you're like, oh, if someone on my team had come up with that, that would, I've been pretty intrigued by it.

• 17:00Speaker 00

  Yeah, we've actually had like some minor, I think, I think quite impactful ideas come from even like GPT five point two pro that that was entirely, but you know, I think it's still very, very small compared to where I expected today.

• 17:15Speaker 01

  Yeah, I mean, it seems like almost inevitably like these models are going to get better. They will be used in research, they'll be used in science more generally. You're like one of the first people interacting directly with these models as like research partners almost at this stage. Anything like you've learned around the right way to do that, or do you think about like what a research organization, you know, as these models continue to get better, might look like?

• 17:35Speaker 00

  Yeah, I think we're definitely kind of at a transition point where kind of the short term immediate quality of the model is about to be a quite determining factor for the pace of our research progress because the models are going to drive a lot of that. And so that definitely requires, you know, rewiring some intuitions about how to run a research organization. You know, normally you kind of try to not be too focused on like immediate quality. You try to be much more focused on like the longer term. I think we have like a lot of very exciting staff queued up that we are kind of working towards, but I feel a lot of urgency to kind of, yes, to actually execute on it

• 18:18Speaker 00

  and to actually use this advances in model intelligence to accelerate research on the AI and especially AI alignment.

• 18:26Speaker 01

  Yeah, it's such a fascinating point because I've heard you talk before about running a research organization. And I feel like in the past, it was like giving people the space to, you know, pursue a lot of things that weren't like directly, you know, hey, this is for a month or two months of progress. But it's like, what are the ideas that are really going to drive things forward? But it makes total sense that we're in a time now where you're like, look, everything we do will be so much better if we just focus on this in the, in the short term and make it better. It must be like fascinating to navigate that and like these maybe further off research ideas at the same time and like running an organization.

• 18:56Speaker 00

  Yeah, yeah, it's definitely, yeah, it's definitely something we spend a lot of time on with Mark nowadays, yeah.

• 19:01Speaker 01

  Right now you have, you know, a ton of compute as a company, but you obviously, you have great scaling laws on the pre-training side. You have great scaling on the RL side. You have probably lots of experiments going on that have nothing to do with either of those vectors, but are like interesting new ways. How do you even think about like allocating compute across all of this stuff?

• 19:20Speaker 00

  Yeah, it's going to be very complicated, right? Because there are so many things that we need to do. One thing within, one kind of discipline we've started keeping is we, we try to make sure we just explicitly budgeted like a large chunk of our compute to the most scalable methods of the things that we believe are the most responsible for driving general model intelligence. And even if it's not the most efficient allocation of compute at all times, because if you're allocating so much compute to like one experiment or like one set of experiments, there are so many things you could accelerate for a little bit of that compute elsewhere. But, you know, but I think it's easy to kind of, like with all the, all the interesting and important things that we're doing, I think it would be very easy to kind of portion all of it and like not, not really end up doing the things that we believe are most important.

• 20:03Speaker 00

  You definitely want to like understand the kind of empirical evidence, you definitely want to make sure your evaluations are in order and the kind of experimental rigor is there. And then you also want to apply some regulation based on like, okay, do we understand this method? Do we actually expect it will scale? Do we expect this is something you can actually build on in the future? Is this kind of a one off, right? And I think, and based on that, determine the price.

• 20:23Speaker 01

  Yeah. It's interesting. You probably find all the ways that you like know you could improve things, but they feel maybe like off, off a little bit to the side of where you think the overall arc of progress is. And so you end up leaving some of these like low hanging fruits to some extent, because really the most important thing is finding the future direction and then the scaling within that and devoting compute toward that. Obviously, the, the place where we talked about Codex a lot and the success of coding and it feels like, you know, last year was like the year of just incredible hill climbing on, on coding. I'm curious, you know, obviously Codex has been a super successful product in many ways, like Anthropic was kind of first to this market, you know, Claude Code, you know, it was a dominant product there. What do you kind of like, you know, reflecting on that, I guess, like, what do you make of the success Anthropic's had in this space?

• 21:06Speaker 00

  Yeah, I think, I think it's a matter of, you know, really focusing your product direction or on where, where you believe the kind of the, the next application of the technology is, right? And, you know, if you look at the kind of prioritization we've had. On the on our product, I like, I mean, we have been right like working on on cutting price, but they have kind of been like a secondary thing, right? Compared to like our main priorities. And interesting thing is that is not very reflective of like the priorities of the research organization within OpenAI. I think, you know, given that like we've kind of had this, you know, explosive success of ChatGPT, you know, ChatGPT as it was, you know, I think I think ChatGPT is evolving quite a bit and it's going to

• 21:50Speaker 00

  evolve quite a bit. But as it was in 2023, right? This is particular, you know, product that's maybe not, you know, I think it's definitely quite aligned with our vision of like where AI is going. But but like, it's not really like the like representative of like everything that that that that it enables. And so the majority of like our work in research has been focused on like that that future thing. And I think increasingly it has decoupled from our our our kind of like short term product strategies, right? Yeah, I'm very kind of confident about the things we've been building and the things we are building on on the research on the model intelligence side. You know, a lot of our referitization and increased focus on the on the product side is about actually

• 22:33Speaker 00

  kind of getting to deploy them and the belief that actually they are the thing that really matters now.

• 22:38Speaker 01

  Yeah. And now it feels like, you know, the clearly the whole company priority, you know, is so locked in and focused on this. And you've seen just incredible improvement in Codex in recent months. For all the developers who listen to the podcast, like, again, it's almost like hard to comprehend like what the world looks like as these models keep climbing on longer and longer tasks. Like, what do you think will look different in their lives? Or like, how will they be using Codex in, you know, three, six months? I realize three months and six months are very different timelines in this world. But take take whichever whatever in between point you'd like.

• 23:08Speaker 00

  I would expect just a gradual increase in just the level of autonomy you feel comfortable funding the model, just the fakeness of description that's going to work with, you know, the level of supervision it needs. I think we're not very far from models that can work autonomously for a couple of days, maybe just quite a bit more computer than they're using now and produce much higher quality artifacts on their own.

• 23:32Speaker 01

  Do you have a gut instinct on like what like, you know, there's always been this question of like, will the world, you know, do you need that software engineering skill set to supervise these models running for a few days? Or like, hey, does it turn out at some point of like, being able to run for a while, you know, anybody can can use coding agents and supervise them to to some sort of output?

• 23:48Speaker 00

  I mean, I think definitely for like a lot of outputs, you already don't need much experience, right? I think I think still the distinction I would draw between like, you know, an intern here and like really an autonomous researcher or software engineer would be that like if you want to build something bigger, like, you know, you probably still want to apply supervision. You still kind of want to have like an overarching thing. You want to recognize like what what what building blocks fit in and what we've done. But yeah, I definitely expect that like that desired skill set to shift quite a bit over time.

• 24:18Speaker 01

  Yeah.

• 24:19Speaker 00

  Towards towards like more general vision setting.

• 24:23Speaker 01

  You know, I guess on the on the on the research side, I feel like there's been, you know, maybe maybe like a month ago, I feel like all anyone could talk about was continual learning. And there's just, you know, this is in the zeitgeist. There's all these Neo Labs starting to go focus on continual learning. Some folks left OpenAI to go focus on that. I'm curious, like, you know, I think maybe part behind that is a belief that like, you know, RL alone, you know, either won't get us there or will get us to like some level of very inefficient scaling. And it's kind of different than the way, you know, humans learn. I think even I've heard you say before like that, you know, RL is still very different today than the way that humans learn. What's your take on on like that, you know, that whole movement?

• 25:02Speaker 00

  Yeah, I am a little bit confused by it because, you know, in my mind, like, the whole kind of like excitement that like we've had. I mean, even if you look at the titles of like the GPT, you know, three paper, right? Like, it is that like, oh, you know, this class of models is actually capable of continual learning, right? It's capable of like learning, learning to learn in context, right? That has been really, you know, the driving force behind the kind of excitement to like scale these GPT models further. That has been like the premise for why we really need to teach them with RLs, like learn in context more efficiently. And so I definitely agree that continual learning is. Really the thing, right? Like it's really the thing that we're building.

• 25:46Speaker 00

  But I don't really think this is like a problem that's like, oh, you know, it's kind of ignored and off the path of what we're doing currently. I think it is what we're working towards.

• 25:54Speaker 01

  Yeah, like in your mind, this is like the single best path to get there is to continue to kind of scale the pre-training in RL.

• 26:00Speaker 00

  I think that is kind of how we've made the most progress on this problem so far. And, you know, I think there are, I think that there definitely are like more ideas, more steps. I think also a lot of improvement that will just come from scale.

• 26:12Speaker 01

  Yeah. And I guess like, you know, we have a lot of folks listening that maybe have, you know, have been able to do a lot of simpler things with these models. And then they try to do like some of these more complex, you know, I'll call it 100 step or a longer term tasks. And they're like, oh, you know, the, the models don't work for this yet. And I think it's harder, you on the inside constantly feel this improvement, but for them it feels like, hey, this is like night and day away from, you know, being able to do this much longer thing. How do you kind of articulate to them, I guess, the set of things that need to be true for these like much longer steps to happen? Is it around kind of checking in more often as you were talking about before? Or I feel like there's just this belief among the research community of like, oh, all of these tasks will be solved in the next year or two. And then in the wild, a lot of people may be not totally rocking that like improvement line that we've been seeing.

• 26:58Speaker 00

  Yeah, I mean, definitely a lot of that prediction comes from just looking at like historical improvement lines, right? And I think increasingly we can, we can roughly see the, the, the shape here. And I do think a lot of this about just the models becoming intelligent enough to recognize like whether, you know, they're making progress.

• 27:18Speaker 00

  I think some of this is like, yeah, this very kind of pragmatic work of like, are the models actually, you know, can they actually access, you know, all the context, all the files, all the infrastructure they need to do the work you want them to do? Which, yeah, I remember like in the past when we were discussing, you know, the kind of the roadmap that we're taking with RL. You know, I definitely feel like, okay, we just need to teach the model to kind of reason if it's on token as that's kind of the priority. And then of course we'll need it to use tools like the entire environment. You know, at some point, we definitely need to see, right? At some point, we need to use a physical body, right? But like, yeah, I mean, I think we're definitely like well into the stage where, you know, really needs to like interact with the environment and really needs to see. I think someday soon, we'll really come up with robots, but yeah.

• 28:02Speaker 01

  Yeah, I mean, it does feel like a lot of the times when I hear people complain about, oh, a model can't do X or Y. It's like literally just because you haven't fed, you know, or connected it to systems or fed enough context into it. It actually, I do wonder if like context was universally applicable and able to flow into these things, like I felt a lot of these problems would actually just be solved with today's models. You know, I want to talk about some of the AI for science stuff that you guys have been working on. And one thing in particular, you know, I feel like the coding stuff is something that everyone feels very viscerally, because, you know, in every company, they're using these tools and getting tons of productivity. You know, the math side, not all of us competed in IMO competitions and necessarily have as much of like an intuitive feel for some of these breakthroughs. And so one of them I know that was really interesting that you guys did is use some compelling work around like first proof, right?

• 28:44Speaker 01

  And I think these are like very different problems than kind of traditional competition math. I wonder if you could just speak a little bit to that, because I think it's just a space that our listeners might be less familiar with and kind of less familiar with understanding the implications of models being able to do pretty cool work here.

• 28:58Speaker 00

  Yeah, I mean, you know, I think, yeah, I was very excited with the first proof challenge and, you know, again, I kind of, did you ask what particular ones kind of a benchmark, right? It's like a couple, you know, respect some mathematicians, theoretical computer scientists who are seeing problems that like they believe are like representative of their day to day work, but haven't been published anywhere so that we can really have our models take a crack. We were so excited about this challenge, but, you know, it was kind of dropped without any, any, any, any advance warning with like a week long deadline to actually execute. We had a, we had a very exciting model training at the time.

• 29:34Speaker 00

  And so one of the people in charge of training, James Lee, kind of started prompting that model just.

• 29:46Speaker 00

  By hand and yeah, and actually I'm seeing, oh, okay, it's actually solving this problem. It's really fascinating things to see. You know, one of these problems actually is from a domain that I did my PhD in. And yeah, seeing the model kind of come up with these ideas, which I will be quite proud to come of like in a week or two, seeing it come up with them in like an hour or so. That was very, yeah, it's a very weird feeling, right? Like, yeah, I think like in the past, when I felt like that was like when watching our data bot like play just like very interesting data games infinitely, right? And feels like just there's some sort of magic happening because like, you know, interesting things should not be like indefinite.

• 30:31Speaker 00

  Yeah. And so seeing that happen for MAF, right? For something that I believe like, you know, is actually like quite representative of our precursor to a lot of the work that we're doing and a lot of the work that like really matters in the world. Yeah, definitely, really, it gives my feeling of urgency.

• 30:49Speaker 01

  One thing that's fascinating too is the idea that you're training these models and it's like, you know, you probably you throw these problems in and it's like, nobody knows whether, you know, how good will they be at solving them? And I think just like, it must just be fascinating to see something that you know so well and a space that you spend so much time in and realizing, hey, probably the previous generation of models wouldn't have been able to do that. And you wouldn't even thought necessarily that this was like the benchmark to do, but it's like just generally showing the general purpose capabilities and improvements of the models.

• 31:15Speaker 00

  I mean, it is at a stage where like, you know, we needed to like seek out experts in the in the particular domains to be to be able to tell us whether these particular proofs are correct or not. But, you know, it's still much easier to like tell whether you've actually made progress than, you know, than for something like even coding, right? Like because sure, the competitive programming, you can evaluate, but most programming is not competitive programming and it's, you know, it's about like, oh, are the abstractions right, are they handling all the cases?

• 31:40Speaker 01

  Yeah. I guess like, you know, I feel like there was this maybe common criticism a year ago, and I don't know if it's as strident now, that like, okay, these models are like pattern matchers, but like you really want AI for science, like we're not going to get new ideas or like, you know, entirely novel things out of out of pattern matching. Feels like we continue to like chip away at that narrative. Are we getting closer to kind of fundamentally disproving that?

• 32:01Speaker 00

  I believe so, yeah. I mean, I think kind of on schedule, we're starting to see like minor advancements, right? Like not huge things, right? Like a small idea here or there. I mean, maybe maybe some like bigger papers in collaboration with. With scientists, right? But, you know, was AlphaZero a pattern match or AlphaGo a pattern match? Or, you know, our, our Dada Bot a pattern match? Or like, they did kind of come up with new strategies for their respective games.

• 32:29Speaker 01

  Yeah, it's funny that there's counterexamples to it all the way back to, you know, 2016, 2017.

• 32:33Speaker 00

  Right, right. And, and, you know, and you can say like, well, I guess you can always pull to flaws in that, which I think is interesting. Like AlphaGo can be beaten with some strategy. Our Dada Bots could have been beaten with some, with some strategy. I think, I think there will be a lot of differences for a while of, of like these models, right? But, but I think also like they, they are able to discover new things because they have a lot of these capabilities. And like the way, you know, yeah, I mean, it's, you know, taken a couple of years to like go from like this, like very tiny game environments to like this much more general scientific research that requires kind of going through, you know, like a decent approximation of like all human knowledge in the meantime. And, you

• 33:17Speaker 00

  know, learning all the human languages and so forth. But, but, but I think the basic principle is, is very similar.

• 33:22Speaker 01

  Yeah. It was funny. I think like when you guys had these first proof results, I remember like the organizer said, you know, they were commenting on these AI solutions and they were like, this feels like, you know, 19th century mathematics of like brute force, you know, computation heavy approaches rather than these like elegant modern techniques, which I'm not sure is a feature or a bug of, of, you know, obviously the, the way these models work. But like, you know, hearing that, I mean, does that like, does that concern you, excite you?

• 33:47Speaker 00

  It doesn't concern me. I mean, I think it's expected that like, I mean, I'm sure I, I, I fought for at least one of the problems. Like actually, actually our roles produced a pretty, pretty nice, pretty, that was quite a bit shorter than like the intended one. You know, but I think in general, you would expect like, yeah, this model is kind of, you know, they can produce so much more reasoning in a short time than like a person can, right? Just like in terms of this raw number of like tokens or funds. I, I don't expect that to be like kind of a long-term feature.

• 34:12Speaker 01

  It feels like there's so much momentum behind AI for science right now. And you mentioned, obviously, like, you know, at some point you do have to connect these models to the physical world. And you guys released some cool stuff with Ginkgo and like some of these other things you've been experimenting with. I'm sure you've thought a lot about like AI for a bunch of different areas of science. You know, as you've kind of dug into some of this stuff, have you dealt with any intuition for, as you think about like three years from now, the spaces where of science where you're like, oh, that, there's going to be crazy progress there versus the ones that might prove like a little more resistant to immediate change?

• 34:43Speaker 00

  You know, a tempting answer would be that like, oh, you know, it's really about like, you know, do you, you know, what are the things that kind of require some, some, you know, manual work, like where the models are not quite plugged in. The ecosystem or, you know, like the different laboratories will also kind of evolve pretty quickly to adopt to like these new technologies.

• 35:04Speaker 01

  Within those STEM fields, obviously, you know, I feel like there's a question of, is it like an LLM with access to the physical world? Or you've obviously had companies that are, have been started specifically around these domains, right? Like an isomorphic in biology, or periodic in material sciences, or physical intelligence and robotics. What's your kind of gut instinct on the extent to which it makes sense to pursue some of these things, like independent with different model architectures versus like all within the context of one place?

• 35:30Speaker 00

  Yeah, I think it's kind of similar to, you know, my answer with like the UI for, you know, for Codex, which like I would build around the capabilities of a technology and not around its limitations so much. So, you know, you definitely like, if you have something that can sadly design like a huge amount of like interesting like chemical or biological experiments, like yeah, I mean, it makes sense to, you know, build labs that enable that. And I think if we, if we did get to a place where like the model is like very capable of designing high quality experiments, it would also make sense to like have it work with humans in a loop, right? Like we shouldn't think of it as like, oh, it's either kind of automated fully and you have this like fun thing, using some tools on the side. Like we will get to a world where like it's just very natural to be collaborating with,

• 36:14Speaker 00

  you know, AI scientists that are working hard on a problem.

• 36:16Speaker 01

  Yeah. It's so interesting. It's almost like a different vision. It's like one world where this works is like, hey, you just train a model, you know, to basically run these end-to-end tasks and like be the automated, like, you know, biologist or, you know, chemist or whatever it is. And there's another one which is like, well, you're building really tools to, you know, both propose, run, kind of work in tandem with a bunch of human researchers.

• 36:36Speaker 00

  Yeah, I mean, you know, I wouldn't necessarily characterize it this. I mean, of course there are tools and sound sense, but I think like, you know, we'll get to a point where they're driving a lot of the like design and ideation for the whole process.

• 36:46Speaker 01

  Yeah, with like an LLM architecture, but just like, you know, being able to figure out the right, the right kinds of experiments to run and then actually design it.

• 36:53Speaker 00

  And yeah, when it comes to like different architectures and, you know, I mean, you know, for sure, like, you know, like natural language. Reasoning, like the kind of the kind of things that we're prioritizing that gives you a lot of generality. Like there, there are things that are that, you know, you kind of want to train, you want to train a different model to model, right? You know, I think even like, yeah, if you wanna create a very good, you know, goal model, I don't think like large language models are like the most efficient way to go about this, although they might result in the best model eventually. But, uh, you know, I think it's similar for like, uh, you know, protein folding or other tasks of this kind.

• 37:29Speaker 01

  Yeah. So you think it makes sense to have like some independent efforts around that. But obviously the like, you know, that will end up being paired with like a core, really good researcher, large language model that is, you know, helping drive a bunch of the stuff.

• 37:39Speaker 00

  Yeah.

• 37:40Speaker 01

  I want to also make sure just to talk about AI safety, because I think that's an area that you've done a lot of really pioneering work on. And, you know, I'm not sure all our listeners will be familiar with, uh, you actually did some really interesting work across the labs, right? Uh, and we're focused on, you know, chain of thought monitoring. And so maybe to start, just tell us a little bit about that work and, you know, uh, you know, what you found.

• 38:00Speaker 00

  Yeah. So this is a realization that actually we had, um, around the time we actually saw like the first, um, reasoning models of kind of the current crop. We realized that like, okay, like, well, this works, right? And we were pretty, uh, you know, we were thinking a lot about what this means. We kind of were like, okay, like probably there were really changes over the next annual year or two or three. You know, we were thinking what this means for, for safety and for our ability to kind of understand what these models are doing. And we realized that because of the way we train these models, and because we don't supervise the reasoning process directly, right? It's not like, you know, ChatGPT is trained to kind of, um, you know, be polite and nice and like, uh,

• 38:43Speaker 01

  And it always tells me I have great ideas.

• 38:45Speaker 00

  Yeah. Well, you know, that's a separate issue, right? Like, but, but, you know, but, but like, even assuming it's like aligned exactly in the way we'd want it to, which is definitely not, you know, uh, sycophantic. Like, it's still kind of not going to be, uh, you know, there are just still, still some things it's not going to reveal about its motivations and time, because, you know, maybe it would be unsafe, or maybe it would be unkind. Um, or, you know, or maybe because it's not, maybe it's actually not aligned in the way we think, but it wants to hide that, right?

• 39:18Speaker 00

  And the way we train the reasoning models, like the chain of thought doesn't have any of that. It's not optimized to be in any particular way because it's just not directly great. It's only great in how it relates to like producing a high quality output. And realizing this is actually a very powerful paradigm for being able to interpret what the model is doing. Right. It's actually not a very different idea from mechanistic interpretability, right? Because in mechanistic interpretability, again, like you kind of have this model. You have these activations of the model that are not directly supervised to predict any label.

• 39:56Speaker 00

  They're kind of indirectly supervised, but the model has never been trained with any sort of inspection of this activation. And so these activations might reveal something about this inner workings. But the big advantage of the chains of thought is that by default they're in English. Right. And so it's so much easier to understand what is going on, especially as the concepts get more advanced.

• 40:19Speaker 00

  And the other interesting thing is, you know, we were just talking about how probably, you know, how we believe in the future where we go, well, these models work for a very long time. They work autonomously, right? And so there is much more of this reasoning. And so if this is a big axis of how the capability of these models increases, that our ability to supervise them will scale commensurately. Yeah, this really comes down to this principle, though, that you're not supposed to supervise the chain of thought. And so this is actually something, when we originally, you know, we were releasing the own preview model, like we made this decision to like hide the chains of thought. Yeah, I remember.

• 41:01Speaker 00

  And for me, that was the primary motivation. That was the reason like I didn't really even want to consider realizing it in different ways. You know, there definitely was a bit of a challenge discussion about this, but like the reason I felt very strongly like we should, we should just hide it is because of this. Then there was this other concern that like I didn't initially think about, but I think was also like very valid of like, well, you know, this model is going to distill to some extent, blah, blah, blah. And that's definitely also been like a big factor here. But, yeah, but I actually think that like this. You know, allowing the models some sort of private space. Oh, and by the way, like, why do I think it's important that we don't like, you know, show this change of thought in product?

• 41:43Speaker 00

  You know, if I'm saying like the importance is not to supervise them during training. Well, I think if we did show them, if we like establish a paradigm where like, oh, you just show this change of thought in product.

• 41:54Speaker 00

  Eventually, you kind of have to train them, right? Like, you have to train them for the same reasons you have to train like whatever models you ship.

• 42:01Speaker 00

  And I just think that would be.

• 42:02Speaker 01

  We might not all want to know what the chain of thought our model has that gets to a response for.

• 42:05Speaker 00

  Right, I mean, you know, I think I think it will be useful to some extent, and we are trying to capture most of that value, you know, either with like chain of summaries, which I think are kind of like a little bit of a stopgap. I think the longer term solution here is having the model actually talk to you in real time, which, you know, the later, the latest version of Codex, kind of the latest version of the reasoning GPT models kind of do. I think I think that will get much better.

• 42:30Speaker 00

  Yeah, but but yeah, I think there's something very exciting here about just like not not having the training signal fight against us, right? And not not, yes, because, yeah, I think if you

• 42:46Speaker 00

  if you want to be able to understand what the model does in the long term, but you know, you're scaling a method that is like kind of going directly, it kind of said, it's, you're probably not going to have a lot good time, right, unless the others have a better lesson.

• 42:58Speaker 00

  And so this decoupling, I think, is a very, it's an idea that gives me a lot of hope for our ability to at least understand, you know, how these models motivations and generalization evolve as they get better, as they work for longer. Yeah, I don't think it's a complete solution to AI system alignment by a long shot. I think it's just another tool in our toolbox. But I am hopeful that by building our toolbox with technical tools like this, we can actually continue chipping away on the fundamental problems here.

• 43:27Speaker 01

  Yeah, it seems like almost like over the, you know, medium term, it's like something's going to be incredibly helpful. Probably not the catch all solution for long term alignment.

• 43:35Speaker 00

  Yeah, I mean, I think it's a tool that can help us understand. I think it's actually very useful to like build understanding of long term alignment, right? So, for example, there has been this very exciting work.

• 43:45Speaker 00

  From a planned collaboration with other labs on model scheming, where they investigate, depending on what environment you put the model in, how you train it, is it prone to start having hidden objectives that it pursues? And what enables that whole line of work is chain of monitoring, is this notion of, oh, you can actually inspect what the most motivations are. So, and I think from that, that might take us in a completely different direction in terms of mitigations, right? Like, maybe the right way is changing the pre-training data of the model, or maybe it's something like the inoculation prompting from a topic. I think those are very interesting ideas, but I think having this ability to understand is very helpful to evaluate these.

• 44:30Speaker 01

  Yeah, it's almost like foundational for any further area of research. What are the other research areas within alignment that you're paying attention to or that you think are promising areas to focus on?

• 44:41Speaker 00

  Yeah, I think a lot of the longer-term challenge with alignment is about generalization, right? Like, we can train our models to do well,

• 44:54Speaker 00

  or at least mostly to some extent. Like, we can mostly kind of control their behavior in the things that are in distribution that they have to be trained for.

• 45:05Speaker 00

  But the things that are worrisome is like, well, what happens when a model is asked to do something very, very different? Or it finds itself in a very different situation, or it's like much smarter than it ever was before, and it has all these capabilities. It's like, we haven't really thought about how to train for. And so, yeah, so I think the study of this kind of longer-term value alignment is really a study of generalization. Like, what are the values that the model falls back on?

• 45:30Speaker 00

  Like, one line of research I'm very excited about here, and something that we're investing in quite a bit, is understanding how the generalization falls back onto the pre-training data.

• 45:47Speaker 00

  Yeah, and yeah, I think there's quite a lot there.

• 45:52Speaker 01

  I guess over the last six months, have your concerns around alignment increased, decreased? Like, where are we trending overall with this work?

• 46:01Speaker 00

  I will speak to the longer-term challenges of value alignment, right, or what happens when you have very smart models. The way my thinking about the problem has evolved over the past few years is that I've kind of gone from

• 46:13Speaker 00

  You know, oh, is this like a very nebulous problem that like is just like very hard to even grapple with or define to like, oh, you know, I think I actually make progress at it by very concrete technical solutions and technical insights. And this is why we've really been

• 46:29Speaker 00

  viewing alignment as like just a core part of of research and really, you know, making sure that like we are, you know, designing our reasoning models, thinking about this and we are, you know, and we are kind of like conducting our alignment research with like these reasoning models in mind and so forth. So I think my general kind of believe that there is like a research path here that actually gets us to an extremely happy world has increased quite a lot. At the same time, right, I think my timelines to very capable models have definitely decreased. All right, I think we're we're not that far, right? And again, I don't think these are models that are smarter than us in some of the ways, but I think these are models that are just very transformative.

• 47:12Speaker 00

  And so I'm quite optimistic that we can keep a good grip on like how we're doing on the alignment problem, how to roughly evaluate the risks of of of of of our models or the problems with them. But, you know, but I do think we have to be, you know, as an industry, as really prepared to like take trade offs and, you know, and possibly, you know, slow down development, depending on what we see.

• 47:36Speaker 01

  It's already interesting to see a lot of this work happening across the major labs, you know, the fact that you did this in collaboration with, I think, Anthropic and DeepMind. And, you know, it seems like, has that just come up organically or imagined? Like, is there a lot of like alignment talk between, you know, the the major players, you know, given, I guess, the three of you are really at the forefront of all this?

• 47:54Speaker 00

  Definitely some. I mean, there's definitely like shared interest in these topics, yeah.

• 47:57Speaker 01

  I want to shift a little bit to going inside OpenAI. I feel like no, no company probably, or the world has been more interested in over the last two, three years. And, you know, I think particularly what it's like to run a research organization. You know, we talked a little bit about this previously, but you talked before about how it's, you know, important part of your job is giving researchers, you know, to to kind of have comfort and space to, you know, almost be cave dwellers, right? And think about what the models will look like in a few years. You know, we were kind of alluding to it earlier. We're also in a time where it feels like there's just massive competitive race and, you know, it's certainly, you know, everyone's going really gung ho on these coding models. I'm wondering, like, how do you actually operationalize this balance today? And, you know, anything you've kind of changed in your thinking?

• 48:41Speaker 01

  You know, overseeing this organization around the right way to do this.

• 48:44Speaker 00

  Yeah, I focus on on just high quality experiments, recognizing, you know, are we actually making progress, being honest with ourselves and, you know, and promoting honesty about about the results. I don't think that has changed, right? And and, you know, even though our work will evolve a lot, I believe we still have quite a lot of work left to do. And so I don't think it's like, oh, you know, we need to wrap up all our projects, you know, very, very quickly. So, yeah, I don't think those fundamentals change. I think what what that changes is, you know, a level of virility to a kind of bring some of these things that we think are most promising to fruition.

• 49:19Speaker 01

  And then obviously, you know, I feel like there's been, you know, some very public internal moments of OpenAI over over the years. You've been here for a long time. As you got to reflect back, like, what were some of the difficult decisions that you guys made that maybe were like fifty one forty nine that really, you know, defined the company? Or any, any, as you think back on the movie of the last, you know, seven, eight years of your life, you know, the key moments that kind of stick out to you?

• 49:42Speaker 00

  Well, yeah, I mean, there's certainly a number of, you know, dramatic moments like this.

• 49:47Speaker 00

  You know, I think the ways the company underwent the most changes, not really this like snap changes, snap decisions, but more like just like shifts in how it operates, right? I would say like OpenAI has gone for a couple of phases. You know, when I joined at the start of twenty seventeen, very much a kind of felt a very academic lab, pursuing like a lot of different ideas, not so, you know, scaling build in practice. And I think that was like the first like big change, right? Like with the data product with GPT, we've kind of moved to, okay, like, we actually are going to have to buy big computers. We're actually going to have to scale things. We're going to have to develop the science of scaling, or we'll have to develop the infrastructure for it.

• 50:28Speaker 00

  And so that kind of started the second phase of of, okay, now we're scaling. I like where we're, we're still going to pursue like a lot of these basic research ideas, but we are going to evaluate them like for the actual, are this, are this scalable?

• 50:43Speaker 00

  Then, yeah, then there was this interesting period that I talked about earlier, right? Where you kind of have ChatGPT as this big thing.

• 50:51Speaker 01

  Complete surprise, right?

• 50:52Speaker 00

  Yeah, I mean, I felt it would look a little differently, right? Like, I think, I was actually surprised at like text models. I was pleasantly surprised, like, text models are actually kind of the first thing. I thought we would be in a world where like, it's more the kind of like, you know, video style, users of AI are kind of like the first.

• 51:11Speaker 01

  Yeah.

• 51:11Speaker 00

  The first big things take off and like, and we'll have to like trade off like pursuing the kind of longer, longer term text based research. So, yeah, so, so, so, but, yeah, but I think definitely like we anticipated that like this sort of tension would arise, right? Where like, you have a thing that is kind of like popular now, but it's like, you know, you believe it's going to evolve quite a lot before you get to where you're going. And so I think that's kind of the phase we've been in for a while.

• 51:38Speaker 00

  And, yeah, I think now we're, we're like, uh, well, yeah, I mean, we believe we are kind of starting to be in this phase where, yeah, we're actually deploying AGI or, you know, deploying models that are actually very kind of transformative.

• 51:53Speaker 01

  No, it's, uh, it certainly seems that way. Well, I guess we always like to enter interviews with a standard set of quickfire questions, which are basically me just stuffing all my overly broad questions I couldn't fit anywhere else. So if you'll shamelessly indulge me, you know, I guess to kick it off, with love, what's one thing you've changed your mind on in the AI world in the last year?

• 52:11Speaker 00

  Yeah, I mean, I think, I think it's really, you know, starting to reconcile this tension between, you know, the AI that you build ultimately is something that affects the world, but, you know, until you, until you kind of get pretty close, it's like a pretty theoretical thing that you're just kind of, you know, training and developing algorithms for. And so, you know, recognizing that, okay, now we actually need, we really need to.

• 52:40Speaker 00

  You'll make a lot of progress and focus on like how actually we're deploying this technology in a while. This is definitely something I've been thinking about a lot lately.

• 52:50Speaker 01

  Yeah, it's interesting. Basically, like, you know, outside of chat GPT, it was almost like more in the abstract or research hill climbing, you know, with some usage in the real world. And then in this last year, we've obviously seen primarily via coding agents just, you know, it trickling, you know, in a pretty massive way.

• 53:06Speaker 00

  Yeah, I think I believe chat GPT is kind of going in the same direction as like the coding models where like it's actually going to be something, you know, very useful. It's going to be something that's like a meaningful part of people's lives.

• 53:19Speaker 01

  When you say going in the same way, you mean just like executing longer term tasks or more like, you know,

• 53:24Speaker 00

  I think that's part of it, right? But also just, you know, coming to become like a dependable, trustworthy assistant or companion.

• 53:32Speaker 01

  Yeah, it's amazing to watch the way younger people use chat GPT. I'd argue it's already pretty much there for the way a lot of folks in high school and college and, you know, seem increasingly, you know, comfortable using it. You know, I wouldn't be a shameless podcaster if I didn't ask a top researcher, you know, timelines for a few things. I think particularly actually interesting is the stuff outside of the core LLM world. And so I think there's a lot of buzz around robotics these days. Do you have any like, I mean, obviously, it's hard to pinpoint like a moment robotics quote works, but I think, you know, whether it's finding scaling laws or finding some sort of like chat GPT esque moment for robotics.

• 54:05Speaker 00

  Yeah, I mean, I definitely think there are like very promising algorithmic ideas that I believe are going to work that are, you know, not too dissimilar from the LLM space of ideas. So I'm quite optimistic about timelines there, although I do think they're longer than like the kind of the virtual AI.

• 54:23Speaker 01

  Obviously, I'm sure you think a lot about, you know, because you're always thinking about the next frontier for what these models can do, you know, just the impact on society as a whole. As you think about this kind of pace of continued model improvement, you know, what's maybe one thing that you think we're underthinking right now as a society in terms of the impact of these models?

• 54:38Speaker 00

  Yeah, I think getting to a point where so much intellectual work can be automated, I think, comes with Pretty big problems. I don't think I have obvious solutions. One, that's always a question of jobs and concentration of wealth. And I suspect this requires like real policymaker involvement. Yeah, I've heard some kind of optimistic takes on how is this resolved? But I think at a fundamental level, it does seem like some things that like used to be very valuable, used to kind of cost a lot and used to provide something like now can be done pretty cheaply. And in the long term, it should be a good thing, but I think it does lead, I think it can happen quite quickly.

• 55:27Speaker 00

  And there is a related question of, you know, you really can, like if you actually have, you know, an automated research lab or an automated company that can do so many things, like it can be controlled by a very small number of people, right? It can be, it can do a lot, right? And this gets, this gets even more crazy when you have robots, but you don't need to have robots. And I think figuring out like what does governance of such things looks like, look like, right? Like what are these like organizations that are like so powerful and yet maybe made of like only a couple of people? Like what, how to think about these things, I think is a new question to grapple with as a society.

• 56:03Speaker 01

  I want to see other new questions. One thing that's very top of mind for me, I recently had a kid and I've been thinking a lot about like, you know, what is his life going to look like in 10 years? You're really close to this stuff. How has your work on AI changed the way you think about like the way in which, you know, this next generation should be raised?

• 56:20Speaker 00

  A task for all of us, right, is to build the AI, right? Build the world in a way where, you know, the idea of like humans have the agency, right? Humans set the direction, right? And, you know, maybe a lot of the, the technical challenges that we cherish right now will become more of a, you know, past line, that's something that we really kind of like need to do in order to make progress. And the challenges will be more and like figuring out like what are the things that are important? What are the things we should go do? You know, I think that that will still be. I think in that world, like, people can end up with more things to do and definitely more exciting things to do. And I think you still want to have an understanding of like,

• 57:06Speaker 00

  some understanding of like technology, like all the kind of like basic education, however you want to acquire it for the sake of being able to think about these problems.

• 57:15Speaker 01

  Well, this has been fascinating and I really appreciate you sitting down and talking about so many different things. I want to make sure to leave the last word to you, like anything you want to point our listeners to, whether it's research you're doing or products you're excited about, or really anything you'd like to plug the floors ears. You know, anything, I'm sure there's tons of threads people want to pull on in this conversation.

• 57:35Speaker 00

  I think this set of problems we just discussed, right? And also the questions around alignment and monitorability. I think I think those are going to be very urgent challenges. And I don't think they're challenges only for AI researchers, right? I think there are challenges for policymakers, but also also just things we have to think through as a society. And yeah, I'm, you know, I'm happy to see some discourse starting to arise and I think we need part of it.

• 58:05Speaker 01

  Yeah. Well, I thought I could talk to you for hours more, but I'd be doing the world a great disservice by keeping you from your actual work of continuing to improve these models. Thank you so much for doing this. This was a ton of fun. I'm Jacob Effron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.