OpenAI’s CPO on what’s coming next: Hardware, GPT-5, Jony Ive, agents, more

This week, I'm speaking with Kevin Weil, Chief Product Officer at OpenAI, who is steering product development at what might be the world's most important company right now.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We're in this transition from ChatGPT being a thing that answers questions to a product that actually goes and does tasks for you in the real world. You should be in control of any actions that it takes. The best way to do that is to kind of co-evolve together.

• 0:13Azeem Azhar?

  Sam has sort of talked a little bit about generational differences. So what is it like if you're in your early twenties, you're using ChatGPT?

• 0:19Speaker 00

  If you're younger, it's just a core part of the way you operate your life. There's sort of an always-on nature. You realize that you have this super assistant in your pocket that can not just answer any question, but it can teach you anything that you want to learn.

• 0:33Azeem Azhar?

  How far out are model capabilities being delivered? Are there things that are being worked on today that might not make their way into models until mid 2026?

• 0:44Speaker 00

  It's unpredictable. Sometimes things take longer than expected. Other times you see these capabilities that you didn't expect at all that are kind of emergent. And all of a sudden, something just works. It's a completely different way of building products. Computers can like do things that they couldn't do two months ago and we're constantly in that state.

• 1:06Azeem Azhar?

  Today, I'm so excited to welcome Kevin Wile. He is the Chief Product Officer of OpenAI, coming off the back of a storied career in companies like Twitter and Facebook. So much to talk about. So first of all, Kevin, thank you for making the time. You've been busy today already. It's morning in San Francisco and you've already launched some things.

• 1:28Speaker 00

  Yes, we have, and thank you so much for having me. It's my first, uh, my first live Substack, so I'm excited.

• 1:33Azeem Azhar?

  Well, hopefully many more, many more to come. You are really keeping us on your toes. And so just an hour before we went live, there was another spate of product launches from OpenAI. What do they tell us about your vision for ChatGPT and where the product is going?

• 1:48Speaker 00

  The biggest thing that we launched this morning, we launched like six different things this morning. But, um, the, I think the most important one kind of for the long-term future of AI is we launched a series of connectors that can connect to your either personal data or if you're in enterprise, your enterprise data. So this is connectors into Google Docs, to Gmail and Calendar, to SharePoint, to OneDrive, Dropbox, Box, Linear, all of these different tools that you use every day to get things done.

• 2:19Speaker 00

  With the rise of our reasoning models, connecting them into the services and the data that you use helps the models be way more useful. So it's not just that you can now ask, at work, for example, if you connect it in for your Google Docs or SharePoint, you connect it into the docs that you use every day. Suddenly, ChatGPT can get all of this context on the enterprise and on the state of projects and on the latest about any particular thing that's going on. You wouldn't ever have an employee at your company that you didn't give access to docs. It's like where the conversation and the strategy and other things are happening. So now you have ChatGPT that has the ability to do it.

• 3:01Speaker 00

  And today, they're read-only, so they're able to access the information but not create on the other end. But you can imagine going forward, another big part of this is that ChatGPT should be able to take actions, should be able to help you write a document or create a presentation or write tasks to your task management system and ultimately combine all of these together to begin really working like an employee.

• 3:25Azeem Azhar?

  Right, so that's the hint, right? That direction of moving ChatGPT from something we interact with, with a challenge and response, into something that is more evolved and feels like it's really doing work for us as an employee. We're going to spend some time digging into that. It's such a big vision. Let me ask about you. You're the chief product officer of what I think is the most important company in the world today. So how does that feel like to you day to day and week to week?

• 3:56Speaker 00

  I mean, it's a privilege. It's the most exciting place I've ever worked and I've been very fortunate in my career to work at a bunch of awesome places with really great coworkers. But I think the opportunity in front of us, the rate at which AI is changing all of our lives, means we have an ability to make a really big impact and we take that super seriously. So I get to work with amazing colleagues. I get to kind of have a front row seat to the way that our models are evolving and that the way this is all impacting our lives. And hopefully, we get to build some products that make a difference in your life and in the lives of everybody listening here.

• 4:34Azeem Azhar?

  Well, nearly a billion people have made, at least voted with their, their mouse finger to use ChatGPT just since November two thousand and twenty-two. Technology always changes us. The television did, we got TV dinners and we got water cooler discussions. The internet obviously changed us in, in different ways, cars as well. We got big box retail, we got the suburbs, we got the picket fences, desperate housewives. How might AI products reshape our everyday life?

• 5:08Speaker 00

  Well, I think one of the other interesting things you get when, when you have these waves of technology is you start by doing the thing that you were doing with the previous wave, but sort of with the new medium. You know, the first TV advertisements were people standing on a stage, reading their radio advertisements. And then people slowly figure out that you can actually, you know, you do the what we have as commercials today, which are much more interactive and, and sort of dynamic. And so, you know, we're still probably in that mode of people, when they look at the impact that AI can have in their lives or in their work, they're kind of like, okay, I have these processes. How do I, you know, sprinkle AI on top of this to make it better, faster, et cetera? And like, that's fine, that's all good.

• 5:53Speaker 00

  As in previous technology transitions, the power comes from completely reimagining the work that you're doing from first principles, using the new technology. You know, so mobile wasn't just about, you know, a, a computer in your pocket. It, it was, you know, you have access to GPS and you have these totally, you have notifications and totally new ways to interact with, with technology. I, I think over the next, you know, year, we're all going to be in the process of reinventing the way that we do things with AI. And the fun part is the technology is moving so fast. It's not, I mean, faster than any technology I've ever worked with before, I've ever seen in my career.

• 6:32Speaker 00

  So even as we're reinventing, the technology is gaining new capabilities and it's just a, it's an exciting time to be alive.

• 6:40Azeem Azhar?

  I mean, is that that rate of change is quite something. I did a, I was looking for something today, which I know would not have been possible four months ago. So I was using O three to help me find a little portable mobile five G router for when I, I travel and I just need to get a good quality signal. And of course, O three went into all the technical specifications of the radio frequency chipset and said, well, you know, this earlier router has got a slightly better Qualcomm chipset. They don't know, no longer use and you'll get one extra signal bar in rural environment in the U S, but not in Europe. And I'm thinking that's in a game. This is, this is kind of insane. But I'm really curious about that, those types of behaviors.

• 7:24Azeem Azhar?

  What are the surprising behaviors that you've seen emerge around chat G P T from your, your users? Sam has sort of talked a little bit about generational differences, but give us a flavor of things that you've learned about behavior that are just really hard for product teams to, to research or figure out a priori.

• 7:41Speaker 00

  Well, I do think, I mean, Sam has talked about the way that, like I said, the, the people today, I think a lot of people are sort of sprinkling A I onto their existing workflows. And it's, it's young people that are, you know, there's sort of, this is native to them in a way that it's not native to, you know, those of us who grew up without A I. And it's, I mean, my kids just, they're like, of course you can talk to a super powerful A I that like, you know, can customize itself and answer any question you have. And kids that are graduating college these days as like engineers are like, they don't know any other way to write code than using, you know, A I editors like cursor and windsurf.

• 8:25Speaker 00

  And it's just completely natural to them. And so it gives them superpowers in a way. And it's one of the reasons that we look a lot at how our youngest users and our young users, you know, late teens, college users, things like that are using the product because it teaches us a bunch.

• 8:41Azeem Azhar?

  Can you characterize those differences though? So what is it like if you're in your early twenties, you're using ChatGPT? How does it feel different to somebody who might be in their mid thirties or early forties?

• 8:53Speaker 00

  There's just a, there's sort of an always on nature. You just, you realize that you have this super assistant in your pocket that can not just answer any question, but they can, it can teach you anything that you want to learn. And so when you, when you go about life that way, the rest of us are like trying to remember, you know, trying to think about the processes that we go through and how we can reimagine them. If you're younger, you might not have had those processes. And so you built them from the ground up with AI. And so it's, it's just a core part of, of the way you operate your life. And in some ways they're ahead and the rest of us are catching up.

• 9:30Azeem Azhar?

  A lot of people are scared of this technology. A lot of people are, you know, nervous when I've just come back from Brussels and spoken to people in, in a different sector. And you do get that sense of a fear. What could change in the product to address that?

• 9:48Speaker 00

  One thing that, that I think is really important is people ask, you know, what, what, what should I do about AI? How should I think about it? And my answer is always just use it. And, you know, of course I think everybody should try ChatGPT, but whether it's us or any of the others, just start using it. Because you, it's the number one way to realize it's not this like super scary thing that you read about. It just helps you get more done and you're like, oh, this is great. I, I now have this like incredible new like thing that can be part of my life and help me get things done and help me like automate a bunch of the like boring work that I have. So the number one thing is just like, just start using it. Also, given the rate that the technology is improving, if you don't start using it now, it's going to be even harder to sort of catch on to it.

• 10:32Speaker 00

  And if you believe that AI is going to be a big part of our lives, that's not a train you want to miss. So, you know, that's the number one thing, just start using it. But then, you know, we, we think a lot, especially as you get towards agents and other things. We think a lot about making sure that the user is in control. So you don't want the, you know, as you're getting to to use ChatGPT or any other AI, you don't want it to go off and do a bunch of things for you without you, you know, feeling in control. And so, you know, it's one thing if it's like answering a question for you, reading some docs and summarizing them, things like that. But as we're in this this transition from ChatGPT being a thing that answers questions to a to a product that actually goes and does tasks for

• 11:15Speaker 00

  you in the real world, and you should be in control of any actions that it takes. And over time, as the models get better and you begin to trust them more than sure, you can give it, you know, sort of more leash and trust it to take more actions autonomously. But you should control that every step of the way. And I think that's one of the most important ways that we're looking to build trust.

• 11:35Azeem Azhar?

  I mean, there's a lot that will change as you move to more and more of these agent-based workflows and the models be able to use tools. And I guess we have to co-evolve. Definitely, we should, we should look into those questions. I'm just curious about your own experience. A lot of us have these oh shit moments when we use L L Ns, right? There's, there's something sublime that happens, or there's two years of your work that gets returned in in five seconds. What was your most recent oh shit moment using using your own products?

• 12:06Speaker 00

  Well, so I'll give you, this one's a little bit more, I don't know, pragmatic, but but it was a really meaningful thing for me. We have, we were talking about kids as people were sort of coming on. And our, one of our sons had a minor surgery. And it was, it was one of those things where all odds were that it wasn't going to be a big deal. But there's a small chance that it was a really bad thing, you know? And so they they do the surgery and like they take the thing and they go to biopsy it. And you're waiting for, you're waiting to hear back. And as a parent, you're nervous, even though you know logically that the odds that it's anything bad are are really small.

• 12:48Speaker 00

  And at some point, we got a letter in the mail with a bunch of, you know, doctories on it that looked pretty intimidating, frankly. You know, there were a bunch of words I didn't understand and, and they're characterizing what this thing was. And it wasn't, it didn't say like, you should worry about this, or you should not worry about this. It just like characterized it and then, you know, finished. And I couldn't get ahold of the doctor. She was in surgery or something, you know? And so there I was like, oh my God, what does this mean? And so I took a picture of it. I put it in chat GPT and I said like, should I be worried? And I said like, can you explain this to me like I'm five? And chat did it and was like, no, this is totally fine. Everything like nothing to worry about.

• 13:32Speaker 00

  And I actually ended up not being able to get ahold of the doctor for 72 hours because she was just super busy. And that's 72 hours would have been a terrible 72 hours for me as a parent. If I was just sitting there brooding and chat GPT was able to answer. And that's like, that's, you know, us with like access to great healthcare. You think about the impact of this all over the world where people don't have the same access. It's really powerful and I think that's sort of an under, underappreciated part of chat GPT.

• 14:01Azeem Azhar?

  I mean, that, that is a wonderful story and I'm glad that your, your son is, is healthy and comes out of that well. But it also speaks to the power of this particular, this particular product, right? It's a really complex, complex product. And there's no way I can talk to you without asking you about how that complex product finds its way in that funny dropdown in the top left of, of chat GPT. There must be something that this, that you, you guys are all so brilliant. There must be some internal joke about, well, how should we order it? What should we call the next one? What's really going on there? And you know, shouldn't we all just be using O3 and? For oh for in a hurry.

• 14:41Speaker 00

  It's a totally fair question. And you can also, you can and you should make fun of us for our naming as well. It kind of comes back. So we have this philosophy of iterative deployment, which is these models are, I think AI is going to change all of us. It's going to change the world. It's going to change society. And we believe that the best way to do that is to kind of co-evolve together, is to get these models out there and put them in people's hands, help them understand. And also, you know, they help us discover the capabilities of the models and the weaknesses and other things. And so we, we sort of learn together and can iterate really quickly and improve. So that's part of it. The other part of it is we're building a lot of new capabilities as we go.

• 15:26Speaker 00

  And if we took the time to only, you know, we, we just had like one model and we just had to build everything into one model, we'd end up moving a lot more slowly. And, you know, things would be simpler for sure, but we'd end up moving a lot more slowly because sometimes it's easier to build a new model with a certain set of capabilities that's great at certain things, not as good at other things. And so together you have sort of a, a, a collection of models that, that can do lots of things. Each individual model has its strengths and weaknesses. And so basically we've just, we've optimized for going faster and getting more capabilities in people's hands at the expense of a bit of confusion. And then over time, as we sort of gain more control over some of the new functionality,

• 16:11Speaker 00

  we understand it better, we build it back into the core model. So you have models like GPT-4 that can do a lot of things well. And, you know, this is what we're trying to do with our forthcoming GPT-5 is take a lot of the things that we've learned and build more of the capabilities into a single model so that it's easier for people to reason about. It's like, what model do I use? Just use GPT-5. And you know, in a perfect world, it knows how hard the question you ask it is. And so it knows whether it should give you an answer like this, or whether it should think for a while. And you know, that that's what we're shooting for. So.

• 16:49Azeem Azhar?

  And you know, in a way, that's lifting the cognitive load that is currently on on the user. Because I will sit there and I'll think, do I have time? Is that conflict case question? Does it need to go to the reasoning model O three? Does it need a longish prompt for O three? Because it might get the wrong end of the stick if I give it a shortish prompt. And in a way, what you're you're talking about is is crunching that all and building it into the model itself.

• 17:14Speaker 00

  Yeah. And and look, what I don't want to do is over promise and say, oh, in the future, like, we'll only ever have one model and it's going to all be simple. Because we we're also, you know, say we launch G P T five. We're then going to have a bunch of new capabilities beyond that that we're trying to build and experiment with. And we're going to want to get those out to to people and deploy iteratively and so on. So I expect you always have this sort of phenomenon of there are new models. And then and you've got sort of your workhorse models. And then you've got some of the new ones that have certain like frontier capabilities that we're experimenting with and learning with together. And then over time, as those mature, they go all kind of get merged back into single models.

• 17:53Azeem Azhar?

  So it seems that a lot of the velocity is is actually about getting through the the loop, right? The learning loop. And you just run run all these horses at the same time so you can gather enough data about what models work against what what capabilities. In a way, that helps you develop and deliver on G P T five. And I I hope you'll share the date with us.

• 18:15Speaker 00

  Down to the minute.

• 18:16Azeem Azhar?

  Down to the minute. Yeah. So we can tie ourselves. But but does that also mean that, you know, how how far out are model capabilities being delivered? Are there things that are being worked on today that might not make their way into models until mid twenty twenty six?

• 18:32Speaker 00

  Yeah, it's a good question. It's one of the most interesting things about working here is you kind of have a sense of what's coming. And you know, I'm not in the research team, so I'm getting this, you know, as I collaborate and work with the researchers. I would say like, you know, on the product side, we have a decent sense of what's coming in the next, say, three months. Maybe a hazy sense over the next six months and beyond that, it's harder to say. You know, you have a certain set of capabilities, you know, where like, you know, a bit, but you see things coming through the haze a little bit. And sometimes, sometimes capabilities are, you know, it's research, right? So it's not like you just, you have the formula and you just turn the crank.

• 19:15Speaker 00

  We're uncovering new things and it's unpredictable. So sometimes things take longer than expected. Other times you see these capabilities that you didn't expect at all that are kind of emergent. And all of a sudden, something just works.

• 19:28Azeem Azhar?

  Mm hmm. Can you give an example of one that just, that just worked that you weren't expecting?

• 19:33Speaker 00

  Well, I mean, deep research is an interesting example where for a while, there were a handful of researchers thinking about the, it was like, okay, we could probably make the model able to do this like iterative kind of research where, you know, with deep research, you give the model an arbitrarily complex query to go research something that would probably take you a week. And it will go off and it'll go off and do like a hundred searches, but not all at once. It'll do, it'll do three or four or five. And then it'll reason about the results that it gets back. Try and understand how they pertain to what you asked and what gaps are still there. And then it'll go off and do some more searches and maybe think again.

• 20:14Speaker 00

  Maybe it'll write some code for a little while as it thinks and then go do some more, you know? And so it's this sort of iterative, I mean, it's what you would do if you were, if someone made you write a very complex research report, you wouldn't, you go do some research.

• 20:27Azeem Azhar?

  Sorry, Kevin, I don't do that anymore. I just go to deep research. I actually can't remember how to do it on my own. But yes, I understand, right? You have to, you sort of inch your way across and you figure out exploration strategies and go down dead ends and come back.

• 20:41Speaker 00

  Yeah. And so to your question, like that, that was something that some folks were like, okay, this is coming together. But it's not clear exactly when it will come together. Like we we can. And so there was this, there was a small team of researchers that just believed in this and and were working to make this real. And, you know, for a while it wasn't good enough. It wasn't good enough. And then there was, you know, some advances and all of a sudden you're like, okay, this is getting good enough. And somewhere, somewhere in that timeframe, we put also a product and engineering team working, working on it with them. And then you have the thing that I think really is the magical part of OpenAI. When you get a research team and a product and engineering team just, you know, in the

• 21:23Speaker 00

  same room, all bringing their unique skills to bear and you're, you, you understand the problem you're trying to solve. And so you're bringing back use cases, creating evals and benchmarks for how you measure whether you're successful against those use cases. The research teams are taking that and using that to improve the model itself. And you get this tight loop of the model improving towards a particular product. And it's, you know, I, I think our best products are the ones that we build that way. And deep research is a good example.

• 21:52Azeem Azhar?

  That's quite, I mean, that's a novel way of, of thinking about product development. I mean, if I think about the history of, of, of product development, quite often it was, you know, before the nineties, before the consumer internet was run by engineers and they'd say, we've got a new chip that can do this. And you then try to figure out some software that can do useful things on that chip. I think the big breakthrough of the consumer internet was to put product managers at the heart of product development. And you, you did, you know, we talked about lean and iteration and being very data driven and user centric. And now we're getting, you're getting to this new model, which I would characterize as not at all a return to the pre-internet product engineering led, but, but something

• 22:34Azeem Azhar?

  that is quite, quite novel because the researchers discover something that's a little bit a new capability. And then you have to have a very rapid discussion about how can that capability be productized? And then this word you used eval, which I guess it means, how can it be measured to see whether it's actually doing its, its tricks? So, so is it really a new discipline that is evolving at this point?

• 22:56Speaker 00

  I think it's a completely different way of building products. It's certainly different than anything I've ever I've ever done in my career. And, you know, within research, there's kind of a, there's a spectrum, right? There are, there are parts of our research team that are, that are just like a deep research. It's almost academic in nature because they're trying to just like, they're, they're looking for new breakthroughs. They're trying to, to find things that nobody has ever, you know, figure out things that nobody's ever figured out before. And those kinds of things, you don't want to be, you know, you don't want to be product driven at all. Because you, you just, you want to give a lot of room for exploration and fundamental breakthroughs. And then there's kind of the other end of research.

• 23:34Speaker 00

  That it's more in the, on the post-training side where you really are trying to teach the models to do specific things very well. And that, those teams tend to be much more like, you know, partnered with product and engineering teams with a common goal. And then it's kind of a spectrum in between. I think the right way for us to be is not, we certainly don't want to be entirely product led. That's not, that's not the magic of this place. It's not maybe entirely research led either, because it's good to know, you know, feedback about what problems you can solve for people and how we can make the biggest impact in the world. It's really sort of a combination of both, with research really at the core, though. And I, like, I've loved it.

• 24:17Speaker 00

  It's, it's the most fun in the world and, you know, move super fast. Computers can like do things that they couldn't do two months ago and we're constantly in that state.

• 24:27Azeem Azhar?

  But when you're working with research, then it's more than just looking at a scaling law and saying, oh, we've just, Sarah Frye has just signed off another hundred thousand GPUs. Therefore, it will, it will be able to do this in six months when the trading run is done. There's, there's more to it than that. But one of the things I find fascinating is how do you map those capabilities against products? And you talked about evals, which I guess are evaluations. What does, what is the structure of, of an eval? And is it, is it what replaces what was in an old product requirements document that we might've had, yeah, 15 years ago?

• 25:03Speaker 00

  Yeah, sort of. I think in some ways for for understanding where the model is good and where it's not. It's if you think of the model as sort of an intelligence of some sort, intelligence is so multifaceted. People are are smart in a million different ways and one smart person is better than another in certain areas and worse than another like. So one way to think of evals is as is a way to measure capabilities and intelligence of models on different dimensions. So you can have evals around, you know, how how good it is at like solving USAMO Math Olympiad style problems and another around how good it is at chemistry and another one about how good it is at creative writing.

• 25:45Azeem Azhar?

  But are you using the public benchmarks for that, the sort of ArcAGI and AMI and GPQA as your as your way of measuring?

• 25:54Speaker 00

  Some. And then also for when we're building specific products, I think one of the most effective ways to build products is to to take the the skill that you want the model to have in order to to meet the product need, turn that into an eval so you can actually understand whether, you know, how good you are at it and also how you're getting better over time. But one of the fascinating things is like the evals that we all used, you know, a year ago to measure models, they're all very kind of cut and dried. Like you're testing against math and with math, there's a right answer. You can talk about creative writing evals, though, and with creative writing, there's no, there's no answer. So how do you grade that? Right? That's one, that's one problem. The other

• 26:35Speaker 00

  is like, as you start to take on more complex tasks, you're not just answering questions, you're actually trying to, you're trying to like automate some multi-step workflow. There may be ambiguity in the right way to do that. If I'm, you know, if I'm an AI booking a flight for you, there's not a a single way to grade which correct flight, you know. You also get into these really interesting, challenging, subjective ways of how do we actually grade this particular task? And part of having an eval, if you want to at least automate it, is you need to also have a grader for it so that you can very quickly understand how you're doing on that eval.

• 27:11Speaker 00

  So it is interesting. It's one of the skills that I think is going to be more and more important for PMs over time is the ability to actually create evals for the products that you're building.

• 27:21Azeem Azhar?

  Yes, I mean, that's one. And the other one is the is the prompt at the front end. Because what we are starting to see with these leaks, and I don't know how real they are or not, but there are a number of X accounts that say, I've just had the leaked system prompt of then insert your favorite foundation model or coding tool. And the the the system prompts, which are the sort of structured instructions that go out with every query are really, really quite complex. I mean, they're products in of themselves now, right? They run to thousands of words. They're highly structured. There's clearly strategies being applied as they get put together. So how important is that skill and capability when you're shipping products to people like me?

• 28:05Speaker 00

  I mean, actually, more than more than people realize, I think. I would love to make it over time, less of a thing. And I think over time it is like, if you go back a year or two, everybody was talking about prompt engineering and it was going to be the skill that everybody had to master in order to do anything with AI. You don't hear about it. You don't hear it talked about quite as much like that. And I think that's a good thing. You know, ideally, it matters less and less that, you know, for any particular user, if they have a question, they want an AI to do something for them. You shouldn't need to get into like Arcana around, did I use the exact right word? And did I give my example? You know, it should just work.

• 28:46Speaker 00

  I think that's part of increasing intelligence is the model can understand what you're looking to do and do a good job of it without you having to work super hard at it. That said, prompts still do matter and the models are very controllable with prompts. And so, you know, you we still find we'll launch something and we'll find that it's not behaving in certain ways, the way we want it to, and we can adjust it with a prompt. A lot of times you don't need to go back and like retrain the model. So it's both that I want to make it less necessary over time and that it is still a powerful vector.

• 29:18Azeem Azhar?

  Well, these are two of the vectors of the direction of the product. But the third one is the idea of agents and what agents will bring to us. I think probably the first agent product that OpenAI launched that I used was Deep Research. And Word Agent is being thrown about quite a lot. I mean, I'll use Word Agent and what I mean it mean is I've strung a bunch of prompts together through your API. And there's a bit of logic to move, you know, a document through a series of steps to the other end. What does an agent mean to you?

• 29:51Speaker 00

  We think of an agent as something that can do independent work. So it's not just a quick, you know, you ask a question, you get an answer, but it's actually off doing tasks for you in the real world. So another, I think Deep Research is a great one where it's off doing, you know, hundreds of searches and putting together a complex report for you that might have taken you a week. I think another is Codex, which is our software engineering agent that we just launched. You know, what you can do is if you have a code base that you're operating against, you're, you're building a new feature in a code base or debugging something. You can just give this agent the prompt, like, hey, I need you to fix this thing. I want you to do this to the background of my webpage. I want you to, you know, build this new feature.

• 30:33Speaker 00

  And it will go and look through your entire code base, understand all of the context. If you're, you know, fixing a bug, it'll go try and figure out where that bug exists. And then it will write new code for you and create a pull request, like a diff, you know, here's the set of changes we need to make to the code. And then you can go review the code and the, the agent did the whole thing. And so, you know, I used to be an engineer. I still write code a little bit in my spare time, but I haven't written a single, you know, line of code for OpenAI. But with Codex, I was, this was like, you know, a few days before it launched. I was, you know, I was like 11 at night or something. I was doing a bunch of work that I had to do before I could go to bed. And I was like, you know what? I bet I could fix a bug right now.

• 31:18Speaker 00

  And so went and found a bug that looked relatively simple and just, you know, pasted the context into Codex, said, can you go off and fix this bug? By the way, it was in a language that I had actually never worked with in my life. So it would have taken me even more time if I had to do it myself. And 10 minutes later, I had a pull request. It looked reasonable. I submitted it and actual legitimate engineer looked at it and said, yeah, this looks right. And, you know, now there's a few lines of code shipping today that, you know, came from me using Codex. It speaks to the power of this thing when you, you can just have this software agent off like actually solving real world tasks for you. And in the meantime, I was like writing email and following up on Slack and, you know, doing all the things that I do in my day job.

• 32:03Speaker 00

  So it was just purely additive, which I think is really cool.

• 32:06Azeem Azhar?

  Yes, because the Codex process takes a little bit of time. It has a lot of material it has to read and understand and, and then, and then make the changes. And, and I'm curious about, this is a question that everyone who's built a product that does code automation or developer augmentation is asked is, so today, what portion of the OpenAI codebase is, you know, in the first instance produced by Codex rather than by a human engineer?

• 32:35Speaker 00

  Yeah, it's, uh, it's pretty meaningful and it's increasing quickly.

• 32:38Azeem Azhar?

  Right, okay. Somewhere, somewhere in the meaningful, I'll go up and ask O3 what meaningful means in percentage terms, uh, and I'll get a good, uh, good distribution.

• 32:48Speaker 00

  The cool thing is you can fire off 10 of these tasks at once, right? So it, we try and actually give you the value of all this parallelism where, you know, it's not just you can do one thing, but if you have a Codex agent working for you, why not have 10 Codex agents working for you on 10 different tasks? And by the way, just to connect it to the previous topic on evals, this is also, evals are, are. There's a really important subtlety to them too, where they have to be tailored to the product that you're trying to build and the problem that you're trying to solve. Where coding isn't one thing. Coding is a small vertical of the entire world. But even within coding, you can be good at lots of different kinds of coding. And with Codex, that was a great example of going and saying,

• 33:32Speaker 00

  okay, what kinds of coding really matter to us? What kinds of tasks? And all the tasks that a developer does, what kinds of tasks do we really want to be good at? And we created evals for those. And then we made sure to monitor as we trained the model, is it getting better and better and better at these? And you go and accumulate tasks and examples for the model to learn from. But you do it against a specific set of evals that correspond to a specific set of problems you want to solve.

• 33:58Azeem Azhar?

  It's very capability driven in that respect. And then it speaks to how do you actually do enough testing, both to make sure that you are getting to the right level of score that you want, but also making sure that it doesn't go off the rails. And I think that as these agents get more and more complex, given more complex tasks, that's something to bear in mind. I mean, in one of my workflows, which was a very, very simple one, where I wanted an agent to go through and grab some data from a series of webpages and populate an Excel spreadsheet. And I was using some third-party agent framework. And it was so diligent, Kevin, that it said, I must check my work,

• 34:43Azeem Azhar?

  which it did about 400 times and left me with a $75 bill. And it had got it correct the first time, right? It got it right and got stuck in this strange loop. So that's one of the things I think that I hear when I go around, people saying, well, how are we going to be able to control these things? Not from a humanity out of control measure, but from a enterprise reliability. How can I make sure that this isn't like the sorcerer's apprentice and the thing runs out of control? When I simply asked it to book one flight to Italy and it's booked me 200. And how do you test for all of that?

• 35:22Speaker 00

  Yeah, I think part of this is about making sure that, like we talked about earlier, that the user is in control here. So you should be able to, at some point, be like, hey, you know what? You've checked enough. You're good. And the other interesting thing in all of this is... The technology is evolving so quickly, like much more quickly than I think we're used to with technology. We're used to things taking like decades to deploy and to really achieve scale. One of the the phenomenons you see with AI technology is there'll be there'll be some benchmark, some eval that we're that AI just, you know, can't crack. And people are like, oh, AI just can't do that.

• 36:04Speaker 00

  And and then one day, somebody ships a model that gets like five percent on that eval. Still mostly can't do the job, but just like begins to get it. And then what you inevitably find is like, two months later, there's a model that's at thirty on that eval. And then four months later, there's a model that's at sixty. And then, you know, within six months, it's completely saturated and like models are great at that new skill and will forever be. And so you go very quickly from like, proof of existence to like, oh, yeah, of course AI models can do that. That like rate of of development is is still, I think, something that we're not totally used to.

• 36:47Azeem Azhar?

  It's that first one or two percent, right, that becomes hard, that proves it can be done. It's the Kitty Hawk Flyer. And then within thirty years, we're moving large numbers of passengers across the Atlantic. But in this case, it's within thirty days. So I want to ask about coding and coding agents. So, you know, if you look at the growth of gen AI applications and Similar Web had some data come out a few weeks ago, you know, the baseline was that the generalized chat bots are growing at twenty five percent per quarter. That's, you know, the chat GPTs and so on. Virtually every other product category, image generation, video generation, sound generation, is growing slower than that or declining in size.

• 37:29Azeem Azhar?

  And I view that as the black hole that is the capability of your core models. The one category where growth was faster, seventy five percent a quarter, according to Similar Web, was in coding. I was curious, have you selected coding from a kind of commercial perspective because you can really see the demand and developers are always willing to experiment? Or do you select coding because being a a testable, structured, verifiable set of outputs, it's a slightly easier challenge than a sort of fuzzy amorphous tasks that occupy the rest of the world?

• 38:06Speaker 00

  Yeah, it's a really good question. And actually, coding is this vertical that kind of hits all of these things. You know, for one, it's really important to us because if we can speed up coding, if we can make, you know, every engineer more effective, we also make ourselves more effective. And so we can build even faster and we can bring AGI to the world faster. So it's interesting to us from that perspective. It's a clear kind of milestone or step on the way to AGI itself. Because it's a very sort of general purpose reasoning. It's also a relatively gradable task. Like you can tell, like, like in math or other things, if you get the answer right. It's also something that our engineers are familiar with. So it's a problem space that they understand and have good intuition for.

• 38:50Speaker 00

  It's also a huge market, as you were saying. It's also a market full of early adopters. You know, technologists leaning into this. It's also, you know, relatively sort of open and unregulated. It's not like trying to go into health or something where, you know, there are all kinds of other things you have to do. And so it's this, this aggregation of all of these interesting things that make coding a really interesting market. And I haven't seen that data, but I totally believe it.

• 39:16Azeem Azhar?

  And would you say that within coding, are you already seeing signs of serving people like you who are not technically engineers anymore? In other words, we're seeing that expansion of the market through, through these tools?

• 39:31Speaker 00

  Oh, yeah. And I think there's going to be so much value in democratizing coding out to the world. There's like, what, 30 million developers or something worldwide, depending on how you define it, which is great. That's a lot of, that's a lot of people. But imagine if a billion people can write code. I was talking to somebody the other day who was just telling me they were, during COVID, they were working for their local county, trying to get, you know, vaccinations and stuff out to people. And they were trying to put together a website to track so people could like sign up and just do basic stuff. And the whole world was busy and they just, they couldn't do it. They couldn't create a website. They didn't have the skills to do it. And as a result, they were, you know, managing things less efficiently,

• 40:14Speaker 00

  doing a bunch of manual work at a time when everybody was slammed. And he was just saying, like, can you imagine if I had these tools? We would have been able to, to create a website overnight. It would have just worked. And, you know, they would have been able to do their work more effectively. And you have that times. A million as you look around the world. And so, I mean, that that is actually the other thing about coding that I think is super interesting. That's maybe like the, you know, ninth reason why coding is a good. It is, it's such a general purpose technology. If you can create code, then you can create all kinds of things. And so there there's some, something really powerful to the idea that a billion people might be able to write code.

• 40:57Azeem Azhar?

  But it it also speaks, though, I think, to how this may fundamentally change the the software industry in the way that the internet changes the software industry, not just because of packaging and distribution, but the way we, you know, we interacted with, you know, social technologies, right? Like, my Microsoft Word, when it was on floppy disk, never allowed me to sort of exchange notes with with somebody else on like Google Docs. One of the the big questions that's out there is, you know, as a platform company that is building the most performant models out there, how much space do you leave for for startups? You know, I remember when Microsoft introduced disk compression in Windows ninety five or

• 41:40Azeem Azhar?

  ninety seven or something, and there were a whole load of third party companies that offered disk compression that essentially went out of business there and then. And that this is something that happens on X every time you release a new foundation model. It's like every time Kevin tweets, another fifty startups die. Where is there space, right? Where is there space in the software world, in the AI software world, which is safe from the increasing capabilities of foundation models that that you and other companies are building?

• 42:08Speaker 00

  So Stephen Sinofsky told me an interesting story about this one time. Stephen used to run Windows at Microsoft and Office and, you know, everything. And he was telling me this story about like the transition from Windows ninety three to Windows, or whatever it was called back then, to Windows ninety five, Windows three point one, maybe, where it was just like the beginning of the internet. And and so most people weren't using the internet. And if you were going to actually get on the internet with Windows three point one, you had to like go to some, you know, University of Oregon professor's website and download a TCP IP stack, compile it yourself, and and like, you know, install some device drivers, and then you could actually go on the internet.

• 42:50Speaker 00

  And then in Windows 95, of course, the internet was happening and they were like, "Okay, we need to ship this stuff with Windows." And so they did. And it was like you were saying, there was this, you know, there were a bunch of people who were like, "Hey, now that, you know, you've just like put that university of whatever professor, "he did all this work and now you just shipped it, come on." And Stephen's point was that you would never want to live in a world where today, "you still had to go to like some professor's website and download a TCP IP stack "and compile it yourself to get it going." You just want to use the internet. And basically, the expectations of a platform, "the consumer expectations of a platform are an increasing function of time."

• 43:31Speaker 00

  And if the platform can provide more of the technology that, you know, "if you see something where in order to build the actual thing that people want to build, "you've got 10 different companies having to go build the exact same piece "of like foundational infrastructure, you should probably just provide that." And then, and then those 10 companies can go do like more interesting stuff. And I've always remembered that story. It's really stuck with me because I, I think the fact that people are just going to expect "more and more out of these platforms is, is very real. But the upside is all for third parties in this, for developers in this world. Because if the platform provides more of the building blocks, "then they can spend less time re-implementing the wheel on these building blocks

• 44:16Speaker 00

  "and more time doing the thing that they actually, you know, uniquely add value in." And AI is going to change absolutely everything in our life. Any industry, any vertical, any geography that you can imagine, AI is going to touch. And so I just, I, there's so much opportunity for developers to reinvent and reimagine. I, I think anything that we can do on the platform side to help accelerate that "by making the more of the building blocks easy, we should be doing."

• 44:43Azeem Azhar?

  So let's imagine my, my son, 20 years old, "and he wants to build a, a product on top of OpenAI. Where is a good place for him to, to go and build it?

• 44:55Speaker 00

  I mean, almost everywhere. There's so much opportunity. Sam said this one time and it stuck with me. He said, if you're, if you're building a, a company and you're building at the frontier of, you know, the model capabilities. If you're building something that really just barely works and you can't wait for our next model because you know it's going to make your product sing, then you're probably building in the right place. Because you're, you're introducing something new to the world. You're like making something possible that wasn't possible before. And that's where you want to be. If instead you're building like some sort of scaffolding around that covers up the weaknesses of a current model.

• 45:37Speaker 00

  And you're actually afraid of our next model because it might not have those same weaknesses. That's a bad place to be building. Because on average, models are going to improve really fast and what's a weakness from one model will not be a weakness of the next. So like the, I think the thing to be building is like what we talked about at the beginning. Reimagining use cases from first principles, building them from the ground up with AI. And if you're in a place where you're excited about the next model that comes out because that's what's going to make your product sing. That's a great place to be.

• 46:07Azeem Azhar?

  That's a fantastic heuristic, uh, which is actually if you're a founder out there. Think about something that you want to build that the models are not yet capable of, but will be capable in a little bit of time and you can build on top of that, that capability. We can't talk about products without talking about your new product buddy, Johnny Ave. So tell us a little bit about what the mood in the office was when that lovely black and white photo was released.

• 46:32Speaker 00

  Oh, people are incredibly excited. I mean, how could you not be like, I use products that Johnny designed like my entire day. He's been a part of building some of the, the most sort of cherished products and hardware that we use every day. How could you not want to work with him? And, you know, getting to know him through the process and, and other things. He's also such a lovely human being. For somebody who's accomplished so much. He is so humble, thoughtful, kind, soft spoken. And, and, and then, you know, he'll say something sometime and you'll be like, oh my God, like that is a completely different way of looking at the world and that opened my eyes to something that I just had never thought about. And so

• 47:15Speaker 00

  is this combination of like genius and also wonderful human being.

• 47:20Speaker 00

  How could you not be excited to work with him?

• 47:22Azeem Azhar?

  And of course, he's a Brit. How is he gonna work, his group and your group gonna work? How are you guys going to to to interface?

• 47:30Speaker 00

  Well, I mean, he's he's coming in focusing on the these sort of consumer hardware products. And then also, over time, I think will play a very significant role in design as a whole at OpenAI. And again, I'm very excited about that. How could you not be? Johnny Ive coming in to to take charge of a lot of your your design, you know?

• 47:51Azeem Azhar?

  You know, he'll tackle the the dropdown, hopefully, at at some point. It's like we've got the I've touched that. And so I think that it's interesting to talk a little bit about hardware and how that interacts just in the last couple of minutes with the overarching vision. So the overarching vision, in a way, you you start at the beginning. You talked about AI systems that will act a little bit like employees. I guess for people in their domestic lives, that's more like like helpers. We don't generally have many.

• 48:20Speaker 00

  We think of it as a as a super assistant.

• 48:22Azeem Azhar?

  Like a super assistant. And and so what is the relationship between that and the need to have a hardware device alongside? I already have a hardware device. It's pretty good. I'm I'm talking to it right now.

• 48:35Speaker 00

  It's more the opportunity. As we've said a few times, AI is going to touch every part of our lives and every part of our days in every part of the world. And that means, I think, that there's an opportunity to reinvent and reimagine a lot of the the services and the products that we use every day. You know, in some cases, there are a lot of products that that I use every day that are great. They probably need to fundamentally change, and they should with AI. And if they're not, they're not taking advantage of all of these amazing new capabilities that we have, especially where those capabilities are going to be in 12 months, 24 months, 36 months. So I think there's an opportunity to reinvent and reimagine here. And that's true both on the software side and on the hardware side.

• 49:21Speaker 00

  So, you know, we have we have some thoughts about how that might occur. Obviously, Johnny has thought deeply about this, and we're excited to see what we can build together. And I'm sure there will be lots of others building in the space. And it's one of the reasons that we have, like, we put so much care and attention to our APIs and our developer platform. Because, you know, the world is not just OpenAI. The world, there's going to be incredible startups and incumbents and everybody else building really cool things using AI. And we'd like to power it in any case. You know, some of these will be first party products that we build and some will be other products that others build that leverage our models. And, you know, both of those things are really important to us.

• 50:01Azeem Azhar?

  I mean, I hear that. I hear what you're saying as well, because I've already started to realize the limitations of the phone as the, as a form factor for working with the models. You can't really put in a, a longish prompt to O three. I'm really reliant on, on talking to it. And if I'm in a noisy place, that won't work. The idea of having an ambient intelligence around me, I always have an AI model listening into my meetings and I'm talking to them regularly to do my work. So you start to see the limitations of, of something that's got the power draw of the phone and the size of the phone and, and, and does other things. So that will be, that'll be a really exciting opportunity. And please sign me up for the alpha test well before GA. We've got a couple of minutes. I just want to throw out some questions.

• 50:44Azeem Azhar?

  How far behind are the top Chinese AI firms in core foundation model capability?

• 50:49Speaker 00

  Not as far behind as they used to be. And I think as US AI labs, we need to be very cognizant of that. I think it's really important that the, the leading models, the models that we all use are models that are built off of democratic principles, not authoritarian ones. And we take that really seriously.

• 51:06Azeem Azhar?

  Is there an AI app out there, whether it's in China or coming somewhere else, that's not built by OpenAI, that you, you quite like and you like to use and play around with?

• 51:16Speaker 00

  I mean, I think a lot of the, the like video apps are super fun. I also think, I, I also find Waymo magical. That's my go-to example of the way AI is touching our lives. You know, again, like self-driving was like two years off for 10 years. And now suddenly it's here and it works and it's going to change a lot.

• 51:34Azeem Azhar?

  It's absolutely magical. You are a keen runner and I'm curious about whether you have a Garmin or a Sunto and what you would want from your exercise tracker that AI could bring that it isn't today?

• 51:48Speaker 00

  Oh, that's a good question. Actually, I have an Apple Watch that I mostly use. And then if I'm doing like a hundred mile race or something, this doesn't quite have the battery, so I'll do a, I'll use a Garmin. What do I want out of it? I think actually, one of the things that I would love is better coaching, just like a little bit. And I don't, I think the AI is totally capable of doing it. I think Strava has, you know, some things that they're working on around this. But I would love to see better, better coaching and like, you know, AI analyze, you know, workouts and, and things like that. So that I got the ki, I, I think it's possible to get the kind of analysis that you would get from like a professional coach today from an AI for most users. And I feel like that's the kind of thing that, you know, five years from now, we're going to be like, oh yeah,

• 52:33Speaker 00

  can't even imagine when that didn't exist. But it's only sort of peeking through right now.

• 52:37Azeem Azhar?

  Just a little bit, but I do, I do hear you. I think the, the possibility of having that personalized coaching is, is, would be quite sublime. So last question, when are you going to ship AGI?

• 52:48Speaker 00

  We're working on it. Every day we get a little closer.

• 52:50Azeem Azhar?

  Every day. I mean, when will we know? Will we know?

• 52:54Speaker 00

  I think, look, I think it's one of those things that we talked about intelligence being multifaceted. There are already a bunch of places today where AI is way better than a human. And there are places where AI is like laughably worse than a human. But every, you know, every so, every like month or whatever, when there's new models, the baseline creeps up. And more and more things, AI is becoming superhuman at more and more things. And at some point, it's going to be superhuman at, you know, substantially everything and we're going to call it. But it doesn't happen all at once. I think it's not like we go to bed one night and there's no AGI and we wake up in the morning and there's AGI. It's an incremental process of AI, you know,

• 53:37Speaker 00

  getting better and better at more and more things.

• 53:40Azeem Azhar?

  Well, with that thought, Kevin, you keep climbing that hill. Thank you so much this morning for giving us your time. It's great to have you.

• 53:47Speaker 00

  Thank you for having me.