OpenAI Codex Team: From Coding Autocomplete to Asynchronous Autonomous Agents

Hanson Wang and Alexander Embiricos from OpenAI's Codex team discuss their latest AI coding agent that works independently in its own environment for up to 30 minutes, generating full pull requests from simple task descriptions. They explain how they trained the model beyond competitive programming to match real-world software engineering needs, the shift from pairing with AI to delegating to autonomous agents, and their vision for a future where the majority of code is written by agents working on their own computers. The conversation covers the technical challenges of long-running inference, the importance of creating realistic training environments, and how developers are already using Codex to fix bugs and implement features at OpenAI. Hosted by Sonya Huang and Lauren Reeder, Sequoia Capital Mentioned in this episode: The Culture: Sci-Fi series by Iain Banks portraying an optimistic view of AI The Bitter Lesson: Influential paper by Rich Sutton on the importance of scale as a strategic unlock for AI.

• 0:00Speaker 00

  In my opinion, the easier it is to write software, then the more software we can have. Right now, if we think of like, I bet you if we look, pull up our phones, well, you folks are investors. But if you're not an investor, I bet you if you pull up your phone, most of the apps on it are apps that are built by large teams for millions of users. And there's very few apps that are built like just for us and the specific thing that we need. And so I think as it becomes more and more practical to build like bespoke software for people or teams, we'll end up having higher and higher demand of software.

• 0:44Speaker 01

  Welcome to Training Data. Today, we're joined by Hanson Wang and Alexander Embiricos from OpenAI's Codex team for a fascinating look at the future of software development. Codex is OpenAI's series of AI coding tools that helps developers delegate tasks to cloud and local coding agents. Unlike the original OpenAI Codex, which was developed in 2021 to autocomplete lines of code, the latest evolution of Codex can complete entire tasks for you autonomously in the background. The key difference between O3 and Codex is that while O3 is great at competitive programming, Codex has been RL tuned to be great at day-to-day enterprise development tasks. Alexander and Hanson share more about the backstory for Codex and the broader paradigm shift from snappy autocomplete to longer-running background agents.

• 1:30Speaker 01

  Plus, they share their surprising vision for how developers will interact with AI in the future as sync and async experiences merge. Hint: It might look more like TikTok than your current IDE.

• 1:42Speaker 02

  Thank you, guys, for joining us. It's wonderful to have you here.

• 1:45Speaker 00

  Hey, thanks for having us.

• 1:45Speaker 03

  Great to be here.

• 1:46Speaker 02

  We'd love to hear a little bit more about what you guys work on. Tell us about the Codex team in your story.

• 1:51Speaker 03

  Oh, yeah, I'm Hanson. I'm one of the researchers that helped train the Codex one model.

• 1:55Speaker 00

  And I'm Alex, the product lead.

• 1:57Speaker 03

  I think for me, the name Codex is such a great callback to the original Codex model. That was like, kind of like a aha moment for me when it first came out. Because I think GPT-3 was really cool. But then Codex was like the first moment where I felt like, it's like, wow, this can really do something that is going to change the world. And it's actually kind of like how I got into the whole like startup space. Like, one of the first couple of demos I did was like using Codex to do data analysis. I think it's actually like a funny story, like I was here for as part of Sequoia's ARC program. That's how I met Lauren. And then when the demos we did, we actually used OpenAI Codex to do data analysis. And that's how I started in the startup space. And I think as time went on, as like the later versions of GPT came out,

• 2:42Speaker 03

  became super clear that, you know, like using AI for agentic use cases was going to be the future. And so I joined the company to work on on agentic coding efforts.

• 2:53Speaker 00

  Yeah, and this was like, you know, a per standard OpenAI style where we like the naming to be as easy to follow as possible. This is the Codex of like, I think it was 2021?

• 3:01Speaker 04

  Yeah, this is pre ChatGPT, right?

• 3:02Speaker 00

  Exactly, yeah. So it was actually like the model powering GitHub Copilot. And then recently, you know, as we were working on this product, which we'll talk about, we thought, you know, this is like a super fun brand, also a very apt name, you know, code, code X, code execution. So like, we decided to sort of like resuscitate the brand and like keep using it.

• 3:17Speaker 04

  You said resuscitate, so was Codex dormant for a while and then you all resuscitated it for the?

• 3:21Speaker 00

  We haven't used the brand like recently.

• 3:23Speaker 04

  Okay, okay, really cool. Can you tell us a little bit about Codex, the agent, and what it does?

• 3:28Speaker 03

  Yeah, I think so. Basically, Codex is a coding agent that has its own container and its own terminal, kind of like fully in the cloud. You you give it a task and it comes it comes back to you with a with a PR in this sort of like one-shot style. And we actually experimented with a lot of different form factors kind of along the way, but kind of in the end decided to to settle on this one.

• 3:48Speaker 00

  Yeah, so like, you know, we've been working on a bunch of agents and we've been working on a bunch of coding products as well. And basically, in our mind, Codex is like this thought experiment for how would it work to code with AI? But where we sort of put all our effort into thinking about what would that would feel like if the AI is working on its own computer? You know, independently from you. And so you're delegating to it rather than like pairing with it. And so, you know, some of the things that we're really proud of with this Codex launch are thinking about like the compute environment and like, how do we set it up so that the agent can actually work on its own, but be productive? And like creating the model, which I have to talk more about, like basically that isn't just good at like writing code that looks good or is functional, but also is really good at writing code that like is useful for professional software engineers and like mergeable ideally without even touching your their own computer.

• 4:33Speaker 02

  So what is the difference in Codex and Codex CLI?

• 4:36Speaker 00

  Yeah, we've definitely gotten some questions about that. I promise this is all going to make even more sense over time. So basically Codex for us is like our brand for like agentic coding. And we have this vision of like, you know, like we're going to have this agent and mostly the agent will work on its own computer. But it shall be able to meet you in like any of the tools that you use, wherever you work, be that your your terminal or, you know, your IDE or your issue management tool. So Codex CLI is basically like Codex like in your terminal. So like CLI stands for command line interface, right? So it's like in your terminal, you can work with Codex. That's like your environment. And then Codex or Codex in ChatGPT is basically Codex working on its own computer. Today, those are, those are just distinct things.

• 5:17Speaker 00

  As a brief aside, one of my favorite things about working at OpenAI is how willing we are to cut scope and just like launch things quickly. But over time, we'll actually bring those things closer together. So you can really think of it as just like, it's just like Codex and, you know, it can be in ChatGPT or it can be in your CLI.

• 5:31Speaker 02

  Very cool. And so what did you have to do differently for the model to make it useful beyond just writing the next line of code?

• 5:36Speaker 03

  Yeah. So I think one of the most interesting progressions. So if you go back to, you know, like the O one, the first reasoning model that we launched. We highlighted like how good it is at math and even like coding competitions. Like as of now, I used to be a competitive coder and like it's better than me, a competitive coding. It's better than most, almost all people at OpenAI at that. But I think one of the things that we saw was that, you know, despite being good at these programming competitions, it wasn't actually that good at producing mergeable code. And so like we even highlighted it this in the blog post with with models like O three, like the the code that it generates often, you know, like isn't quite to the taste. Or style that a professional software engineer would expect.

• 6:19Speaker 03

  So a lot of the effort that we spent on training this model was aligning the model to basically like the taste or the preferences of professional software engineers. And that's something that took a lot of, I guess, specialized training.

• 6:32Speaker 00

  Yeah, I have this like very like producty analogy that I like, which is like, if you take our like recent models, which are great at coding, they're great at coding, but it's kind of like this like really precocious like competitive programmer like college grad, who doesn't have many years of job experience being a professional software engineer at like on a team. Right. And so a lot of the work we did to go from like O three to like Codex one was actually like the equivalent of like those first few years of job experience where it's like, hey, like, what does a good PR description look like? You know, PR titles, like, how do you read the style of the code base and then make sure your code is in the same style? How do you like test well? How do you show that you tested well? Stuff like that.

• 7:07Speaker 04

  What's typically the aha moment for when somebody uses Codex?

• 7:09Speaker 03

  Yeah, I think one of the things we have in the onboarding is like find and fix a bug in the code base. I think that's one of the areas where Codex really shines is like specifically like bug fixing. Just because it can actually like independently try not just to see if, you know, something looks a bit off, but it can actually go and then like verify that, okay, like I can try and reproduce a particular issue. And so I think like even, you know, like leading up to the Codex launch, there were a couple of bugs where, you know, like we were sitting there kind of like wondering what's going on. And honestly, like sometimes the easiest thing to do is just like

• 7:45Speaker 03

  paste in a description of the issue into Codex. And we were surprised how frequently like that would actually end up with a usable fix.

• 7:51Speaker 00

  Yeah, like, like fun story here. Hopefully this doesn't give away too much, but at 1 a.m. the night before launch or the morning of launch. At 1 a.m. we were, we were looking at a bug with like an animation, a lot of animation. And, you know, this is the kind of thing like, okay, I guess we could like cut it from launch scope. It'd be okay to launch without it, but we really wanted to get in and we just couldn't figure this out. And so an engineer ended up like describing what the bug was and putting it into Codex and actually like a fun pro tip for anyone who's like using Codex is that if there's a really hard task, it can be useful to ask Codex to like take multiple cracks at it. So they pasted that description in and ran it four times. Like, hey, there's this bug. We can't figure out what's going on.

• 8:27Speaker 00

  And three of those rollouts did not work. And then one of the four, which is like the fix of the bug that we were stuck on for like hours at 1 a.m. before launch. And so landed the fix, you know, deployed the code. And the animation was in for launch.

• 8:39Speaker 04

  That's awesome. Maybe tell us more about how y'all are using it internally at OpenAI. Like, is every engineer, is every researcher using Codex now in their workflows?

• 8:47Speaker 00

  Yeah. And actually, can I give you the other like kind of magic moment?

• 8:50Speaker 04

  Oh, yeah, please do. Definitely.

• 8:51Speaker 00

  So like, one of the interesting things about Codex is that it's a very different form factor from maybe what people are used to, right? Like, a lot of the AI products that people are used to, especially in software, maybe like GitHub Copilot was like the first really good one. There are really things that kind of like work with you in flow and you're just kind of seamlessly going back and forth. You're kind of pairing, right? And it's flavors on pairing. And we think that's awesome. And like the Codex CLI is a tool that you can use in that way. But for Codex, you know, we really wanted to push this idea of like you're delegating because like in the future, we imagine that actually, you know, the vast majority of coding is actually going to be done independently from like the human like working on their computer who can only do one thing at a time. And so, you know, it'll be done by basically it'll be done by agents working on their own computer.

• 9:37Speaker 00

  And so that is a very different thing to delegate to an agent than it is to pair with sort of an AI model that's like in your tooling. And so you have to kind of use it differently. And so when we actually were working on an alpha before launch, we would just give this agent to people and be like, hey, like just use this however you want. And we noticed that many, many of the people trying to use our alpha of Codex were just like not really finding it super useful. And then we're like, huh, that's interesting. Let's look at how people like at OpenAI are using like internal tooling like Codex. And we realized there was like a big difference, which is the mindset of using it. The mindset that works really well for Codex is like kind of this like abundance mindset and like, hey, let's try anything. Let's try anything even multiple times and like see what works. It saves me time.

• 10:22Speaker 00

  And so we've kind of shifted the way that we even onboard people into the product to try to create this aha moment, which is running many tasks in parallel. So like for us, if we see someone like trying it out and like they've run like 20 tasks in like a day or an hour, that's amazing. And like we, they're probably gonna like, they've understood basically how to use the tool.

• 10:40Speaker 02

  Fascinating. How does that change the role of the human when you have to review all of this code? Like if two of the three work, then what do you do?

• 10:47Speaker 03

  Yeah, I think we've put a lot of focus on also like making the outputs easy for people to review. So like one of the things that we're proud of is like, uh, we haven't seen this in too many other tools is like the ability for the model to cite its own work. So not just, uh, like the files that have changed, but also even like the terminal outputs. So like if it ran a test and, you know, for some reason test wouldn't work, it actually like tells you that and it tells you like, here's the exact kind of like terminal command I ran. Here's the output. Makes it much easier to verify the outputs. Uh, but, but it is like a great point. I think we're shifting to a world where like a lot of the time that we spend, you know, like normally coding, a lot of that's going to shift to actually reviewing this, reviewing the code.

• 11:28Speaker 04

  Do you need humans to review the code? Cause I think of code as one of those things where, you know, it, it compiles or it doesn't. And once it compiles, you can go and check if it does the thing it was supposed to do. Like, do you even need humans to do the code review?

• 11:38Speaker 03

  I, I think, yeah, I mean, for the foreseeable future, at least I do think, I do see that to be the case. I mean, I think a lot of it's also just like building trust with, with the early users. I think people really need to have a feeling for like, you know, what things, um, are working well, what things are not. And I think there's always just like some external context about like, you know, what makes, uh, this code correct. That, you know, might be beyond what, what you initially provided as context.

• 12:05Speaker 00

  Yeah. Like if you think of what, you know, what a developer does, and this is obviously oversimplifying, but there's like, okay, there's coming up with what things maybe should be done, discussing them with the team, maybe deciding what to do. You call that ideation, you know, maybe then there's design, like, okay, what are we actually doing? And then like planning, how are we going to do it? Then there's implementing and then validating, you know, testing those changes and that, you know, that's basically a loop. And that, that small loop of like implementing and then testing is what Codex is great at right now. Although we can talk about how you can use it for planning too. Um, and then there's actually deploying the code and then maybe maintaining the code, writing documentation, et cetera. And so like, you know, I, I forget the exact stat, but I feel like the stat I remember recently is like engineers spend like maybe like 35% of their time coding. It's not actually the majority of even what engineers do.

• 12:48Speaker 00

  And so, um, you know, the, the future that we're trying to build towards is that is one where, you know, if you're a software developer or even like in any profession, All the work that is like easily automatable, that's usually the grungier type of work, you're not doing, you're delegating that. And then the work that is more interesting because maybe it's ambiguous or maybe because it's really hard, that's the work that you're driving. So we're trying to build towards that, that work, that, that world. And I think we have to get there iteratively. So for example, right now, if you're a human and you write code, another human's going to review that code, right? And so we're not going to come in and just like try to change that. And we're like, okay, let's plug into that. So you know, the way the product works right now is like you, the developer are being accelerated by the tool. You ask for some code to be written. You decide if it's good and you want to push it out to your team and then your team can

• 13:33Speaker 00

  review it. And then over time, we'll basically kind of expand what we can do. So we'll help more and more with like planning, maybe even designing, maybe even thinking about what to do in response to things that are happening in your app or at work. And then we'll push to like make review easier and easier as Hanson was describing.

• 13:49Speaker 03

  Yeah, I do think I see a future where you have, you know, like multiple agents collaborating together. So you have, you know, Codex, the Codex agent writes the code and then maybe like the operator agent's the one that's testing it and all of the things that, all the different agents that we've been working on at the company can kind of like come together.

• 14:05Speaker 02

  That's awesome. Have you seen people, now that you can delegate doing, writing code, people beyond engineering teams start to use Codex? And as we get into the world of Vibe coding, you guys are helping us bring us further down that hole?

• 14:16Speaker 00

  Yeah, this is actually super funny. We were, so the answer is yes, but I'll tell you a story. We were working on our launch blog post with Lindsay here and we were talking about like what quotes to, you know, to quote from customers. And we had a customer that wanted to say, yeah, like we on the engineering team love this and also it's like a power tool for PMs. And I remember looking at that quote and be like, this is a really cool quote because I'm on the product team and I use it to just like avoid having to bug an engineer about things or to answer questions. But I remember looking at that quote and being like, do we want that in the launch blog post? Because the target audience for what we're building is like specifically professional software engineers, not Vibe coders. So I think we ended up not including that exact line, but I think over time, like as

• 15:01Speaker 00

  You know, as we have agents that can help us code, I would expect more and more people to be able to contribute to codebases. Yeah.

• 15:06Speaker 04

  Do you think the number of professional software developers goes up or down over time?

• 15:10Speaker 00

  This is just my opinion, but I think it goes way up.

• 15:12Speaker 04

  Huh.

• 15:12Speaker 00

  I think.

• 15:13Speaker 04

  And not vibes coders, professional software developers.

• 15:16Speaker 00

  Yeah, I think so.

• 15:17Speaker 04

  Okay.

• 15:18Speaker 00

  But, you know, yeah, in my opinion, the easier it is to write software, then the more software we can have. Right now, if we think of like, I bet you if we look, pull up our phones, well, you folks are investors, but if you're not an investor, I bet you if you pull up your phone, most of the apps on it are apps that are built by large teams for millions of users. And there's very few apps that are built like just for us and the specific thing that we need. And so I think as it becomes more and more practical to build like bespoke software for people or teams, we'll end up having higher and higher demand of software.

• 15:46Speaker 03

  Yeah, as I think about how I use it, I think it just really is a multiplicative factor right now, rather than any kind of, any sort of replacement, just like, especially looking at the patterns of our internal power users, it's like a really dramatic like difference in like the top users of Codex are like doing, you know, 10 plus PRs every day. It's just like really such a multiplicative factor that I can't see like a world in which like, it's like lowering the bar to creating software so much.

• 16:14Speaker 00

  That said, I mean, I think this is a really important question, and to be completely honest, like, we don't know. And so this is something that we, as a company, pay a lot of attention to.

• 16:21Speaker 04

  I want to talk a little bit about the, you know, what's happening under the hood on the technology side. As you mentioned that, you know, the model itself, one of the things that makes it different from competitive programming is you've, you know, you've made it more, you know, be good at the things that a professional software developer would do. Is that the biggest difference on the model side? Or is like, should we think about it as a, you know, as a close cousin of O3?

• 16:40Speaker 03

  Yeah, so it's definitely the same model as O3 with additional reinforcement, fine tuning. But that said, yeah, I think so. Part of it is kind of like these more like qualitative aspects of what makes a good software engineer versus simply like a good, let's say like coder, you know, like style, even like how it writes comments, that's, I think that's like one of the things that people have noticed with other models. And then on top of that, I also want to highlight one of the big challenges was like making good environments. For the agent to kind of learn in. And so if you think about like real world software repositories, it's like so varied and complicated.

• 17:22Speaker 03

  Like, think about like how much DevOps has to go into like setting up a repository. And that's something we're kind of like learning the hard way with the our environment setups, but.

• 17:30Speaker 00

  Should we talk about the multi repo I was showing you yesterday?

• 17:32Speaker 03

  Oh, yeah.

• 17:32Speaker 00

  Like, yeah, I was I was showing Hanson the repo for the startup that, you know, OpenAI acquired. And so we joined and so we were looking at that repo together, thinking about it at first as an environment. And Hanson's like, so like, where are the unit tests?

• 17:48Speaker 00

  You know, because the agent uses unit tests to verify. And I was like, uh, this is a real startup that has no unit.

• 17:56Speaker 03

  I was the same, so I can't complain. So, yeah, like you have all these like really messy environments. So we, yeah, we have to, over the course of training, like, we have to basically generate these really realistic environments for the agent to learn from. And I think like one of the reasons that we're able to make such like an end to end product work is that we have like the same environments that we use during training and the same like basically this, you know, containerization infrastructure that we're using to serve in production. So this, so our users are, you know, like we're running our own compute environments. When when users use Codex, they're running in the exact same environments that we're using to training.

• 18:36Speaker 04

  So you don't have the agent saying, but it works on my machine?

• 18:38Speaker 03

  Exactly.

• 18:40Speaker 04

  Okay, good. I think these are also the longest running agents I've seen out of OpenAI. Deep Research, maybe, was the previous one that was longest running. And my understanding is, you know, Codex can, you know, sometimes spend 30 minutes on on different tasks. Are there any kind of surprising challenges and things you've encountered just getting inference time to scale up on on, you know, query for so long?

• 19:01Speaker 00

  Maybe I'll start with the product side and then there's many on the modeling side. But on the product side, actually, the thing that I think the most about is like user intent. It's like actually. You know, if you imagine someone using like autocomplete in their IDE, it's like not super hard necessarily. I mean, obviously it's difficult, but it's not super hard to predict like, what are they trying to do like right now for the next like microsecond? But for doing a task that takes 30 minutes, it's actually like fairly difficult to help a user describe the task. Like, they may not even know exactly what they want for 30 minutes worth of work. And so something that we spent a while debating, and it's like still a thing we debate is like, what is the right granularity of a task for someone to give to Codex?

• 19:45Speaker 00

  And like, how can we make it easy so that Codex can like be really flexible? Where you can use it for like one line changes? You can use it for like big refactors that you know exactly what you want, or like larger features where you know what you want? Or maybe can you use Codex when you don't know exactly what you want? And so maybe you should ask Codex for a plan, and then you can like have it, Codex suggests tasks, and then like do those tasks afterwards. So that's still a topic of debate and like iteration for us.

• 20:11Speaker 03

  Yeah, I think that's actually like a good pro tip for you for using it. It's actually like really good at coming up with its own plans. And then, you know, sometimes it's really tedious to specify everything you up, you want upfront. And that's kind of like one of the unique challenges about working. Like, if you wanted to work for, you know, an hour at a time, then you kind of do have to specify a lot upfront, which means that you have to spend like, I don't know, like 10, 20 minutes coming up with that. But if you use actually like the ask mode to first like, you know, generate like a high level plan of what you want, want to do. And then you can like iterate on that with the model before you, you know, send it off for, for an hour.

• 20:44Speaker 04

  It really is like working with an intern.

• 20:46Speaker 03

  Yeah.

• 20:47Speaker 04

  What about on the model side? Anything that's surprising in terms of model behavior as it starts to run for so long?

• 20:52Speaker 03

  Yeah, I think, I think our models have gotten a lot better at kind of like sticking, kind of like on task as a, especially like with these longer rollouts. I will say like, there are cases where, you know, like even the, there is a limit to the model's patience, even though it's quite high. So, so it can be frustrating sometimes, you know, it's like, it goes off for like 30 minutes and then, you know, that this is a case that we're working to get better at where it's like, you know, it's kind of like, just like a human who comes back to you, it's like, "Sorry, I don't, I, I, this is too much. I don't have enough time to do this." Actually, like, that's one of the things it says.

• 21:24Speaker 04

  That's hilarious.

• 21:24Speaker 02

  Just like an intern.

• 21:25Speaker 03

  So, very, very human-like, yeah, in many ways.

• 21:29Speaker 02

  I'm curious how you think about the right interaction patterns and how they evolve and how the suite of products around this evolve over time. We have Codex, we have Codex CLI. What else do you think is out there in the design space for engineering and building products?

• 21:42Speaker 00

  Yeah, so the Codex, as we launched it, is really just like, you know, it's a research preview. It's a thought experiment, a useful one, but it's still still very early. And what we're most proud of with Codex is the model and the, you know, the beginning of this like foundation for compute environments. And the UI we shipped is is one that we iterated towards, and there's some fun stories there. But it's definitely not the final form factor. And for those listening, basically, the UI we shipped is an interface in ChatGPT where you can like submit a task and ask Codex to like either answer your question or write code. And then you kind of have this like something that looks a little bit like a to do list of like things that you can like go look at merging.

• 22:24Speaker 00

  Really, I think for, so we built that to really lean hard into this idea of like an asynchronous agent that you delegate to. But what we want to build towards is a setup where you don't have to think about whether you're delegating or whether you're pairing with an agent. And it's really, it should just feel like working with a teammate and where that teammate is like ubiquitously present in all the tools you work with. So you should be able to pull up any tool that you're working in, be it your terminal, your IDE, your issue management tool, maybe your alerting tool or your errors, you know, the tool that shows you errors, and just ask for help. Maybe even Codex is already taken a look before you even got there and it has like an opinion there. And you could be able to ask something, be it a short question or a long question.

• 23:07Speaker 00

  It'll just like appropriately decide how much time to spend before answering you and just like help you land those changes. So basically, we want to kind of blend this idea of like pairing and delegation. But the first thing we shipped was just like the the the purest thought experiment. The other thing I'll add to this is like one of the unique. Things about working at OpenAI is that we are the makers of ChatGPT, which is sort of the AI system that the most people use. And so we don't actually see a future where as you go about your day, you're deciding whether to use like the Codex Agent, or I don't know, you're like shopping agent or like taxi ordering agent. By the way, I'm just like naming random things here. Or you're like marketing agent.

• 23:52Speaker 00

  Actually, the way we think this should work is you should just have like one assistant that you talk to, and you can ask it anything about anything, and it can just like do the things you need. And so that's, you know, that's ChatGPT that will become our assistant. And then if you're a power user of a certain type of tool, so let's say you're a software developer, you spend a lot of time in certain functional tools, then you can go into that tool and have like a bespoke interface with buttons, with lists that you can use to like efficiently go about your day.

• 24:18Speaker 04

  Do you think we'll still use IDEs?

• 24:20Speaker 00

  Yeah, for sure, but they'll evolve, right? Like right now, they're like very focused on writing code. And like as Hanson was saying, like probably agents will be writing more and more code. And so it's going to become like there'll be a shift in emphasis towards like landing code or reviewing code or like validating them. Or maybe even like a shift in emphasis towards planning like bigger arcs.

• 24:39Speaker 03

  Yeah, I think we're already seeing a lot of people on the team. They kind of like first thing in the morning, they come in like they make coffee and then they like kick off a few tasks just to kind of get a starting point. And then, you know, they come back after their breakfast and they they look at the tasks that were the PRs that got generated. Then they'll take those and the IDE is kind of like the place where you take, you know, it's not, it's maybe it'll get you like 80% of the way there, hopefully, or even more. But then there's always this like last mile where you go in and really like fine tune

• 25:09Speaker 03

  based on kind of like your own vibes.

• 25:11Speaker 02

  How do you see the broader market evolving? Like within OpenAI, you have so many different strategies here. And as you think about async tasks, as you think about some of the things that you mentioned moving into ChatGPT, we're seeing a lot, an explosion of other tools and specialized models. You obviously are biased, but I'm curious what your read is of the broader market.

• 25:29Speaker 00

  Yeah, it's a crazy time to be a developer right now. Like there are just so many new tools that are just so helpful. Like a fun story recently is I was in the airplane and there was no Wi-Fi. And I had thought that I was going to maybe write some code and like build a thing. And there was no Wi-Fi and I was like, you know what, screw it. Like it's just not worth my time to like even try to write code anymore. Whereas, you know, the startup that I was working on like many years ago, like part of the genesis of that startup was like me writing some code without Wi-Fi in an airplane. And I just wouldn't even do that anymore because like the market is just like it's just changed so much. And I think this, I think we're going to see like an equivalent shift in an equivalent amount of time. So like in the next two years, coding will look completely different.

• 26:10Speaker 00

  I think right now, most of the tools that people spend, you know, that people find the most value from are tools that work really closely with you, like in your development environment. You know, like basically pairing. And I think the shift that we're going to see, but we have to figure out how this will happen. But the shift that we're going to see is that actually the majority of code will be written by agents. And those agents won't be working in your environment where you can do one thing at a time, but they'll be working in their own environments. And they won't just be triggered by you like thinking of specific tasks, but they'll be connected into the tools you use doing work there. And so I think we'll see basically that shift towards agents. I think we're going to have to figure a lot out a lot about code review as you were asking about like. Personally, like I don't exactly know how that's going to work, but I do know that even already at OpenAI, we're seeing like,

• 26:54Speaker 00

  like much more code is merged by agents, but actually also even more code is generated by agents as folks are like, you know, like say kicking off tasks four times to like choose their favorite implementation. And so it's like not 100% clear how we should, how we should even like manage all this code that is that is being written. Some things that I will say though, in case it's useful to the audience is that there are definitely things you can do to your code base to make it more addressable for agents. This isn't necessarily particularly novel, but you know, obviously using like typed languages is really helpful. Another thing that's very helpful is like having like smaller modules that are like better tested. Like we joke about. Having good tests at all, yeah. Yeah, having tests. Like we joke about my startups repo, but like I bet you we would have written it differently if we were writing it today.

• 27:39Speaker 00

  And even there's like small things like the code name for this project is WHAM. This is the code name for Codex. It's like W H A M. And when we named it, we were very intentional in doing so because we knew we would have code like in the server, like for the website, in various other places. And we wanted it to be really easy for the agent to like search for WHAM related code and find it. And so we named the project.

• 28:06Speaker 00

  You know, WAM and we grepped the codebase first to figure out how often it was there. Like if we would have called it something like code or codex or agent, you can imagine like it would have been really hard for the agent to.

• 28:15Speaker 04

  Would have gotten very confused. And now you've called it codex and now the agent's gonna be confused.

• 28:18Speaker 00

  Well, so in the code, this is kind of my point, right? Like intentional design, like in the code, we use the term WAM like a lot, because that's actually much easier for the agent to find. Obviously, if we didn't use a word like that, the agent could still find its way, but it would have to do, spend much more time to find the right files.

• 28:34Speaker 03

  Yeah, it is, it is cool that, you know, like a lot of the things that actually make the codebase easier for humans too, also tends to make it easier for the agents, like good tests, for example, writing good docs is another great example, where like now I think there's even more of an incentive to, to do that, because like not only does it make your life easier, it makes the agent's life easier.

• 28:51Speaker 04

  Okay, sorry to be the annoying VC, but Claude Code and Jules are also like, I think, agentic coding experiences from, from others. Do you think the mar, I'm, I'm curious how you think your experiences compare today? And then do you think the market is probably gonna converge towards the same vision of what, you know, sync and async coding look like? And, and in that version of the future, what do you think OpenAI wins on?

• 29:11Speaker 00

  I think we're gonna see a little bit of everything, right? Like even in what you mentioned, like there's like tools that are working on your computer, there's tools that are working on their own computer. Like, as I mentioned, like I think we're gonna see the majority of work being written, where like the agent has its own computer, but it will still be really important for us to invest in accelerating developers who are doing work on their own computer too, right? So ideally, we get the best of both worlds there, but most work is done in agent compute.

• 29:35Speaker 03

  I think the way I see it as well is like, I think one of the, the hardest part of software engineering really is like taking all the context from the world and like encoding it in these requirements, these like design docs. And then the implementation, like I think as we alluded to earlier, is like not actually like that much of the lifecycle is spent on like physical coding. And so I think where ChatGPT shines is like, it is this assistant that has, you know, like has memories now, it has access to like a lot of different connectors, to like all the different tools you use. We have like operator, deep research that have all these like different capabilities. And so I think the vision where that like all comes together is where, you know, like a tool like Codex can really shine. Once it has access to all that knowledge, it's able to like make use of that.

• 30:16Speaker 03

  And I think with that, it should be able to do a much more effective job at, at, you know, just the coding part.

• 30:23Speaker 00

  Yeah, like imagine like hiring a software engineer and like the only thing that that software engineer can do is like take a task from you and produce a PR. Right? Or you know, it has like these like very well-defined features and it can exactly do those things. Uh, and then you're you ask for like a random thing like, oh, hey, like the team is getting together. Like, do you mind like also, I don't know, getting a meeting room and like leading a brainstorming? Like, it'd just be it would just be so frustrating if like you hired a teammate and they refuse to do that kind of work, right? And so similarly, I think like it's really like we're building towards a future where like agents that you're working with are a little bit more generalized. Like, you know, to reference like Hanson was talking about, like, you know, operator and deep research, like if you think. Operator has a web browser, deep research has like a different flavor of a web browser, Codex has a terminal, like really like.

• 31:07Speaker 00

  Your teammate has pretty similar tools, like a human teammate, right? And so like the goal for us eventually is to like. Pick places where we want to really invest in a specific audience to like make rapid progress. So we obviously we're doing that with coding with Codex or like GPT four point one. Where we like generated specific evals for that audience and then like made a better model for them for developers. But then over time, like generalize these things into like simple things that everyone can use. So I think like, again, with us, you know, with with OpenAI and like ChatGPT, I feel like that's a place where. The products we build will look very different from something that's like very only specifically for coding.

• 31:43Speaker 04

  What do you think will be the primary UI that developers use to interact with Codex? Like, do you think it'll be ChatGPT, the CLI, the ID, all the above?

• 31:52Speaker 03

  Yeah, that's I think a mix of all the above. I think we just kind of like want to meet developers where they are. In that moment, so it might not even be like in the editor or in the terminal, it might be like on Slack, like, you know. Someone messages you like, hey, like, there's a bug and you're just like, hey, like, go fix it.

• 32:09Speaker 00

  I'll give you my like fun future UI that is like not at all serious. But maybe uh the future of of like working with agents, if you're like, you know, a startup founder in the future and you have like. A team of just you or you and a couple of founders and a many agents actually looks like TikTok. You know, uh, maybe you have like vertical feed and it's basically an agent has produced video. That you can watch with like an idea like, hey, a customer wrote in with this request, I think we should fix it. And then you swipe right to say, uh, like, yeah, let's let's fix this, let's do this, swipe left to to say, no, we shouldn't do that.

• 32:43Speaker 04

  Tinder or TikTok?

• 32:44Speaker 00

  But sorry, it's a hybrid, it's a hybrid. I didn't say this was gonna make a lot of sense. And then you press and hold uh to provide feedback. So you feel like, yes, like do it, but you know, uh, make sure the font is in italic. And so basically, you have all these agents who are like subscribed to information at your company or on your team. And they're proactively coming up with ideas and, you know, doing them and then giving you updates. And you're kind of just like curating the work that is being done.

• 33:09Speaker 02

  And they show you little previews of what the world could look like.

• 33:11Speaker 00

  Yeah, obviously, that's a half joke, though. You know, I think that'll be like kind of the arms length working with agents. And then there's like, it's definitely going to be really important for people to be able to like go do the work themselves and like pair with agents in.

• 33:24Speaker 04

  I get that it's a half joke, but it is like, it's a really cool visual because I think everyone agrees conceptually with this idea of, you know, collaborating and reviewing all the different changes an agent makes is going to look very different from how we code today. But like, nobody's actually given me a visual of what that might look like. So that's a really cool idea.

• 33:39Speaker 02

  I love it.

• 33:40Speaker 04

  Awesome. Should we wrap with a lightning round?

• 33:42Speaker 00

  Let's do it.

• 33:42Speaker 04

  Okay, recommended piece of content or reading for for AI fans?

• 33:46Speaker 00

  For me, that's like immediate, it's like The Culture by Ian Banks. Have you read it?

• 33:50Speaker 02

  Yes, it's amazing.

• 33:51Speaker 00

  Yeah, it is a science fiction series, started being written in the eighties. And it is unusually positive in its view of like how a future spacefaring like human and non-human like race could kind of look. And there's a lot of questioning about like, what is the purpose and meaning of life when we have AGI?

• 34:15Speaker 03

  Yeah, I think for me, it's like anything by Richard Sutton. I think that was like my introduction to reinforcement learning. And I think it's like, it's kind of a joke here that like we read the bitter lesson like every single day. That's like kind of the philosophy of OpenAI. Like, I think, you know, even with Codex, like we give it a terminal and like it literally uses Plazix tools. That's like the most like bitter lesson way of working with with the computer.

• 34:36Speaker 02

  And your favorite AI apps?

• 34:37Speaker 03

  Gotta be ChatGPT.

• 34:40Speaker 04

  Not ChatGPT.

• 34:44Speaker 02

  Okay, either it could be like a new feature that you guys have released other than Codex or something outside of OpenAI.

• 34:50Speaker 00

  Okay, so I guess I don't, it's funny, I don't really think of AI apps, right? But I do like it when my life gets easier. So, you know, some things that I like are like when you're using AI, but it's kind of invisible. So like just, I'm in products, so I often like file bugs. And like Linear has a really elegant integration where when you file a bug from a Slack conversation, it just generates the bug from the Slack conversation. But they never say AI anywhere, just like you actually kind of don't even notice that it's using AI. Oh, wait, I came up with that answer for favorite AI app, Waymo.

• 35:25Speaker 03

  Yeah, I think, I mean, I think for me, like, Copilot has definitely been the, the, the thing that, you know, keeps delivering value every single day for me.

• 35:36Speaker 04

  Okay, robotics, bullish, bearish?

• 35:38Speaker 00

  Bullish?

• 35:40Speaker 03

  Yeah.

• 35:42Speaker 02

  Which new application or application category do you think will break out in 2025?

• 35:46Speaker 04

  Other than coding?

• 35:47Speaker 03

  Yeah, I mean, I think, I think when you had Ysa and Josh on, it's kind of the same answer, but 2025 is definitely the year of agents. I think we're gonna see agents take off in a lot of different categories.

• 35:58Speaker 00

  Yeah, I have to agree with that.

• 35:59Speaker 04

  What type of agents are you most excited about?

• 36:02Speaker 03

  Aside from coding agents? That's a good question.

• 36:06Speaker 00

  Well, I mean, so my take would be like, you know, if we, I know this is meant to be rapid fire, right, but like, kind of the way we think of agents is you have reasoning models, right? And then you give those reasoning models, like, access to tools of the trade. And then you figure out how to train that agent to like, do the, the sort of specific function, right? So it's like, not just about writing, it's about journalism, or it's not just about coding, it's about software engineering, right? So that's kind of what we're doing. And in my mind, we, the reason I'm so excited about agents this year is because we now have a few agents shipped from OpenAI, and you know, and other companies are shipping agents too. And so we're starting to see what kind of the shape of this is, and starting to like, identify the primitives. And so specifically, what I've been excited about is like, as we bring this together and you come up with like, an agent that, you don't have to provision like, separately for every single function, but it's an agent with a computer that has

• 36:51Speaker 00

  a browser and has a terminal, and it can do like, multiple things without you having to like, exactly specify, like, you are my coding agent or something.

• 36:58Speaker 04

  Really cool. Thank you so much for joining us. Congratulations on what you've built at Codex, and thank you for giving us a preview of how you think the coding market will evolve, and also giving us a peek into how, you know, long-running async agentic experiences will play out. Really appreciate it.

• 37:11Speaker 02

  Thank you.

• 37:12Speaker 00

  Thanks for having us.

• 37:13Speaker 04

  Thank you.