OpenAI Codex Tech Lead On How His Career Grew And How He Uses Codex | Michael Bolin

This is Michael Bolin, the tech lead for the open source Codex repository and a former distinguished engineer at Meta. We talked about his career path, how OpenAI engineers use Codex and the difference between research-led vs engineering-led company cultures. 🔸 My keyboard project: https://read.compose.llc/p/our-keyboard-design-reveal 𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀: • YouTube: https://youtu.be/hN5ZFzWFhhg • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/openai-codex-tech-lead-on-how-his 𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀: 00:00:00 - Intro 00:00:56 - Chickenfoot 00:02:45 - Working at Google 00:06:34 - Overhauling Facebook's build system 00:16:36 - Rewriting Facebook's IDE 00:26:01 - Struggles after Principal Eng (E8) promo 00:28:39 - Building a virtual filesystem for Facebook 00:35:47 - Delayed Distinguished promo (E9) and learnings 00:39:56 - Joining OpenAI 00:43:05 - Research-led vs engineering-led cultures 00:44:53 - The story behind Codex 00:51:00 - How he uses Codex 00:57:00 - Why Codex's harness is open source 00:59:50 - Top technical book recommendations 01:05:02 - Why deep technical skills are still valuable (for now) 01:11:07 - How to start projects well 01:14:27 - Advice on writing better and career planning 01:17:06 - Advice for his younger self 01:19:10 - Outro 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗠𝗶𝗰𝗵𝗮𝗲𝗹: • His personal blog - https://blog.bolinfest.com/ • Twitter/X - https://x.com/bolinfest • Threads - https://www.threads.com/@bolinfest • LinkedIn - https://www.linkedin.com/in/michael-bolin-7632712 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The questions that you asked the agent is going to affect the quality of the thing that you get out.

• 0:05Ryan Peterman?

  This is Michael Bolin, tech lead for the open source Codex repository and former distinguished engineer at Meta. And we talked about how his career grew.

• 0:14Speaker 00

  So I was like, all right, Hackathon, totally gonna like make a new build system. You know, relatively quickly, I had something that was dramatically better, like at least like twice as fast.

• 0:24Ryan Peterman?

  He shared about how OpenAI engineers actually use Codex.

• 0:27Speaker 00

  I almost feel a little bad writing code by hand.

• 0:29Ryan Peterman?

  100% of your code, would you say, is human written?

• 0:33Speaker 00

  Yeah, because every time I sit down, I'm like, should I write this? I mean, the answer is almost always no.

• 0:37Ryan Peterman?

  And we discussed AI research versus big tech engineering cultures. What are your thoughts on that research-led culture versus engineering-led culture?

• 0:44Speaker 00

  I mean, it's certainly an adjustment, right? I mean, I think if anyone who comes here and says otherwise is a liar.

• 0:50Ryan Peterman?

  Here's the full episode.

• 0:56Ryan Peterman?

  I was looking deep into your website and there was something, most everything I could find more information about. But there's this one thing that you seem to be pretty excited about at the time, but I couldn't find any information about it because all the links were dead. What is Chickenfoot?

• 1:11Speaker 00

  Oh, man, that's strangely relevant because it's, it was my, my master's thesis project. It was a Firefox extension, probably one of the very few that was written, you know, in JavaScript for Firefox as a thesis project. And it was, it was actually a little coding tool in a sidebar of Firefox. And it was like a little REPL. It was called end user, the idea was end user programming for the web. And so there were functions like enter and click and things like that. And you'd say enter and you'd have to pass a string arg and it would like find the box and you'd say click search or whatever. And, and you know, a lot of the work was all these heuristics that we built under the hood that if you said, you know, enter first name,

• 1:55Speaker 00

  finding the like the words first and name and finding the text box that was closest. And then like using that and then through JavaScript, like using that as the input. And I just think about that now because it's really funny. We did all, you know, a lot of work and that's. You know, a lot of what these agents are doing right now, right? It's like similar, but now actually truly in natural language, like not in this little like JavaScript REPL.

• 2:17Ryan Peterman?

  Oh, interesting. So it parsed the front end and had this REPL where you could say, I don't know, find the the first name field and, you know.

• 2:24Speaker 00

  Yeah, we'd use like, you know, the accessibility tags and alt text for images and we'd, you know, and it worked well. Was really good at Craigslist, actually, because that's one of the simplest websites you could you could use. But I had I had friends who like made like automated tasks and like made real money off of things they automated with this tool, which was really fun.

• 2:45Ryan Peterman?

  When you entered the industry, you were really excited about going to Google and specifically working on Google Calendar.

• 2:51Speaker 00

  Yep.

• 2:51Ryan Peterman?

  And so what drew you to Google and what was that like?

• 2:55Speaker 00

  Yeah, I mean, you know, so I, you know, got online in the 90s, right? I remember, you know, browsing the web and it was at a time you'd go to like, you'd try five different search engines to like, hopefully find the thing that you wanted. And I and it's funny because I distinctly remember my roommate in March of 2000 saying like, hey, there's this site, this search engine that looks a bit better. I think it was still Stanford dot Google dot edu. And I was like, oh, this is better. And like, you know, and then you started reading and they they were just so different, right? Yahoo was all like very cluttered and like they tried to be very sparse and were more, I think, principled about it at that time. And then, you know, like a lot of things, you start seeing people who, you know, who gets a job there and you're like, oh, man, they're they're taking really good people. Like, I want to work with with those people.

• 3:40Speaker 00

  And I felt like they just, you know, really got the web, like especially at that time when like Microsoft didn't around that time, Microsoft killed the Internet Explorer project altogether. And I was like, this is the portal into the web and you guys are, you know, dismantling it, whereas, you know, Google was way more, you know, web forward. And so between that and like the quality of engineering and the and the impact they are having, you know, certainly a place that I was really excited to go to coming out of school.

• 4:05Ryan Peterman?

  And what was the culture like there at the time? Like, I think in some of your writing, you talked about like products versus infrastructure.

• 4:13Speaker 00

  You know, a lot of companies, especially once they get big like that, you know, it's kind of like whatever they were good at first is. I feel like the founders always have a soft spot for, right? So certainly information retrieval and infrastructure, right? Were like key to, you know, growing that that company. And whereas I was drawn there at the time, because like, you know, they put out things like Gmail, right, which were big, right? But it didn't.

• 4:42Speaker 00

  It still, I think, didn't have the same cachet inside the company, right? As the search and that sort of thing. And so, you know, when I was working on it, Calendar, it was still like mostly consumer. It was like kind, it was also sold to enterprises, right? But like, we were not the ones making the money. Like, we were probably a call center, right? At that point in time.

• 5:04Ryan Peterman?

  I saw eventually you ended up leaving Google. And from your post, it looks like it was pretty bittersweet. So what left you to leaving Google?

• 5:13Speaker 00

  I had been there for four years. And, you know, I mean, realistically, like investing was a thing. Like finances definitely changed after that four year period ended. But, you know, also it was just tough. Like I, I had a, I guess, a bad habit at that point of working really hard on things that like really were important to me, but maybe weren't important to Google, right? So like I worked on Calendar, that was pretty important. And then I worked on Google Tasks, which was like a very small feature within Calendar, right? Like probably at least two to three orders of magnitude fewer users, right? But I was really passionate about it. And then I was really passionate about the JavaScript infra and the closure, that suite of JavaScript tools.

• 5:58Speaker 00

  And you know what's great? I mean, I enjoyed and I'm proud of all the effort that I put into the things. You know, that I went and wrote the book on closure because I was so motivated. But career wise, maybe not the best move, right? You're like, but I'm doing all this high quality stuff. And you see other people getting recognized. You're like, but I'm working so hard, you know? Like, it's, you know, kind of like the harder not smarter type of mistake. And so. You know, it's kind of like I should go somewhere or try something else and see if like the things that I'm excited about are the things that like the company I'm working at is most excited about.

• 6:34Ryan Peterman?

  Later you came back into big tech and you started working at Facebook. I understand that you were kind of like a JavaScript expert at the time. And then one of your first big projects at Meta was kind of build tooling in the Android codebase. So I want to know the story behind how you got involved in that.

• 6:53Speaker 00

  So at the time, it was like, Facebook's gonna make a phone, right? Actually, there are some failed projects, but it was like this time it's really gonna happen. We're gonna partner with HTC. We're gonna like fork Android and, you know, and do some stuff. And, you know, so that seemed super exciting, right? As a person, you know, just coming into the company. And, and I, you know, I'd done like quite a bit of Java. I was more JavaScript.

• 7:21Speaker 00

  But at this point, this is also where, you know, they called it FaceWeb, the version of like, they kind of like put HTML5 Facebook on the phone and like that was clearly not working. And it was clear that like, you know, mobile was going to be the future. That was kind of make or break for the company. And suddenly, you know, a friend was like, hey, like, I know you really like JavaScript, but like, you should really pick Java or Objective C and like get good at this if you want to be a product person. And that was really, really great advice. And so I was like, well, I like Java. I don't like Objective C. So like, let's go. And so that's how I, you know, found myself on that project.

• 8:06Speaker 00

  And I, you know, we, we had a very short timeline because unlike almost every other project, there was a hard deadline. Usually, you know, you ship when it's ready. This was like, nope, we gotta send a bill to HTC because they're gonna like burn it into phones on like March 1st or whatever it was.

• 8:26Speaker 00

  So, you know, it was really a scramble. But, and

• 8:33Speaker 00

  The original Android codebase was, or a bunch of it was also like inherited from a contractor that Google paid. Like, it's like Facebook didn't want to make like a native app and they paid some guy. And then it was like the app in the store and they're like, here, the code's yours now. And we should have thrown it away, but we didn't. And so a lot of things that were frustrating, but also like iteration time, right? I think for everybody, if you'd been a web developer for such a long time and you're used to this edit, refresh. And then, you know, like the Android build system was, was rough, right? It was this, um, like some build tools and Ant and it like, there were no, there was no way to modularize it out of the bot. Like, we had to hack it up just to even have like four modules or five modules or something like that.

• 9:17Speaker 00

  And it was so painful to get anything done.

• 9:24Speaker 00

  Then I was like, I need to, I need to fix the build system. Like, I was like, I know I've done a lot with Java. I know it's not like fundamentally this slow to like iteratively, you know, build this sort of thing. And so, uh, you know, Facebook has this hackathon culture. So I was like, all right, hackathon, totally gonna like make a new build system. You know, like unceremoniously in the style of Google's, uh, build system. And there's actually another build system called FB Build that was already a different, uh, mirror of, of the Google build system as well. Uh, that one's written in, in Python and only did C++ but, um. Yeah. And I was like, either I'm gonna make this work or, and like, and then I'm gonna be happy to be a developer or like, I don't even know if I'm gonna

• 10:09Speaker 00

  make it here, cause it's just gonna be like tearing my hair out, you know, working on this thing. And, uh,

• 10:15Ryan Peterman?

  If you didn't fix this, you would have quit the company.

• 10:19Speaker 00

  Well, it was, I mean, it was, I was like, or at least switch projects or something. I need to find a way to find a happy place, right? And, and come in work every day. Like I came, I want to write code. I want to do work. I just want to, you know, try to do my best work. And so it was funny, actually, I give people credit. A lot of people told me what I was doing was a very bad idea. Almost everybody, except one person. I was a senior Android engineer. Uh, but nobody said no, you know, whereas I felt like at Google, like people said no more. Um, and so, uh, so I just rolled with it and, you know, I quickly, relatively quickly had something that was,

• 11:00Speaker 00

  you know, dramatically better, like at least like twice as fast or something like that. And, uh, so that, you know, it turned a lot of heads and people were like, okay, you know, I guess we, yeah, we'll go with that one. Right? Like.

• 11:14Ryan Peterman?

  I think the interesting thing to me is it seemed like all the odds were stacked in fear. A lot of people who are engineers noticed the same problem would not have chosen to start the project because there's an existing one. And then also Google's got some, you know, competing one. Maybe you're not going to beat that one. What gave you the conviction that your, your project could beat the other ones and become the default?

• 11:39Speaker 00

  Yeah. I mean, I guess a couple of things. I mean, one, like I said, I, you know, I'd done other Java stuff. I was like, like this shouldn't be that slow or just as a software engineer, like this fundamentally should not be as slow. Um, as it is. And there were, yeah, most of the, actually, most of the pushback was of the nature like, well, if we deviate from the standard thing, um, you know, we're not going to be on the standard thing. And what if it gets like a hundred X better next week? And now we're stuck on your thing. Um, which is really funny when you think about it, because there's like so many things like, you know, making their own, you know,

• 12:14Speaker 00

  PHP virtual machine and language, right? Like they've certainly embraced doing their own thing many times. Why this one was different. I don't know. I mean, it was, it was just, I guess everything about mobile was like, how are we going to make this work? I guess there was just like a lot of fear and, and, you know, again, like timelines, it's like, we have the senior person, like why? It seems like, it sounds like they're going to go off and do a science project. We have a hard deadline. Like, is that the best use of our, of our time?

• 12:39Speaker 00

  But you know, it worked out. It worked out. I will say another thing, too, was I also tried to to couch it a little bit. I was like, hey, I'm just, I'm just trying to make an Android built system. I'm not trying to like take over the company. I'm not trying to change anyone else's project, right? Because I certainly felt that that was going to cause more, I was going to invite more friction. And so, you know, I knew that it, I tried to make it in a way that it could support more of the company, but I never, I never pushed it. And so I was like really heartened when the year or so later, the iOS people were like, hey, our built system sucks. Can we, you know, can we use Buck? I was like, yeah, sure. Like, I don't, you know, come on board.

• 13:22Ryan Peterman?

  Yeah, that's also another interesting, because you, you came in and you didn't have a whole lot of credibility. You were just hired. And then you're trying to build something and get every, and also everyone's saying, don't do this. And then you have to convince them, hey, this is the right direction. How'd you influence that change without any credibility?

• 13:43Speaker 00

  I mean, I borrowed some. There's one senior Android engineer, you know, who was also ex-Google, John Pirlo, and he was like, I think he said, right, he's like, you're gonna do it, just do it fast before like anyone gets, you know, like Kansas City or something. He's like, well, I'll support you if you, if you get this done. So he was one of the first people, and he was a very prolific coder. So he was like genuinely very happy to have a faster dev cycle. So he was one of the first people to, you know, give it kudos. And I think that, that helped a lot. Also, but I will say, you know, I, I made a big, some big mistakes on my way in there. You know, you mentioned like about having no credibility is like, you know, I came from Google

• 14:24Speaker 00

  and I was like, oh, this is where, you know, the X Bell Labs people are, like all these like great people. And then you go to like Facebook and you're like, oh, it's a bunch of college kids. How could they possibly know what they're doing? Right? And you, you know, I made a few times, certainly I made the mistake like, well, like Google, we did it this way. And people were like, we really don't care. You know, and they were, and they were right in, in, in most cases, right? That like, just because it worked there didn't mean it was going to work here.

• 14:50Ryan Peterman?

  One last thing on this topic is, I mean, what you built was way more performant than anything else by many multiples. What's the intuition, like the technical intuition behind what was then and what you did that made it so much more efficient?

• 15:08Speaker 00

  I think the big thing was is that I just sat down and looked at the tool from Google and I was like, what is it actually doing? Right? And where? And I think the big thing is that the Google one would kind of just, if you changed anything, it would just start over from scratch. And so that's why it was so slow, especially for incremental builds. And so then I started really understanding like, okay, but what depends on what? Cause there is these, I still aren't sure, like Android resources, which had like a very bespoke thing. And it was, it was complicated. And because it was somewhat complicated, I think that's probably why it just by default, I blew everything away and started over.

• 15:48Speaker 00

  But once I got in there and I was like, oh, okay. You know, we can take advantage of it. If these things don't change, then, you know, we can cash this result from this step. We don't have to redo it. And like suddenly, you know, that made things like quite a bit faster. And then also just even supporting the idea that you could have more than like the four modules that we had. I mean, I think, I think every time someone added a module, you know, they had to add like 200 lines of XML of like Ant build script that like nobody really understood. And so it was not, so no one, no, no one wanted to modularize anything, right? You didn't want to be responsible for that. So a big thing with Bock was making it that it was like a lot simpler to add a new module. And so then that also meant we ended up with more modules and then builds are more incremental as a result, right? So it's really a change in the mindset.

• 16:33Ryan Peterman?

  So less redundant work, basically.

• 16:35Speaker 00

  Yeah.

• 16:36Ryan Peterman?

  After you solved this build problems, I guess, in Android builds and I obviously I went later to further parts of the company. I saw that you started to work more on like the IDE. What was the problem that you saw that you saw in IDEs that made you want to get into that?

• 16:53Speaker 00

  So I, yeah, I took, I did a brief stint after Buck on Messenger. I iOS Messenger. So I was like, okay, I've done Android. Maybe I should actually branch out even though I don't like it. And I still didn't like it. I, people don't even realize in Objective C, there's a thing that happens for a long time now called ARC. Like the automatic reference counting. Like nowadays, the compiler injects it. But you used to have to like actually add code in Objective C to do every like reference count, like explicitly. Yeah, or like every reference that you created, like you would have to do it and

• 17:33Speaker 00

  and the, and most people have never seen this type of code, but the iOS Messenger code that came in through acquisition was so old that it was still written in this other way. And it's incredibly painful. I guess now you'd have like Codex cleaned it up or something. But at the time, we just sucked it up and just Xcode just didn't, you know, feel right to me. I didn't like header files and implementation files. I still don't.

• 17:57Speaker 00

  And, and also the, I would say

• 18:02Speaker 00

  for both Android and iOS, you know, Facebook always had like the biggest app, right? We had like one app with every, with every feature in it and we, you know, ship it as opposed to Google with like, they'd have like a Drive app and a you know, a Sheets app or whatever, but they also owned one of the platforms. So they could put 20 apps by default on there. And so what that meant was is that Facebook was always hitting the scaling limits of every mobile developer tool before everybody else. Which was painful, but you know, actually as a DevTools person, it was kind of interesting because we got to solve problems that nobody had solved before. And like, you know, not just as a science project, but because it was like real, you know, business value.

• 18:42Speaker 00

  And so, you know, Xcode similarly, we talked to Apple, we were like, hey, Xcode's like not really scaling to our project or like your project's too big. You should make it smaller. Like that was kind of the feedback that we got from them. And so it seemed justifiable to like go and build an editor. It was a similar thing. It was like, I was like, what is, you know, what is an IDE? You know, doing right? It's like, it's talking to clang, like the language server and that sort of stuff. I was like, we could build like a nicer shell on top of that, you know, and then we had started by that point.

• 19:12Speaker 00

  Deviating from Git and switching to Mercurial as a company, I was like, no one's gonna support Mercurial out of the box, right? And so like, that should happen. And then like, oh, actually, if Buck is gonna be our build system, then like, we really like, you know, like, that's Xcode is never gonna do all of these bespoke Facebook things, right? So it seemed justifiable to like, invest the time to, to try to like, improve that experience. And I didn't feel that way about Android, because IntelliJ was actually like, quite good. And we had figured out how to make that work at scale. But but Xcode was a little more difficult at that point.

• 19:45Ryan Peterman?

  So there's Xcode, which was bad and fits the existing needs. And then there was another IDE that another team was building. I think it was web-based, if I

• 19:56Speaker 00

  It was it was web-based. I'm not letting you get that part. That part's fine. It was, but it was built off of an abandoned Google open source project written in GWT, which is Google Web Toolkit, where you would write Java and it would code gen to JavaScript for everything. And I tried to build on what they had. I tried to build some credibility. It would actually sped up their build a bit. But again, kind of like, looking at the iteration times. And also, I was like, this is an abandoned open source project. It's written in Google Web Toolkit. And I was like, we are the React company at that point, right? Like, how, why would we not empower, you know,

• 20:38Speaker 00

  people who want to build dev tools to use, you know, to build on a, you know, technologies that we are the leader in and actually really like, because, you know, we think we're actually really good. So I was like, you guys are crazy. And so I started, and it's similar to the thing where I, you know, I mentioned with Buck, I was like, I started it as the Java build tool, not the everything build tool, right? I was like, it was a similar thing. I was like, hey, I'm gonna go over here and start this other endeavor, but we're just, we're just gonna focus on iOS. Like, I'm not trying to take things over.

• 21:12Ryan Peterman?

  Yeah, you didn't want all the friction. And I mean, that team, they had all the existing users, though, right? They had thousands of engineers.

• 21:20Speaker 00

  Even maybe a thousand, I think.

• 21:23Ryan Peterman?

  Eventually, leadership sided with what eventually became New Clyde, which is what you built. But why did they side with you when you didn't have any users on your share profit?

• 21:35Speaker 00

  Yeah, I mean, I think it was a combination of two things. I mean, I think the the arguments that I made in favor of

• 21:42Speaker 00

  like the technology stack to build on. Another was that

• 21:48Speaker 00

  New Clyde was was a desktop application and part of it was if this is actually going to be a an iOS, you know, Xcode replacement, like, people are going to want to use and you'd be able to talk to the simulator or plug in the phone or whatever. Like, in theory, we could go through the web and do all those things, but it just seems like a lot more, a lot of extra work.

• 22:07Speaker 00

  And and I think the other part is that, you know, I had built up some credibility with the the Buck thing that

• 22:17Speaker 00

  they're willing to like, take a gamble. Like, well, the last thing worked out, like, wait, let's try this one.

• 22:22Ryan Peterman?

  I saw that this, in combination with some of your other work, led to your promotion to E8, which is also known as principal in the industry. What was your reaction to that at the time?

• 22:33Speaker 00

  Yeah, I mean, certainly, I was I was very excited. You know, I felt like, you know, in the ways that I had been out of step, I felt like, you know, when I was at Google, I was like, this is also not just, it was also validation that like, oh, now I'm also growing, not just like technically, but like, understanding like what it means to kind of, you know, do the things that like are in line with your employer. And so like, that also just, you know, was equally valuable or satisfying.

• 23:01Ryan Peterman?

  I I know that New Clyde was open source, and I think Buck was as well, if I recall correctly. What's the rationale for open sourcing if, yeah, like, what are the pros and cons of open sourcing technology that you build?

• 23:13Speaker 00

  Yeah, I mean, let's see, the Buck is the more interesting one. New Clyde kind of didn't really get adopted externally. I I think in both cases, I think.

• 23:25Speaker 00

  You know, all these companies have benefited so much from open source. Like there, I think there was just a feeling of like, if it's not really the secret sauce, like, but let's share this with with other people, right? Like, I mean, done like Codex as well and then the brother things in my career. So I do feel like that, that that sharing of information is just good. Even if no one uses your tool, just as like seeing as a reference of like how a thing could be done, could be great.

• 23:52Speaker 00

  You know, and in the in the better scenarios, you actually get, you know, meaningful contributions back and things like that. And, um, you know, I remember like, I think Uber adopted Buck. I think Airbnb did, like, it was funny, right? Like I said, like, Facebook was the biggest app, so we would hit all these problems first. And then this next wave of companies would start thinking these things and be like, let's check out what these guys did. I think, I mean, honestly, we also, you know, at at Google, it was a, it's internally, it's Blaze, externally, it's Basil. So we were open source first and we always did, we always did kind of wonder like, if we put some pressure on, like, you know, we get, and you ultimately did. I think, I think, I think you've gotten a little bit of credit from that unofficially from some of the folks who worked on that.

• 24:36Speaker 00

  But I think also that, you know, we, you know, it's also like a recruiting thing too. We're just showing like, hey, if you want to like, you know, being at like the, you know, the the bleeding edge in like whatever area this technology is, and you want to do this all the time, not just as like a drive by contributor, like, you know, this is what we do, this is who we are.

• 24:53Ryan Peterman?

  And then this decision to open source, is this, was this a bottoms up decision where engineers said, we're just gonna do this? Or is this kind of also leadership buy in? Hey, this is gonna be valuable and you got like a doc?

• 25:07Speaker 00

  Yeah, yeah, I think, I think both cases, it was certainly like bottoms up. I don't think.

• 25:16Speaker 00

  So, you know, things like React and PyTorch, like, those are like the the big success stories, right? Where where the value back to the company is like unquestionable. And then there's this like longer tail of things where. Um, I think depending on the economy and other things, the managers get grumbly if they feel like their engineers are like doing too much open source or things like that. So it's almost, almost always, I feel like, like bottoms up. I think, yeah, I would say that's, that's often the case. But it wasn't like met with a lot of resistance, you know, and you usually get like a good conference talk or two and a nice blog post. And those blog posts like, you know, do, I would say, pay dividends over time.

• 25:55Ryan Peterman?

  Right, drawing, recruiting.

• 25:57Speaker 00

  Yeah, like they have like a longer shelf life than if people realize.

• 26:01Ryan Peterman?

  Okay, so you got your, your E8 promo at this point. And I imagine the expectations maybe in your mind are kind of going up a little bit. And now you need to go and find a E8 problem, I guess. So what'd you do after you got promoted?

• 26:16Speaker 00

  Uh, I think that's where I did a, uh, I, I got a little over my skis and I tried to help with web speed, I think is what I did. Um, cause again, it was a thing that was like a really big problem for the, like, like the, just the load time of facebook.com was, was not in great shape. You know, the architecture was a bit stale. Um, and it was just, um, the problem was so big. And, um, and I didn't have, I think the, you know, a lot of people who had worked on web at Facebook had worked on it for like a long time. And like I had, you know, put myself in mobile and DevTools. I actually wasn't in that world. And, um, I remember I was like, ah, you know, what do we do? I mean, actually sit down with

• 27:01Speaker 00

  another person, we started like compiling V8 from source and trying to see like if we could like change the way that we generated JavaScript, if it was like it'd be friendlier to V8. We were just trying like wacky things, none of which panned out, by the way. And I just, it was, it was, there's different, there's different people who are like good for different projects. And I think that was just not the one. Like I'm better at a project that involves writing a lot of code from the beginning. And I think a project like that was like more about like looking at data and talking to a lot of people. And like, I just, that's not my strength.

• 27:36Ryan Peterman?

  I think you mentioned that at this point in your career, the, the idea of a hero quest.

• 27:40Speaker 00

  Yeah.

• 27:41Ryan Peterman?

  What's, what's that mean?

• 27:43Speaker 00

  The idea that, you know, that, uh, it's like, you know, it's embarrassing because there's like something about ego. This idea that, you know, there's this like, you know, Gordian knot and all these engineers are like, oh. If only someone would come in and like solve this, you know, engineering problem. I was like, I know JavaScript. Um, just come in that way. And, uh, but I didn't, I did, I definitely did not, you know, uh, and I think that, um, it's been an important learning and I've had to relearn it like at least one other time that,

• 28:15Speaker 00

  you know, I can do a lot of things, but there, there is a smaller subset of things that I genuinely enjoy doing and that I'm going to be a lot more successful, right? If I stay in the things that I genuinely enjoy doing, you know, I, I try to expand that over time, but you know, we don't all have to be the best at everything, right? I think that's, if you accept it and, you know, embrace it.

• 28:39Ryan Peterman?

  So then how'd you find the EA problem after that? Like, what was the next step?

• 28:44Speaker 00

  That was, uh, I think a bit of luck as well. I guess, you know, we had, we had, sometimes we'd have these little summits, uh, with like smaller groups of engineers about like brainstorming what's gonna, you know, bite us in the future. Yeah. I mentioned, I was like, you know, the repo keeps growing. You know, we're, you know, we're gonna have some scaling problem at some point and, um, uh, a person or my manager, then became my manager, uh, Brian O'Sullivan.

• 29:11Speaker 00

  He decided to get some people together to like work on making a virtual file system. To, uh, to, to get, try to get ahead of that, that problem. And so you got myself and, um, Adam Simkins and Wes Furlong, um, you know, both tremendous engineers. I actually felt I was the worst engineer on the project for quite a bit.

• 29:30Ryan Peterman?

  You mentioned a virtual file system. What, on a high level, what's the benefit to a company like, uh, Meta for, if you had something like that?

• 29:39Speaker 00

  If you, uh, have bought into the monorepo philosophy, right? Put all the code in one repo. You know, most things that people do need only a subset of that repo, like have to look at the files at any point in time. And so the idea is that, you know, you design all your tooling around those virtual file systems that when you say, clone the repo, or then you update to a different commit or anything like that, you don't actually have to like write out every single file in the repo on disk when you make that change, which is, which is the default in your, you know, traditional file system, because that is going to grow proportionately to the time with the, the size of the repo, right? So at some point, you're, you know, going to be very sad.

• 30:22Speaker 00

  And, um, and, uh, and so, you know, there's actually kind of two, at least two parts to it. One is building this virtual file system that's like, hey, I know the user's at this commit. If, if the operating system asks me for the contents of the file, like I can go get it and it will appear like they had actually, you know, laid out all the files. The other part of it, which is the part that I maybe I was actually better at, I think, is then try to anticipate, well, we have all these tools in our tool chain that are used to just reading all the files. And you just rip graph, it just reads everything, right? That sort of thing. And how do we start changing also our, our development flow and our tools

• 31:03Speaker 00

  such that they are designed with the virtual file system in mind? Because if you just, you know, materialize all the files with your tool and they've like actually just lost all your, the benefits that, that you've made.

• 31:15Ryan Peterman?

  So on a high level, it's basically just lazy loading a huge file system. So it's more efficient because you don't need to do everything at the beginning.

• 31:23Speaker 00

  Yeah.

• 31:24Ryan Peterman?

  And so that part of this project that you said you're better at is, is integrating everything into this on top of that primitive.

• 31:31Speaker 00

  Yeah, yeah. So one thing that I did, um, actually with Hanson Wong, who's here, actually, I'm on the Codex team with me now, um, was, I was like, oh, okay. You know, the, the traditional way you have, you know, you want, um, really fast file search in your, in your IDE and your editor. And, you know, the way most of these things do is the first thing you do is they walk the entire file system to find out what the files are. And I was like, well, that's going to be a, that's going to be a serious problem, right? It's going to undo all the benefits, like, and, and then it's one of these things is like, not, you know, first it was like, okay, how can we? Implement file search that's not gonna undo all the benefits. And then it was, how can we do it even better than how it's done today?

• 32:12Speaker 00

  And so we ended up building this file system called Miles, short for my files. And it, you know, on like, kind of like a cron job, it would index, it would ingest all the new commits that had come in on, you know, on trunk. And keep track of like which files had been added or removed. So just like the names of the files, not the contents, because all you need is the names. And, and then, and then Hanson had some clever ideas about how we maintained that index such that we could support fuzzy file matching. So instead of just substring matching, right? Like you type, you know, like a, if you have like a camel case thing, you want to be able to type just maybe the uppercase letters or just your spelling spatter, what have you.

• 32:55Speaker 00

  And so it came up with this really interesting way to represent, you know, kind of all the files that had been seen at some point and then some bits to represent. You know, if I'm at this commit, was this file present or not present at that point in time? And then also, when you ask, I mean, you sent a query to this thing. So you'd send like what commit you're at and like if you've added any files locally or removed any locally. And I think we got this, you know, like, you know, over a million files and like, I don't know, like 10 to 20 milliseconds or something like that. So that you would be in your editor, you know, typing. And so that was way faster than, I think than, you know, like what Xcode or, or is there like even VS code

• 33:38Speaker 00

  or something like would give you out of the box. And so, so that was, you know, so like it was solving a problem for Eden, that was the name of the file system originally, the virtual file system, you know, in anticipation and before it was even ready. But then it was so fast and it was available as a trip service internally. Like people started using it for all sorts of other things. I think when I left, like, I don't know, I think there were at least like thirty servers running Miles like spread around the globe. And so clearly that was more than just people like personally searching for files and then you did that much, you know, capacity.

• 34:13Ryan Peterman?

  Interesting. Yeah, I mean, when you talked about, I guess, the implementation details. Most, most people, they, they don't use leaked code on the job. But that sounds like, I was thinking about the, I don't, I don't even know how you input that. Is it like a tree, like a try thing?

• 34:29Speaker 00

  No, that, so that's what's funny that try, yeah. This was, this was pretty cool in that we, we had kind of like two parallel arrays. One was like the file contents and one was maybe an integer into that. Index, I forget. And then we had a, we had a sixty-four bit.

• 34:53Speaker 00

  Mask that was like, it was like, so you had like twenty-six lowercase, twenty-six capital, ten digits and like maybe dash and whatever. And, and it would, and it was a, every bit was set if the.

• 35:07Speaker 00

  If that character was used at all in the file that you were searching. And so the first thing was we could like blow through that list. And, you know, exclude a bunch of things like right off the top. But we was also like very designed, like all these arrays were in parallel to each other so that. For like cache wise, we knew it would be very efficient for, for the CPU to, to like read memory, you know, linearly. And then it led itself to, you could just partition that array. It, you know, led itself to parallelism. So it's really, yeah, I should really probably write it up at some point.

• 35:41Ryan Peterman?

  Cause that is, that is really cool. It was, it was funny, cause it wasn't just kind of like out of the textbook. That's, that's for sure. I understand that, yeah, you worked on Eden and then obviously there's Miles as well. And then these eventually led to another promotion and, but prior to that promotion, there was some learnings you might have had about influence and conflict and org. So maybe if you want to share that?

• 36:05Speaker 00

  Yeah, I mean, that's.

• 36:08Speaker 00

  One of the things, I guess, about being an E8, who primarily writes code, right? There's other people, I'd say actually the majority of people about level or higher are not writing code, right? They're actually kind of exclusively spent doing more like influence or working across teams, you know, writing the big Google Doc and getting everyone on board and all that sort of stuff. And so, you know, as an E8, trying to have that level, like, you know, that level of impact that you're expected to have, it's like, oh, it's hard to do that just writing code. So it's like, I need to spend at least some time, probably, probably influencing other people. That would probably be, be good for me. That would probably make my manager

• 36:53Speaker 00

  happy. And I think, you know, sometimes you're just so confident in, in some insights you have that, or certainly I was, that I was just, I, I just came down like way too hard, I would say. And, and that did not go well for me. And yeah, and so like, like promo got delayed because I was very excited or anxious about when Microsoft acquired GitHub. And, oh yeah, because new Clyde was built on this, you know, primarily GitHub technology.

• 37:36Speaker 00

  And I was like, that's going to go away because VS Code is going to not make them be a project anymore, which was true, which did happen. And I was just so anxious that, that this was like now a risk, right? And I was pushing people, but you know, people were, I didn't really account for the fact that people were happy with what they were doing at the moment and like, didn't want their cheese moved, you know, like right out from under them. And so, you know, I got like a little bit of a talking to, you know, I had to wait a little bit for promo and things like that. But I, you know, I actually even got some coaching, I think, after that point, I can't remember the exact technology, to, to work on that, like specifically.

• 38:18Ryan Peterman?

  What's the number one thing you learned from that coach?

• 38:21Speaker 00

  I would say, for me, is like, I think I'm more aware of like things that trigger me. Like, it'll be your technical decisions or what have you, and recognize when that's happening and be like, okay, let's not act in that moment, or You know, if I if I don't think I can have the conversation like, uh, or I'm not the in the best place to have it, like, maybe I go talk to the person's manager instead, rather than like, you know, like bull and china shop to the engineer and be like, so I have this thing, I'm thinking about it this way, I'm kind of riled up about it, like, how like, how can I, you know, work with your team or whatever, like, that's happened to you.

• 38:59Ryan Peterman?

  Well, it's interesting to me in your career, because the promo got postponed and you were you were seeing that VS code was gonna come up and the thing that new cloud is built on was gonna go down. And you were actually right in hindsight, too, like, in the future, that is what happened and you were battling for what was right, yet you were told, calm down a little bit and this, so I mean, yeah, what are your thoughts when you saw things play out and you go, actually, I was right the whole.

• 39:32Speaker 00

  Um, I did have some conversations like about a year later and I was like, hey,

• 39:38Speaker 00

  can we balance the ledger a little bit, like, it didn't be pretty hard for that thing and it was kind of good and we we worked it out. I would say.

• 39:46Ryan Peterman?

  Okay, so I guess the the learning is just how you went about it, not what you were like.

• 39:51Speaker 00

  Yeah, yeah. I'd say that's true.

• 39:56Ryan Peterman?

  You seem to have a great time at Meta and eventually you left, so what was the thing that drew you to OpenAI?

• 40:03Speaker 00

  Yeah, I mean, um, a number of things. I mean, one was, uh, so I I interviewed at the end of 2023 with OpenAI, I believe, and um, and I had spent 2023 at Meta, um, trying to do LLM-based developer tools, right? With, uh, there's a, you know, we had our own light, I mean, here's even Meta me, it was like a little version of a GitHub Copilot, right? That one, we did like a paper and a talk on that one. It was a code composer, that was code composer, yeah. And, um, you know, we, uh, like now, uh, there's a lot of, uh, uh, enthusiasm around, you know, delivering quickly and pushing the boundaries of that sort of stuff and, uh,

• 40:45Speaker 00

  you know, I mean, truth, I mean, you know, you get feedback on some of these things, it's like, why is this not, you know, GPT-4? And I was like, well, we're on like Llama 2, it's not the same thing and, uh, you know, I was not, not a researcher, right? I was like, but I, and I, you know, just wanted to to build the experiences, right? And that sort of thing. And, um,

• 41:07Speaker 00

  And so that was that was one thing was that like I wanted to build stuff and I wanted to go to the place where I could actually build with the best model. You know, two again was similar to like, you know, Google originally, like seeing the people who were coming here to OpenAI and I was like, well, those are people I really respect. I was like, those are, I could actually work with more senior people. You know, like Meta eights and nines at OpenAI, I felt like that I could where I was. And, you know, third was just like, this also seems like, and it has been just a very special place at the point in time. So I told a lot of people, I was like, you know, I felt like this would be like the most similar to starting at Google in 2000.

• 41:53Speaker 00

  So not like, not to 1998, but like, let's say 2000, right? Like they kind of got some footing and got some, you know, product market fit. And so like, just as a personal level, that was just a very exciting thing.

• 42:04Speaker 00

  And, you know, the last one was that, funny thing is that when I, I really enjoyed calendar because I, it was consumer and I shipped to a lot of people. I went to Facebook because I thought huge consumer place. Ended up not doing consumer at all, right? Ended up doing developer tools, but like, you know, my users were my friends at work. It was just like 20,000 people, not like a billion people, but it was good enough for me at the time. And then, you know, thinking about OpenAI and then the chance to actually, you know, come back to, to consumer. Or at least like have a large user base, right? And so now working on Codex. You know, I wonder if there are like over a million weeklies or I forget how much it is right now.

• 42:46Speaker 00

  And, and just keeps growing like, like, you know, hockey's there more like vertical line than hockey stick even. That, you know, so it's like way more than the 20 to 40,000 developers, you know, I could affect it at Meta.

• 43:00Ryan Peterman?

  Yeah, absolutely. Dev tooling for the industry almost.

• 43:04Speaker 00

  Yeah.

• 43:05Ryan Peterman?

  Meta, when I think of the company, it's kind of engineering driven, like engineers are king and queen, I guess. It kind of like drive everything's very bottoms up from that sense. And I feel like a lot of the lab companies. Are also like that on the research side. So rather than engineers, the first class citizen, it's like, let's make sure the research goes well. And for good reason, right? Like, that's also part of why you came is the models are good. But as an engineer, you mentioned you weren't doing research. What are your thoughts on that research-led culture versus engineering-led culture?

• 43:41Speaker 00

  I mean, it's certainly an adjustment, right? I mean, I think if anyone who comes here and says otherwise is a liar.

• 43:47Speaker 00

  You know, if you've been at like the Fangs or whatever. And but, you know, they are, you know, you talk about impact, right? Which I think is important, like a real, like you genuinely, you know, mean it.

• 44:00Speaker 00

  Like, I, you know, I love the work that I do on Codex and the Harness, and I think we do a lot of very meaningful things. But, you know, if the model weren't very good, it wouldn't really matter what we did, you know, on the Harness, right? And so, so I don't.

• 44:17Speaker 00

  So, you know, that's how it is, I would say. But I, you know, I feel really great, like, you know, we sit right next to the research team. We're really close with them. And so that relationship that we have, that we get to co-develop the thing. I mean, that was another reason, you know, leaving Meta, I mean, was that for in the LM space was like, you know, I want to build the product with the people who are building the model so that we can do this thing together. I mean, you know, maybe you could do that there, sort of, but not. Anyway, the impact was not, you know, quite, quite the same.

• 44:53Ryan Peterman?

  So you mentioned, I mean, when you got here, it sounds like you were working on Codex and starting that project up. So I understand with the initial launch of Codex CLI, it was not exactly what you hoped for, like in terms of the how it was received, but later it kind of really all came together. Can you tell that story?

• 45:13Speaker 00

  Yeah, sure. I mean, it's been a wild ride. So Codex CLI, we launched it in April 2025. It was kind of like this, like one more thing moment at the end of the 03, 04 mini livestream. And so we demoed it live, we open sourced it, you know, through a repo at that point. A lot of people tried it out, you know, everyone was excited to try a new coding agent. You know, and it was and it was pretty good, but it was it was pretty rushed to get it out the door. Right. There's um, which in some ways was good for engagement because never open source. We were getting pull requests, you know, coming in all over the place. And I forget, you know, I feel like we're like 10 to 20,000 stars in like a week or two or something like that. So like that that part was fun.

• 45:58Speaker 00

  And, um, you know, there were people who did like, you know, it's like nobody liked it or anything like that. But then, um, but that I think we didn't, we weren't maybe quite staffed to like really drive that the way that we needed to. Because, you know, we're we're trying to cover multiple things as a company. And so, you know, just a month after that, and you know, a team of like seven engineers, and I forget how many researchers, launched Codex Web or Cloud Web, where you can use Codex in a container or you could just, you know, kick off anything from your phone, which is super cool. And, um, and that was like a more well-staffed effort. And I and I think, you know, that one is still the, I think the the long-term vision is is correct.

• 46:43Speaker 00

  But I think with a lot of this stuff that we've seen, you know, you have to bring the the users with you. And and and I think that was just like a little bit ahead of its of its time. And people were still actually more into the the local coding agents. And so we kind of, you know, we saw, you know, the like the web product, like there's a big adoption initially. It wasn't, you know, as sticky, I would say, as we we had hoped.

• 47:09Speaker 00

  And then through the summer, you know, both products still kept kept, uh, you know, we're working on both. But then somewhere that midsummer, again, like I said, is like, you know, this moment, like local agents are still the, I guess, the stronger product market fit. But again, I still think in the limit personally, right? Like, uh, you're you're you're going to need more machines than just your laptop to as a place for agents to run. And, um, so we we shifted quite a bit over the summer. So we we brought more people on to the Codex CLI. Yeah, a few subkey things, bringing more people on. You know, GPT-5 was gonna come out and that was looking really, really good. And, um,

• 47:50Speaker 00

  I think I was personally excited about because I prototyped it a couple of times before. Was that in addition to the CLI, then we also had enough staffing, we also started working on the VS Code extension. And, you know, and I felt actually very strongly that, you know, the, the, the terminal is good for a lot of things, but it has a lot of limitations always. Just, it's just like, you make a lot of compromises to make, you know, a nice UI in the terminal. Whereas VS Code, we didn't have to make quite as many compromises. And so I think like August, I think was a crazy month. I think GPT five came out. I think we re-released our new refreshed terminal UI. Also, the GPT open source model came out. We supported that

• 48:31Speaker 00

  in the, in the two E as well. So that was really cool that we had, you know, an open weights model and the, and open harness. And, and then later that month, the VS Code extension, you know, came out. And so we were just like, you know, shipping like, like crazy. And, and that's really where like that confluence of things, I would say, is where we started to see the inflection point that's brought us to like, you know, the vertical growth that we're, that we're at today. And so that's, you know, that's been really exciting. I mean, you know, some of these things you can, you can go in the repo, check for yourself, right? And, and, and you got check me on, on some of these things. Terms of number of people, a number of commits and all this sort of thing.

• 49:14Speaker 00

  And yeah, so it's, it's, it's been quite a ride.

• 49:17Ryan Peterman?

  You mentioned the local versus the remote version of these coding agents and it sounds like you have a lot of conviction that the right long-term direction is not the local versions. It's remote and in the cloud. Why is that?

• 49:31Speaker 00

  Well, I think about, I think that the people who actually for whom it is sticky now, and I think there'll be more of it, is that you imagine, if you imagine you want to just automatically.

• 49:41Speaker 00

  You know, for the agent at every like GitHub issue or linear task thing that comes in or anything like that. I mean, obviously, you know, there's, there's costs with those things, but or you can, it could be abused. But like, you know, let's say for an internal private repo, right? That you, yeah, want to be able to just have it be a piece in any sort of automation pipeline, right? And so you can't have all that happening on your, on your laptop. And so I guess, I mean, you know, as a, as an individual, I see, maybe we'll still personally spend more time with the local agent. But in terms of, uh, you know, compute time of agents doing work, right? I think getting that set up in the cloud.

• 50:21Speaker 00

  I think, you know, a little bit, it's like a little more to get set up right now, but once you have it, it's, it's quite nice.

• 50:27Ryan Peterman?

  I see. Okay. So you're not saying that product, the local one will change. You're saying that like across the industry, the compute that goes towards agents, majority will be in the cloud and.

• 50:39Speaker 00

  Yeah. I mean, even in, um, I think when the VS code extension first came out, you know, one of the things was, um, the ability to take the conversation that you were having and like hit a button, have it like transferred to the cloud if you were set up to do it. Right? And, um, so I think that we'll continue to see that where you may, you're working on something, you want to throw it over here. You're like, bring it back, you know, when it's done.

• 51:00Ryan Peterman?

  Your Codex usage is, uh, five X since the beginning of the year and over a million people are using it now. I'm curious, has your AI workflow changed a lot since you've, uh, you know, started to use this newer version of Codex?

• 51:14Speaker 00

  Yeah, it has, it has. Um, I'm a much bigger user of the app now, maybe than I thought I would be. Um, I, uh, yeah, like for a while I was, I was, uh, very strongly in like our VS code extension. Like I need, I want that sidebar. I want all the code there next to me. I feel like these things should be, you know, together. I'm not a, I'm not a person who doesn't look at the code. I'm not a, like a, I don't. For for projects that are like true prototypes that are throwaway, like I will actually not look at the code. It's very freeing. I understand. I'm so excited about it. But for the code that goes into Codex itself, I'm like, no, I still need to look at this, right? This is pretty important. This is like what's gonna, you know, affect everybody else.

• 51:57Speaker 00

  But, you know, you start to get a sense of, okay, I like, I'm confident the model's gonna be able to do this. You know, this this change and this sort of thing like I, you know, I don't need to babysit it. I'm gonna just like write a lot up front. And so, you know, I have like four or five clones of the Codex repo of my machine. And I I have enjoyed in the Codex app, like the multitasking I just is is just a lot easier. Cause now you're just kind of hanging out in one window. And, you know, I try and it's, it's the game exactly, but it's like, you know, it's like, how much throughput can I get? Right? In terms of how many balls can I juggle in the air? Sometimes it feels like, um, it, you know, it can't feel like a little hectic when you're really trying to.

• 52:40Ryan Peterman?

  Content context.

• 52:42Speaker 00

  But at the same time, you're like, I I'm getting a lot more done, you know, and, uh, and that, you know, I almost feel a little bad writing code by hand sometimes because you're like, I could have, if I had asked this in the right way. I I, you know, it's like, it's like when you started, it was like, oh, I'm just gonna change these three lines. And then like, you know, 30 minutes later, you're like, oh, I kind of, you know, but, you know, we all, we all like to type still, I think, and that sort of thing.

• 53:08Ryan Peterman?

  What percent of your code would you say is human written versus the model generated these days?

• 53:14Speaker 00

  Oh, man, it's models that would be like 80 to 90%. I mean, yeah, yeah. I mean, like, especially like debugging a test or like the CI thing is bad. Like, I'm like, I'm like, hey, you know, you know how to write, you know, print debug whatever like, and that's great. That's really freeing.

• 53:32Ryan Peterman?

  Digging into the problems that are suitable for LLMs and the ones that are not, like, what? What do you need to see where you think, okay, I need to go in and, you know, write it myself? What's that 10%? And also, what's in that 80, 90%?

• 53:47Speaker 00

  Yeah, I mean, that's a good one. I think about that, you know, because every time I sit down, I'm like, like, should I write this? I mean, the answer is almost always no. Um, I, there's.

• 53:59Speaker 00

  Things that are like lower level and someone, you know, so Codex, the harness, the part that I spend my time on is in Rust. And that means that, you know, we can do, and we do do like operating system specific things in that code base.

• 54:15Speaker 00

  And actually, and I spend a lot of time on the sandboxing, right? So the thing that really upholds like, you know, the security integrity of what of what we're doing and the model can't go outside. You know, the bounds that you set.

• 54:28Speaker 00

  You know, I, I do more of that by hand because I need to make sure, really sure that that's all correct, right? That our air test coverage is is good. Or sometimes I'll, you know, seed it and then, you know, once I've got like the groundwork and like, okay, I have the pieces that I that I. A lot of feelings about and then could let it like fill in the rest and that sort of thing. But, you know, a lot of like refactors and things like that. Actually, I think that like a lot right now is like. You know, have a build up a big, you know, PR that does too many things, but I know it does too many things. So they're like, okay, please, you know, please split this up into reviewable sized commits. Things like that. Like, I think about how much. Time I probably spent on that sort of stuff before and now I can.

• 55:12Ryan Peterman?

  Right. I'll get freeze you up. What about like code review? Is there? Like what percent of lines of code do you? And at OpenAI generally, are people reviewing manually or are there agents that are like, yeah, imagine you write at rest or something. I don't really need to review that. I mean.

• 55:31Speaker 00

  Yeah. I would say I liked the approach where, you know, the, the agent should do. You know, multiple rounds of review, whatever, until it's confident and that it that it's like, I'm, you know, worth a human's time of looking at it. But we still do look at it. Like ultimately before it goes in, I would say generally speaking. I think there's, you know, we have like our agents MD file like everybody else does. There's still. Sometimes you find like a gap in knowledge that needs like that some context that needs to get added back into there or it's. Yeah, just, just things that we haven't yet memorialized that, that the agentness, but like as a human, I still happen to know.

• 56:11Speaker 00

  Um, and things like that. So we do catch things. Um, but you know, I'll say actually now that people are uh, also using AI to write their like pull request summaries, um, our summaries across the team, it's they're getting way better. So now it's also when I'm going into review, right? It's been reviewed by codex. There's a summary that's like, that we actually make sure it has like the why and the what of why, you know, the reason behind the PR. So that is certainly, I think, helping like the get through these PRs faster, which is good because there's a lot more review to do.

• 56:45Ryan Peterman?

  That's, I mean, that's amazing. I feel like 50% of diffs have like, you know, almost blanked. Test plans, so you know, uh, I don't know, arc build or whatever the, oh, I know, man.

• 57:00Ryan Peterman?

  I want to talk about the, the, the codec CLI that's, that's open source. Why is it open source?

• 57:08Speaker 00

  You know, for something that's that critical to like what's going on on your machine, right? Like that, that like, that's one of the aspects of open source is that I'm not, you know, the most zealous about this sort of thing, but I, but I, I sympathize with it. This idea that like, hey, you're going to put this thing on my machine. I care about what it's doing, right? And I think in this domain in particular, it's really important that people can look at it, you know, and have an idea of what it's doing. Uh, because you know, people have a lot of questions about AI agents and that sort of thing. Um, and so I think, I think this, this,

• 57:47Speaker 00

  this area, this domain, I think it's actually really important. Um, and also we, um, you know, we have gotten a lot of, uh, I think like great contributions, above reports and things like that, that we, um, would have missed out on. And then, and then also I think it's, you know, um, just. You know, sharing with the world with like how this is done. So we do it through code, right? I've I've put out one blog post about how the agent loop works. Like, there is a plan to do more of that. I'm actually excited to it. Just, you know, just it's just time. Is this the main factor? So it's a desire.

• 58:20Ryan Peterman?

  Really not.

• 58:21Speaker 00

  Yeah, it was funny because I had two candidates who came through and one's like, he's like, AI wrote it, right? I was like, no, no, I I wrote it. And another person came in. He's like, oh, you can tell that, you know, that you did not outsource the okay, good. Right. I was like, oh, thank you, you know. So, yeah.

• 58:38Ryan Peterman?

  You mentioned the blog post. How does Codex find what is available to it in its environment? And when I'm running these things, I'll see it's kind of amazing. It's thinking to itself and like discovering all these things in my terminal. So, yeah, how does that typically work?

• 58:55Speaker 00

  Yeah, I mean, there's, you know, there's a few. I mean, there's obviously what is, you know, when Codex is base training, like it loves to use RipGrep, it uses RipGrep very well to find all sorts of things. And then there's, you know, if you have your agent's MD file and you're saying, hey, in this repo, like, these tools are really important, right? You should use these, right? Or the ReadMe's or whatever. Or obviously, if you use MCP and you associate these MCP servers with your, you know, where you're working on, right? That that that injects the set of tool definitions like at the start of the the conversation, right? So then I kind of, yeah. So that's that's not even like discovery on Codex's part, right? It's kind of just like put it front and center there.

• 59:38Ryan Peterman?

  I see. So some of it's the harness explicitly throwing that into the context. And then there is a big chunk, though, where the model is just doing all the heavy lifting of finding things. Is that right?

• 59:49Speaker 00

  Yeah.

• 59:50Ryan Peterman?

  Kind of like reflecting over your career, the breadth and depth of your work. If we look across all of it, I mean, it's it's insane. You are JavaScript, front end, then you have all these dev tools, you know, build, fuzzy file search, you know, virtual, now you're working on Codex. You, I'm sure you had to continue your engineering education to kind of get through all these projects. What are the top technical books that have helped you educate yourself?

• 1:00:20Speaker 00

  Yeah, I mean, um, you know, so one was this, uh, this book on operating systems. It's like a thousand pages or something like this. It's the Addison Wesley book, trying to remember the author, but it was funny because I was, when I was working on the virtual file system. So I had actually gotten to that point in my career without ever writing C. Like undergrad, like it was like more theoretical, like we just never had to touch it. And then yet I'm like working on a virtual file system project and that was like very low operating system. That's why I was, you know, joke that was the kind of the worst engineer of the project. And, you know, and someone said something and I realized I was like, I don't know what they're talking about. This is kind of embarrassing. And I was just like, you know, what book do I have to read?

• 1:01:03Speaker 00

  And I think my manager, Aaron Kushner at the time was like, he was like, well, there's this thousand page book. And I was like, done. So I bought it, read the thing cover to cover. I like took it to Y with me. I took it everywhere until I finished it. It's, it's kind of amazing and sad or weird, like how, um, how far many of us can get like in software engineering without really having a clue how computers work. Like there's just so many of those levels of abstraction.

• 1:01:29Speaker 00

  But, you know, on one hand, it's very freeing. And then on the other hand, it's a little bananas. And I think that, um, what I would say to people right now is, um, is, you know, actively trying to go like deeper through the layers and understand these things.

• 1:01:46Speaker 00

  And like, cause many times I saw other people do it and now I can do it is that there are problems some other people could solve that I couldn't solve. Because I just didn't know that like there was this cruff between these two layers. And if you got rid of it, right, you could get like a 10x improvement or something like that. Right. If you, if you're just operating so high up, you don't really know, you know, what you can, what you can break down.

• 1:02:09Speaker 00

  You know, so that, um, and so in terms of books, like, uh, I've enjoyed the, like the O'Reilly, uh, the Rust books. I, you know, big kind of rust. I think that they're also like well-written, um, and thorough. And then honestly, another thing I've been telling people, um, that's not in the book category, but probably more fun. And I learned a lot, uh, actually, uh, are, um, the CTFs are like, uh, capture the flag, like, security-type competitions. And just, you know, it helps with like, kind of like, adversarial mindset. It helps with, I think, uh, they're usually just like, it's like a, it's like a decathlon, like a computer decathlon, I feel like, right? Cause like, like there'll be multiple challenges and maybe this one, you need to like, understand assembly and this one, you need to

• 1:02:54Speaker 00

  understand like, what someone's like, janky PHP admin pages doing and all this sort of thing. And it, and it just forces this, um, breadth on you, uh, in a way that's kind of hard to, to generate otherwise. And it's also, you know, just more fun because it's like a game and that sort of thing.

• 1:03:12Ryan Peterman?

  Can you give some context on what CTF is?

• 1:03:15Speaker 00

  Yeah. So it's, it's, it can be a number of things, but, uh, it's, it's usually a competition, usually in like the InfoSec, the security domain. And there'll be a set of, uh, there's like the Jeopardy-style one, which is like, there's a bunch of challenges, um, and like, you know, designed, like crafted ahead of time. And they all have point values associated with them. And so, you know, there's usually some fixed round of time and it could be individual, it could be with a team. And you're trying to solve these things. And, and there's a flag, there's always like a secret, you know, um, piece of text, um, in there that usually has some format. And the way is that you, basically if you can discover the secret piece of text, that means that you have, the, the challenges set up that you would only have gotten that if you had figured

• 1:04:00Speaker 00

  out, you know, reverse engineered or whatever you were supposed to do. And then you get submit that piece of text and that's your, you know, token to demonstrate that you had solved the challenge. And so it could be like a race to like, solve everything first or get the most points in a certain amount of time or different things like that.

• 1:04:13Ryan Peterman?

  So it's like, it's kind of like an escape room, but in your terminal.

• 1:04:17Speaker 00

  Yes.

• 1:04:17Ryan Peterman?

  Only computers to figure everything out.

• 1:04:19Speaker 00

  Yes.

• 1:04:20Ryan Peterman?

  I see. Okay. And your recommendation is people who want to become better engineers, they should invest in doing some of these CTF because they make you solve problems that make you a better engineer.

• 1:04:31Speaker 00

  Yeah. And you know, and you, you know, also develop skills and things that you just wouldn't have. You know, if you're just like a, you know, writing React every day, like you're probably not gonna like open up GDB and like reverse a, you know, a tic tac toe game. But like I did that because of a, you know, a CTF challenge was to do, you know, exactly that.

• 1:04:49Ryan Peterman?

  Right.

• 1:04:50Speaker 00

  And but then it's like, but then I learned how to use GDB and like, and then then you start, you know, when you're faced with other problems, you're like, oh, my, my just toolkit of, of ways to solve things is just much broader now.

• 1:05:02Ryan Peterman?

  A lot of people, especially earlier in their career, they see the power of Codex doing everything for them in the terminal. And I just imagine them saying, oh, I don't need to learn GDB because Codex knows it. So.

• 1:05:16Speaker 00

  Yeah.

• 1:05:17Ryan Peterman?

  Would, what would your advice be given the landscape of these AI tools for people thinking about their engineering education?

• 1:05:24Speaker 00

  Yeah, no, that's, I mean, I think everyone's like struggling to answer that question right now. I do think about it, you know, a lot. And I, I don't have a great answer. I, I kind of personally come back to my thing about like,

• 1:05:39Speaker 00

  I still think like trying to forcefully kind of go through the levels of abstraction and just understand, you know, more like at a deeper level, how things work is going to be important. I mean, I think this will change. I'm sure this will change over time, but right now it's still kind of like, you know, the, the questions that you asked the agent is going to affect the quality of the thing that you get out, right? So if you're not asking the right questions, you're not maybe going to get the best engineering solution out the other side. You know, as things go on, perhaps that will also be, you know, another layer that's removed. I mean, I think, you know, that's certainly where we're going. I don't know what timeframe we'll get there. Things do seem to be happening faster than we expect, but.

• 1:06:23Speaker 00

  Yeah, I think in general, but.

• 1:06:27Speaker 00

  Learning how to ask what the right question is and I haven't, you know, for myself even like totally pinned down what that means for for somebody who's like starting out new, right? You know, unfortunately have experience to to fall back on or like that's where that that that taste or that

• 1:06:43Speaker 00

  intuition of what to ask has has developed, but you know, if you're if you're starting out, I'm still not sure yet and it's also hard to say because we we don't really know a hundred percent where everything's going.

• 1:06:54Ryan Peterman?

  When you reflect over your career, the expectations at these really high levels is kind of crazy. I mean, it's like already for most people, this E seven or senior staff level is this unattainable level of impact. So someone who gets promoted to that level is thinking, I gotta really work super hard now because the bar is like up here. And then you went two levels past that. And so I I'm curious, like in the day to day now, what what are your thoughts on these like crazy expectations? Is this stressful for you?

• 1:07:30Speaker 00

  I mean, it never, it was never not stressful, I think, for me, because I really, I think also because I sat in calibrations, right? And I, you know, so for a bunch of other people, you know, you talk about their level, you talk about their impact and you know, you want to be really fair and have this integrity around the ball, this level means this thing, this is the impact of like, that's what it means all is, that's when it ceases. And then knowing that like, then you're, you know, playing it out in your mind like, well, someone sitting in my calibration and talking about this thing that they want to be fair, right? And it and it is a little scary, like, you know, you get to E eight and you're like, oh, you know, it's a D one or a director. And you're like, how do I have as much impact as a person who has maybe like over a hundred people in their org?

• 1:08:14Speaker 00

  And so that's hard. A lot of people do it. Again, a lot of people who are in I C who do it, do it more by.

• 1:08:22Speaker 00

  A different form of people management, right, where they're trying to, you know, like, write the right doc and get the people aligned and do that sort of thing. And they're, and the reason that they do it and are not a director is that oftentimes the people, I've talked to people that were in this, like, that they're like, well, I have this technical credibility. Or because I built this thing, like, when I go to the team and talk to them, it lands differently than if, like, a engineering director does. There's some version of that, right, happens a lot. And so then, you know, when you, when you say you're influencing, you know, 50 to 100 people as a senior IC, then you're like, oh, okay, that's like, you know, D1 impact. You know, whereas as if you're a coder IC.

• 1:09:06Speaker 00

  You know, being really thoughtful about the projects that you pick and it can't just be like, this is fun for me type of project. I mean, you know, if you actually care about, you know, not getting fired or getting, or you're getting like Sam Vito or better.

• 1:09:22Speaker 00

  And even when I would start a project, it's certainly like the later I went on, I would think a lot about like, okay, like I, maybe there's this feature, I just want to write it because it's fun. And I'd be like, oh, no, I should let somebody else do that and think about, okay. And it's kind of like with, with Codex now, like, what code do, should I personally write to maximize my impact versus if someone else could do it about as well as me? Let's say 80% is good, I should probably let, you know, have someone else do that. And but, but even then, you know, if you're leading a five-person project, is that still, you know, still getting E8, E9 impact still hard?

• 1:10:02Speaker 00

  So really finding that, that project that's a force multiplier. So I, you know, like, like, the virtual file system was a really great project because that was gonna, you know, we knew that down the road, like, this was really gonna unlock so many things, prevent us from, from being completely blocked, right? Actually, another big part of it is, and, you know, one of my managers talked to me about it is.

• 1:10:28Speaker 00

  You know, not actually recognizing senior managers enough who are the person who pairs the senior IC with the right project. And because some senior engineers are amazing, you know, fixers slash coders, but they're not the idea come uppers with. And they're, but they're the person you want for like, you know, very difficult technical projects. I don't want to like out anybody here, but I have people in mind. And but, but, you know, a lot of times it's the manager who realizes like, oh, this project needs this person, right? And like that person would have never realized it themselves.

• 1:11:07Ryan Peterman?

  One of your old colleagues, Adam Erst, I think I asked him, what are some other engineers that you admire and why? And he mentioned your name, of course. And specifically, he talked about your ability to start projects was really good. And obviously, I mean, look at all these projects, right? I mean, a lot of them, you created them out of nowhere. Like there was just, you had an idea and you went off and built a prototype. You came back and you were very convincing that was a better solution. Do you have any advice for engineers who they have a problem and a solution and they want to like build a project from scratch?

• 1:11:41Speaker 00

  I mean, I think, I don't know, a lot of good projects, I think, come from being a little bit dissatisfied about, about something, right? And I think, you know, it's funny, like sometimes I, for better or for worse, I was just so charging ahead and just like building the thing that like not really thinking about, like what the best way to do it was. So actually, a really funny example is Google Calendar, so going back really far. I was like, I want weather. I want to have weather, little icons showing the weather in Google Calendar. And I was like, I'm gonna do it, you know? And I had mostly been JavaScript especially and not done any of the backend stuff. And I just like charged through and I cobbled things together and, you know, made it happen.

• 1:12:24Speaker 00

  And then my tech lead was like, wait, how are you, um, how are you storing that information about like the weather and stuff? And I was like, oh, I, I just like, you know, threw a blob of like XML and everything. Like, thank you. He's like, he's like, we should have talked about protocol buffers. And I was like, you know, like binary formats to save bytes that I like, I was just so, you know, set on weather of all things that like, I never bothered to ask anybody if there was like a better way to do what I was doing. I was like, it's done.

• 1:12:51Ryan Peterman?

  So I, I mean, I guess that's a. A unique skill of yours is the digging into the dissatisfaction and solving your own problems. Seems like almost every single one of these projects is, I want my, I want something to happen. This shouldn't be this way. And then you went and you solved it.

• 1:13:08Speaker 00

  Yeah.

• 1:13:09Ryan Peterman?

  Yeah. Um, but I think another unique thing that I see is a lot of people, they, they get to work and they, I don't know. They're, they're in their dev environment. I'm sure there were many other people who saw with the Buck story. For instance, they go, oh wow, this build is really slow. Oh, well, like I guess I'm gonna go to the micro kitchen and get it around call its builds. What gave you the confidence to know that you could make it so much better?

• 1:13:34Speaker 00

  I mean, that one was, you know, I, I have to credit, like, having been a Google, I never worked on the Blaze team. I never knew how that like worked, ever touched their code or anything, but I was like, I know that there's a thing out there that has this shape. And is a lot better than this thing. So that's an existence proof. Whether I could do it or not, like TBD, right? But I think that was helpful on that one. Um, you know, I, yeah, I mean, there's a lot of, I guess, prior art that gives me, you know, confidence and things. And I think, and I, you know, I usually generally identify myself as a coding machine. I guess with Codex, everyone's a coding machine. But, um, that, you know, I felt that I always had confidence, like, you know, build a prototype quickly, right?

• 1:14:14Speaker 00

  And at least answer that question or like the basic, my test my basic hypothesis, like spiritually, should there be a way forward to this thing? That I think should exist. And, you know, generally like, you know, you're, if you are determined, you find a way.

• 1:14:27Ryan Peterman?

  Yeah. I've noticed that pattern. Um, a lot of people go to big companies, they see this world-class infrastructure. And then they go to the other company, oh, you don't have this, you don't have this. And they build their, you know, those new versions of it. I've read a lot of your writing at this point and it is so clear. It's some of the best examples of good technical writing. What advice would you have for engineers who want to write better?

• 1:14:52Speaker 00

  I mean, I think a lot of it is, well, I think reading other good writing is a, is a certainly a good start, right? You start maybe, you know, consciously or subconsciously start to pick up patterns of like, what, what is out there? I think, um, you know, really a high level thinking about like what? What is it that I'm trying to convey? What would someone actually really want to know? And outlining a lot upfront, right? Is a lot of people give this feedback, but it's impressive how really important that is. And just being like, does this set of things like linearly follow? I think is a big thing. And, and then asking yourself, oh, I went from this point to this point. Was that too big of a jump? Is there something that like somebody actually like would have reasonably missed? And I think

• 1:15:39Speaker 00

  personally, I feel like that's I have a reasonable feel for like what where that gap would arise. And like, and then if you can kind of anticipate that and magically put in the example that like someone would have, that someone needed to make that jump. I think like that, at least for technical writing is like, is a big deal.

• 1:15:59Ryan Peterman?

  You have that career note that I love. And at the, at the beginning of it, you lay out this three step plan for impact. Can you explain that three, three stuff on early? I feel like that's a good algorithm for people to use in their careers.

• 1:16:12Speaker 00

  Step one is figure out what you really like to do. Right. And you know, as I mentioned, like, it's good to broaden that, but it's also good to, to be honest with yourself. So you don't end like the quote, the hero quest that they went on that that didn't pan out because I was working on stuff that I didn't really, you know, truly love. And then two, step two was figuring out what your employer is, what's really invaluable to them. Right. And as I talked about at Google, I didn't do a good job of that. I did stuff that I was really excited about, but it wasn't, you know, it wasn't, you know, AdWords for Google or anything like that. And then step three is like, find that intersection and then just really lean into that.

• 1:16:53Speaker 00

  And you know, the more that you can do that, I think the more successful you're going to be. And, and the, and the challenge is sometimes it's not always there. Right. And maybe you have to go, you know, find somewhere else to make that happen.

• 1:17:06Ryan Peterman?

  Okay. And then, yeah, last question for you is if you go back to yourself at the beginning of your career, knowing everything you know now, what advice would you give yourself?

• 1:17:14Speaker 00

  I think I should have been. Open to learning more things sooner and and I, you know, to be a little gentle to myself and I think other people are in that situation is that, you know, there's so much to learn when you're starting. And then whatever your first programming language is, I think it's funny. I think everyone has a soft spot for it, like make excuses for it like ever like, oh, that's a totally good language. And I think it's because it's like the first thing that enables you to do a thing, you know, do anything. Right. And and then it's like, oh, okay, I can finally do something. It's such a relief and um, and but it's also like a hazard because like then you kind of want to hold on to that thing. Cause like you're finally productive and now you're like, ah, it took so long to get to this foothold.

• 1:17:58Speaker 00

  I don't know how long it's going to take to get to the next foothold. So I think like, I mean, you know, my particular case, I probably did. Maybe went too deep with JavaScript and like, like I said, it took a long time before I wrote NEC. Um, and I think, uh, you know, if I had been a little bit more curious and a little more flexible in terms of what like types of projects I was willing to take on or things I was willing to learn. You know, it came eventually, but I think if that is probably the biggest thing that maybe could have made a shift for me earlier.

• 1:18:25Ryan Peterman?

  Yeah, I I gather from your story, there was um, a point with the X code where you said you you you hated Objective C. And then you you were you were coming up with like ways to compile the Objective C and the Java or Java into Objective C or something like that, maybe. And then, you know, also talking about the C++ for miles. It was like, it it seemed like a very concerted, okay, I'm gonna, I'm gonna learn this as opposed to like just kind of being open to it. So, yeah, yeah, makes sense. Well, maybe with codecs in the future, it'll be less of a hurdle for people. You could just kind of say, hey, I know JavaScript, write this in Rust or whatever.

• 1:19:04Speaker 00

  No, it's true. Opens a lot of doors, for sure.

• 1:19:06Ryan Peterman?

  Awesome. Well, thank you so much for your time. I appreciate it.

• 1:19:09Speaker 00

  All right. Thank you, Ryan.

• 1:19:10Ryan Peterman?

  Thank you for listening to the podcast. It's a passion project of mine that I've really enjoyed building. Another passion project that I've been working on, kind of in secret, is building an ergonomic keyboard that I wish existed. And I finally have a prototype, so I'd love to show you what we've built. It's ultra-low profile and ergonomic, and I couldn't find anything like it on the market, so that's why we built it. I'll put a link to the keyboard in the description. You can take a look and learn more about the project there. We could definitely use your support. Also, if you have any feedback for me about the show, I'd love to hear it. Comments on YouTube have led to guests coming on like Ilya Grigorik and David Fowler. I wasn't aware of them until someone dropped a comment. Also, feedback in the comments helped me learn to reduce the number of cliffhangers

• 1:19:55Ryan Peterman?

  in the intros. So your comments definitely make a difference. Please keep letting me know what you'd like to see more of in the show, and I'll see you in the next episode.