Microsoft Distinguished Eng: Why And How To Port Your Entire Codebase to Rust | Stephen Toub

Stephen Toub is a Distinguished Engineer at Microsoft who single-handledly rewrote the GitHub Copilot runtime (400,000-line codebase) in Rust. We talked about why/how he did it, how Rust compared with other languages, and everything he learned in the process. • My ergonomic keyboard project I mentioned, you can follow along here: https://read.compose.llc/ • The Kickstarter page for it: https://www.kickstarter.com/projects/ryanlpeterman/compose-simple-ergonomics-beautifully-done Podcast links: • YouTube: https://www.youtube.com/watch?v=PLMOVlZC4eU • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/e073-stephen-toub Thank you to this episode's sponsor for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at https://workos.com/ Timestamps: (00:00) Intro (01:06) Why the Copilot runtime needed a Rust port (05:38) Why he chose Rust (11:28) Discussing rollout strategies (21:57) How he caught regressions (32:08) Guiding the agents (42:11) Catching regressions before prod (45:16) Which models he used (50:03) The end benefit and costs (01:03:24) When to do a full rewrite (01:06:40) Career learnings on the way to Distinguished (01:16:00) Top technical book recommendation (01:17:49) Advice for young engineers (01:21:00) Outro Where to find Stephen: • GitHub: https://github.com/stephentoub • X: https://twitter.com/stephentoub • LinkedIn: https://www.linkedin.com/in/stephen-toub-75979745/ Where to find Ryan: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman Referenced in this episode: • Stephen Toub’s post on porting the GitHub Copilot runtime to Rust: https://github.blog/ai-and-ml/generative-ai/migrating-the-github-copilot-runtime-to-rust-using-copilot/ • Jarred Sumner’s post on porting Bun to Rust: https://bun.com/blog/bun-in-rust • Code: The Hidden Language of Computer Hardware and Software: https://www.charlespetzold.com/code/index.html

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  All told, it was about 136 billion tokens. It came out to approximately $120,000.

• 0:07Ryan Peterman?

  This is Stephen Tope, distinguished engineer at Microsoft, who single-handedly rewrote a 400,000-line codebase in Rust. And I asked him all about what he learned from the process.

• 0:18Speaker 00

  These projects that were once unaffordable are now quite doable.

• 0:22Ryan Peterman?

  How did you discover and fix regressions along the way?

• 0:26Speaker 00

  Me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective. Something I would do differently if I had to do it again was...

• 0:35Ryan Peterman?

  Here's the full episode.

• 0:41Ryan Peterman?

  I wanted to focus this conversation around the port that you did for the GitHub Copilot runtime and basically rewriting it into Rust, because I think this is a new type of project that is now feasible with the recent models. So I imagine a lot of people in industry are going to attempt something similar, maybe with their legacy codebases, into some more performance language. So I wanted to start off with asking you, what was the project and why did you rewrite everything in Rust? What was the problem that you were solving?

• 1:15Speaker 00

  Yeah, that's a great question, especially because people, sometimes I think, assume rewriting things into Rust is kind of all the rage these days. And so people assume, or have had questions that assume, like, oh, you're doing this just because, or like, you can, and so you will. And that is not at all what this was. There were, you know, there are certain shapes the project could have existed in where rewriting it wouldn't have been a conversation at all. But the impetus is largely that AI has been this incredibly fast-moving target, as we all know. And over the last few years, the capabilities that it has enabled has skyrocketed.

• 1:58Speaker 00

  And so across GitHub, across Microsoft, there are all of these projects that have kind of blossomed from the ground up to incorporate AI capabilities into them. And all of them basically started from, oh, we can make these LLM calls. Oh, we have to wrap a little bit of, you know, goo around it. We have that more around it and more around it. And building up effectively what is now called agentic harnesses, or agentic loops, or agentic runtimes, whatever you want to call them. And we ended up with a lot of them, because everyone kind of had their own homegrown implementation. And that's not a good place to be,

• 2:41Speaker 00

  especially when you're trying to have all of them kind of be best of breed and be on the forefront. And in this world of AI, like, there are new models coming out weekly, if not daily. Like, you know, a couple of weeks ago, there were all these releases from both OpenAI and Anthropic, you know, one-upping each other. And there are new techniques that a month ago were researched, and today, every harness has to have, or they're immediately out of date. And you don't want this myriad of different implementations all needing to stay up to date. And so you really want a small number, ideally one, but a small number of extensible harnesses or runtimes

• 3:23Speaker 00

  that everyone can use and customize and extend to their needs. But the Copilot runtime that was being used for the Copilot CLI and the GitHub app and Copilot Cloud Agent, like, when you're on GitHub and you ask Copilot to do stuff or code review, like, that was written in TypeScript

• 3:47Speaker 00

  using Node, using V8. And that has implications for where you can use that as an embeddable library. So as a, like, one of the main blockers for that runtime being able to be used everywhere I/We wanted to be usable was fundamentally what it was implemented in, the frameworks it was using, the runtimes it was using, and so on. And that, basically, not moving to Rust, but rather moving away from TypeScript and Node to something that was more embeddable, had better efficiency, the lower working set,

• 4:30Speaker 00

  faster throughput, better able to be consumed into any myriad of environments, regardless of what language they're implemented in, regardless of what frameworks they're using.

• 4:43Speaker 00

  That was basically the catalyst for this. And to your point earlier, there are any number of reasons folks might be looking to do these kinds of ports or migrations. They are incredibly common across the industry, whether moving from one deprecated framework to a newer version, or from one language to another, or from a legacy codebase that has become an unwieldy, you know, mess of spaghetti to something cleaner and modern and fresh. These migrations and ports are everywhere. And a lot of times, people just don't take them on because they are too expensive, they take too long. And we would have been in that situation a year or two ago as well.

• 5:27Speaker 00

  But with modern models and modern harnesses, these projects that were once unaffordable are now quite doable in a very small period of time.

• 5:38Ryan Peterman?

  In the post that you wrote, I saw that there was a line that said, "I didn't set out to move to Rust. I set out to move away from Node.js and V8." And that makes me wonder, if you didn't need to go to Rust, did you consider other alternatives? And why did you end up going to Rust specifically?

• 5:57Speaker 00

  So the very first thing I did was basically create a prototype agentically, because you can now create prototypes super fast, in, still with TypeScript, but with different runtimes. So, you know, Node versus Bunn versus whatever, as well as, I think I did Python, Go, C#, Java, and Rust.

• 6:22Speaker 00

  And created a very small end-to-end sort of steel thread through one of these systems, kind of backed by a local chat completion, you know, dummy server. Probably 5,000 to 10,000 lines for each of them. And looked at them in terms of end-to-end performance, in terms of working set, throughput, what kind of scale that we could get if we threw a lot of load at it. And then also, what kind of, whether it could actually meet requirements and what it would take for someone, for us, we have, currently have six language SDKs that are thin bindings on top of the Copilot runtime.

• 7:07Speaker 00

  What it would look like for this to be usable from all of those. And in particular, right now, because we don't want to force, or previously, because we didn't want to force Node into the consuming applications process, the SDK basically spawned another process to host the actual execution of the sessions. But we'd really like, as an option, for it to be in proc as well, like, pretty much every other library in the world. And so, basically evaluating all of these languages and runtimes

• 7:43Speaker 00

  against these performance goals and constraints, and just looking at the data to see what worked and then how well it worked. And for the most part, TypeScript and Python were very quickly out from kind of all of the performance perspective and the embeddability goals. Go had pretty good perf. And, you know, had a pretty clean codebase. But it's not really designed for embeddability. It's not real, like, Interop is not. It's raison d'etre, right? It's, you can do it,

• 8:27Speaker 00

  but you're kind of cutting against the grain. And then, and Java is also not great for that.

• 8:35Speaker 00

  Java's perf was okay.

• 8:39Speaker 00

  The two main contenders ended up being C sharp, in particular, C sharp with native AOT. So that it could be compiled ahead of time down to assembly and Rust. And from a perf perspective, they were both very much on par with each other. Across pretty much all the metrics, in some places Rust was a little better. In some places, C sharp was a little better. And so to your, you know, question earlier, if the project was already in C sharp, we wouldn't be having this conversation. We would have stayed with it. There would have been no reason to rewrite it. But it wasn't, and so that it came down to these criteria. So from a perf perspective, they were both, they both look good. From an embeddability perspective, they both look good.

• 9:24Speaker 00

  C sharp has very strong interrupt characteristics as well. And then it came down to primarily softer factors. So there are other projects around GitHub in particular that are built in Rust and very few that are implemented in C sharp. There are more people around GitHub that know Rust and fewer that know C sharp. That's not the case within Microsoft, obviously, but within GitHub. And so factoring in those kinds of criteria, we ended up going with Rust.

• 9:54Ryan Peterman?

  I know Anders recently did, I think the project started a bit before your project started, but he did a port with the team of the TypeScript compiler from type being written in TypeScript to being written in Go. I was curious if you compared notes with him?

• 10:13Speaker 00

  Definitely. Yeah. And this is a great example where the different projects can have, can reach different conclusions and be right. There's no requirement to embed the TypeScript compiler into other applications. And conversely, the shape of the, the shape of the work that is compiling TypeScript looks very different from the shape of the work that is sending requests to an LLM and manipulating prompts and dealing with sandboxes and all that kind of stuff. The nature of the projects was fundamentally different.

• 10:56Speaker 00

  Although interestingly, in the last few weeks, there have been several agentic ports of, which you can, if you search X or Twitter or whatever it's called now. You know, you can find people that are porting the TypeScript Go compiler into Rust. And doing so very fast with modern models, given the size of the codebase and the cleanliness of the codebase. And so it's interesting, interesting to see.

• 11:28Ryan Peterman?

  So, once you decided that there was motivation for the project and it's clear problem to solve, I wanted to ask you a little bit about the migration strategy. Because I've seen different strategies in different projects. So for instance, you mentioned Bun earlier. Bun, it looked like they kind of built up a giant, basically a new branch that's just completely entirely in Rust. And then they shipped all in, all at once, a stable Rust release. And they did it all in one, one go. Whereas when I was reading what you had done, it looked like a lot more incremental and little by little. How did you think through the different strategies?

• 12:11Ryan Peterman?

  And when might you select one versus the other?

• 12:15Speaker 00

  The interesting thing about Bun, first, I loved Jared, who wrote the Bun post. I loved the post that he wrote. It was really exciting because that post, I wasn't aware that they were doing that port. And then that post came out about halfway through our migration. And it was really cool to see that this other project of approximately, you know, order of magnitude, same amount of code, it was in a different language. They were in Zig rather than TypeScript. But that they had, you know, thought that this was a sensible thing to do and had landed it. And in the post, they talk about it taking, I think 11 days for them to get to like 98% of their tests passing or something like that, which is really cool.

• 12:55Speaker 00

  And to your point, like they did it sort of in a, an en masse, like rewrite the world and go. It is interesting. I am not involved in the project, so I don't know all the details of what's going on behind the scenes. But it is interesting to me that if you look at their NPM page, they were shipping near daily builds. I think, you know, a lot of Canary builds basically through mid-April. Like for a long time, daily builds. And then mid-April, it stops. And then they talk about doing the, you know, the 11 days of porting, which is an impressive feat to get from nothing to this, you know,

• 13:36Speaker 00

  a 500,000 line codebase ported to Rust in 11 days and have a passing 98 or 99% of the tests. But then they didn't have another NPM release of any kind, at least as far as I could see, for another two and a half months. So there was this window where, yes, the core port took a small period of time. But it seems then like there was some aspect of stabilization and things like that happening, such that there was this three-month gap between releases. Which interestingly is also approximately the same amount of time that it took, the other fourteen and a half weeks that I spent porting Copilot. So, you know, there's lots of reasons why they differ.

• 14:19Speaker 00

  But fundamentally, it seemed to be about the same amount of time. And we just chose sort of different ways of going about it. In the post, Jared makes a point about not wanting to do the incremental approach that Copilot did because it would incur lots of temporary interrupt junk. And he's right, it did. We just chose that approach, knowing full well what we were getting ourselves into because of the, sort of the constraints that we were under. So, you know, for a JavaScript runtime, obviously fixing bugs and new features, like that's all valuable.

• 15:01Speaker 00

  But you can pause for three months and the world isn't going to fall over. Right? JavaScript three months ago isn't all that different from JavaScript today. And you can do sort of a rewrite the world approach. Pause everything and then continue shipping when you're done. Agentic harnesses in the summer of 2026. If you pause for three months, the world you ship into will be very different from the worlds that you started in. And so trying to pause for, for a long period, for, you know, a period of time,

• 15:40Speaker 00

  even if it's a few days, was not something that we were willing to entertain. And, you know, even if we've been able to do it in two weeks, that would have been two weeks worth of either blocking the whole team or dealing with a constant influx of, of changes across a very large codebase and having to rationalize and, you know, all of those changes in and merge all those changes in as ports. Instead, we opted for the approach, which I think makes sense for Copilot and made sense for Bun, where we just incrementally moved the codebase

• 16:21Speaker 00

  more and more from TypeScript to Rust in pieces in main, in the shipping codebase, shipping releases daily. And it's just that at the beginning, it was 100% TypeScript and 0% Rust. And at the end, it was 100% Rust and 0% TypeScript. And it was just, you know, more and more over time. And for the most part, it worked out quite well. It did mean that there was, to the point of the post, of the Bun post, there was a lot of temporary code that came in and left. Right? I think we were using Nappy, Nappy RS for interrupt between TypeScript and Rust. And at the peak point, we had several thousand interrupt signatures

• 17:06Speaker 00

  for, I think, 2,000 or 3,000 for calling between them. And then by the end, it had dropped to zero because there was no more interrupt required. So for this project, it made a lot of sense. There are other approaches. You could do a rewrite the world. You could do, you know, have everyone stop what they're doing and just rewrite in main. You could have a feature branch and allow main to keep going and then constantly be taking ports in. And for the incremental approach, you can do what we did. But rather than do the approach of what I chose, which was sort of atomically swap TypeScript with Rust, you could have sort of an A B thing going where you, you have both the TypeScript and the Rust component and you ship with both and you have a feature flag that allows you to ship between them

• 17:51Speaker 00

  and do sort of flights and experiments of those components side by side. There are, you know, various reasons you might choose which. And I think what we chose ended up working out well for us. But

• 18:05Speaker 00

  All the other options would be valid or probably better for other projects.

• 18:09Ryan Peterman?

  So it sounds like that that A B approach, you would also still have some

• 18:16Ryan Peterman?

  temporary testing harness that would be comparing the two legs anyways. In the case where you have a separate branch and then new commits that come in on the old project, or I guess main, need to be continuously ported.

• 18:35Ryan Peterman?

  I imagine that would be relatively doable with modern models as well because each of those single commit ports is just probably trivial for a model. Did you consider doing that one? Or how did the math compare between doing that versus incurring the interop costs?

• 18:56Speaker 00

  So there's a couple of questions there. It's not just about the having a separate feature branch and bringing everything over. If we had a separate feature branch that we were maintaining and at the end, we did this swap where we deploy like the whole new stack, we would only be testing in the wild, everything all at once. And that's a pretty risky gambit.

• 19:23Speaker 00

  Whereas if you, one of the big benefits of the approach that we chose was we were effectively able to use production for the last mile of testing and have each deployment be a small diff on top of what the previous one was. So over the course of the port, I think we shipped about 130 releases, something like that, each one of which contains just incrementally more. And so if there were issues, it was very, pretty straightforward and fast to root cause those issues, even without using a computer at all. Like someone reports an issue like, oh yeah, I know exactly what that is, right? And then you can very quickly narrow it down.

• 20:06Speaker 00

  Whereas the whole rewrite the world, it's harder to do that kind of validation. And there were absolutely regressions that came about from this. A good chunk of those were because of issues with merging in changes from main. Because you know, I still had feature branches. They were just much more time boxed. Instead of being weeks, they were a day for, you know, porting a component. But within that day, there might still be 10 or 20 PRs that came in and touched that component. And a non-trivial percentage of the regressions were places where the agents just missed something

• 20:51Speaker 00

  as part of doing the merge and the port and the merge at the same time. One of the, the nice things about sort of the atomic replacing from TypeScript to Rust versus having an A B thing going is the agents were instructed to delete the TypeScript that they were replacing the moment that they ported it. And that, in a good way, created conflicts for those merges from main. Because it meant that if any code in main touched any of the TypeScript that had been ported, it instantly created a conflict which sort of raised red flags and pointed the agent's attention at that code.

• 21:33Speaker 00

  Whereas if we weren't deleting it, it would have been very easy for more changes in main to that TypeScript to find their way into the existing untouched TypeScript in the branch. And then the agent would have to reason about, you know, all of the code that it had ported, looking for changes to that to bring that in versus it was sort of instantly pointed to those places because we were doing this swap at the time.

• 21:58Ryan Peterman?

  Yeah, I can imagine a lot of people, a lot of teams or engineers that want to rewrite their entire codebase. Probably the biggest concern is the correctness concerns, which is there's so much code that you don't have the luxury of a human even reading the code to confirm that it's correct. So how did you discover and fix regressions along the way?

• 22:23Speaker 00

  So the, the most important thing was our end-to-end tests, our integration tests, or whatever you want to call them. The

• 22:36Speaker 00

  we basically had two flavors of tests for Copilot. We had unit tests, which were largely sort of, you know, they're, they're testing micro functionality throughout the codebase and a lot of them have internals access. So they're able to test this little function here that's not exposed or that piece of code there that's not exposed. And then there are the end-to-end tests, either at the runtime level, the SDK level, or the CLI is in the same codebase. And so its end-to-end tests also effectively exercise the runtime. And it, and this isn't 100% true, but the CLI can largely be thought of as sort of a

• 23:17Speaker 00

  just exposing the runtime's capabilities as sort of the lowest layer possible in a, in a human interface. And so we had all of those end-to-end tests to validate behaviors. And by and large, it's not 100%, but the vast majority of regressions that we had were because we didn't have end-to-end tests covering that functionality. We generally had unit tests covering it, but the problem with unit tests in this situation is they were also implemented in TypeScript, testing TypeScript with internals access. And so those, in many situations, those unit tests had to be rewritten at the same

• 24:01Speaker 00

  time that the code was being ported. Now, because of how we did it, we got the benefit of if I replace this little piece down here and then the TypeScript, so this is now Rust and the TypeScript sitting above it was interacting with this through interop, this guy had its own unit tests. And so indirectly, those unit tests are still validating this ported Rust component. So we got benefits from that. But the safeguard was the, the integration tests or the end-to-end tests across everything. And I had written a document for the team, sort of motivating this port and laying out exactly how we would want to go through it and approximate timeframes and approach and strategies and technology we'd use and so on.

• 24:43Speaker 00

  One of the things that called out was we needed to beef up our end-to-end tests as the very first thing we did. And I did. I think one of my failings and something I would do differently if I had to do it again was I would have done it more.

• 24:58Speaker 00

  Because again, the vast majority, it kind of goes without saying, but if you have tests for something and those tests don't regress, then like the behavior is good as long as you have tests covering all the behaviors you care about. And so the fact that we had any regressions sort of highlights that there were gaps. Now, that's not 100% true. We had cases, the funniest or saddest cases were where we neglected to prevent the agents from rewriting the end-to-end tests at the same time that they were changing the behavior that was being tested. You know, the agents want to get to green, like that's their goal. And so if you let them, they will change the oracle, like they will change the tests

• 25:42Speaker 00

  at the same time as the code to make it pass. Or we had one case where, this one didn't slip through, I noticed this one, but like the, the agent was making changes and it ended up changing some public APIs. And CI caught this and said, no, no, no, you can't do that. But we have an escape hatch in CI where if someone really knows what they're doing, they can put a label on the, the PR to basically say, no, this particular break is okay. And the agent found that and applied the label to say like, oh, this, this break is okay, even though it, you know, it wasn't. And then there are other things that are just really hard for, you, you could have end-to-end tests, but they're sort of in the weeds.

• 26:26Speaker 00

  Things like, you know, you're, you're sending data out on the wire and making sure that every byte that you send in all possible situations is what's expected at the end of the end of the wire. That requires an extensive suite. And if you have any gaps, there's the potential for things to slip through. And then there's also, you know, there's a, TypeScript and Rust have very different threading models and concurrency models. And as much as people like to say that, you know, if your Rust compiles, it's correct and it prevents all these kinds of bugs, that's true, but it doesn't prevent like, you know, all kinds of bugs.

• 27:10Speaker 00

  And there are plenty of concurrency bugs that slip through. And so a bunch of the issues were lifetime issues, concurrency issues, things that the Rust compiler does not, does not catch.

• 27:24Speaker 00

  And when you're coming from a, a largely serialized, single-threaded execution model and going to one where on purpose, we wanted much better scalability, we wanted things to run concurrently, there are plenty of, of issues that, that creep in there.

• 27:41Speaker 00

  So anyhow, to, to your question, the, the number one thing is to make sure that all of the behaviors that you care about are encoded in tests.

• 27:53Speaker 00

  And then let the agents.

• 27:56Speaker 00

  Port the code while maintaining, you know, keeping everything passing and making sure that there are guardrails in place to make sure that they don't change the thing that you're using for the validation at the same time that you need that validation to be correct.

• 28:10Ryan Peterman?

  It still feels like a regression is inevitable because the test coverage is, it's like it's opt in kind of where it relies on a human saying these are the things that are important, but you could always forget something. So I kind of wonder if this type of project, it's just inevitable that there will be some breakage.

• 28:36Speaker 00

  I think so. I mean, it's, it's hard to imagine. I mean, who knows what things will look like in a year. Things are moving really fast, but right now, yes, a hundred percent, there will be regressions. If there aren't, I would be astounded. Unless we're talking about like, you know, a tiny, tiny little code base, but you're talking about rewriting hundreds of thousands or millions of lines of code. It's just law of numbers. If, if the agents get it right, 99.99% of the time, which is probably a nine or two higher than

• 29:16Speaker 00

  most humans, that still leaves 0.01% of the time where they're going to get it wrong in some way, shape or form. And if you're dealing with million lines, that 0.1% is a very real, small, but real number. You can throw more agents at it and we did, you, you know, we just, you know, before merge, doing lots of agentic reviews and then after merge, running agents to continually look for bugs, look for concurrency issues, look for whatever. And they find things, but they never find, it's never, at least in my experience, it's not a hundred percent perfect.

• 29:55Ryan Peterman?

  For such a huge volume of, of code, I imagine most of it wasn't human reviewed. How, how did you handle code review at scale? Or did you do any at all?

• 30:08Speaker 00

  Yeah. So definitely did some, but I did not read every line. That would be impossible. And if I did, it would also be a waste. There's no, like me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective, not just faster, but more effective in quality than me doing so. I, I absolutely looked at the code for every PR that I put up, but I was focusing more on big picture stuff, right? Is the direction of this correct? Is the architecture of this correct? Or is the component, you

• 30:51Speaker 00

  know, componentization of this correct? Is the approach, are the algorithms that are being used appropriate? Is there anything fishy about like, you know, if I, a lot of it is like squinting at a 30,000 foot level and looking at the two code bases and does anything look off, right? And then relying on the agents to focus on the line by line details. And in many situations, I would sort of do a quick scroll through all the changes to just see if anything popped out. And I might notice a few things here and there, but for the most part, a really good agentic reviewer is going to be way better

• 31:33Speaker 00

  at sort of the line level details than certainly me and I would argue most humans. But at the macro level, there are lots of times when I basically stopped the agents and said, whoa, there like this is, this is not the direction to be going. Or the inverse where the agents would stop and be like, we're done. And I'd be like, no, you're not. The, this, this is, this is not what we're, we're going for. And so it was a bigger picture

• 32:06Speaker 00

  than sort of the, the micro level.

• 32:09Ryan Peterman?

  I saw in the blog post that you, you have this cool bar chart where you basically categorized every single message from you to the agents and what your intent was. Looking over that and reflecting on how you guided the agents, where was most of your time spent? And what were the most impactful times where you would step in and change something?

• 32:34Speaker 00

  There were two or three even higher level categories of things that I was doing.

• 32:42Speaker 00

  There was one or two categories where I think this just speaks to the, the state of sort of long horizon tasks with models and harnesses today where it was me being frustrated with their progress in one way, shape or form, and either telling them to keep going in some fashion when they had stopped or telling them to stop when they were going off down, you know, bad paths. But the, the majority were more, were in a category that was more of, there were trade offs to be made. And I wanted to make sure, and reasonable people or reasonable agents could choose multiple paths. And I wanted

• 33:27Speaker 00

  to make sure that we were going down the right one. And a lot of that would be times where I would just have a conversation with the, the main agent to basically evaluate where we were, the decisions that needed to be made and look and kind of talking through, rubber ducking through sort of the pros and cons of those choices, given the current situation we were in. And some of that to the, to your earlier questions about sort of the, the incremental versus sort of pause the world or big bang rewrite approaches. Some of those were less about the end to end and more about where we, like what I

• 34:12Speaker 00

  wanted to do for this one slice that we were putting up. So, you know, the, the agents would pause at a particular place saying, you know, I'm done with, with this component. And I'd look at it and say, this is not a good place to leave things in terms of the, the tendrils that this has into other systems, you know, temporarily, or either maybe we need to do just a little bit more to kind of get beyond this architectural boundary. And then that'll clean up a bunch of stuff. So that when we merge this, the code base is in a reasonable place. A lot of times it would sort of declare success for the

• 34:53Speaker 00

  slice. And if you squint, yeah, okay. But if it were do just a little bit more, or in some case revert and do a little bit less, it would have left the code base in a, in a better place. Or situations where it actually tried to do more than I wanted it to do. Like one of the, one of the rules that I put in place with instructions that wasn't always abided by was I wanted these ports to be as strict ports as possible, not removing functionality, not adding functionality, not making any huge changes and algorithmic approaches, not trying to fix arbitrary bugs that it comes across along the way. I would

• 35:35Speaker 00

  have it note those for me so I could revisit those separately, but I wanted these to be very strictly as one-to-one functionality ports as possible. And there were times when I would see it violating those rules. And I would basically have it back off from that and go back to what it previously had, or at least talk through what it did to make sure that I was comfortable with that particular change being included. A non-trivial percentage of the regressions that we have had were from places where it violated those rules and I didn't notice. And it included fixes for things that shouldn't have actually been

• 36:18Speaker 00

  done. In fact, there was just one, yeah, I, for the most part, we're not seeing any more regressions being reported from the porting phase. We see regressions all the time. That's the nature of merging hundreds of PRs a day into a repo. But, you know, it's just to be expected. And we fix them generally before they ship. And it's just where we see them in main, right? But we had one yesterday or the day before where there was this really corner case thing. And I looked at it as like, that's really weird. Like I don't, it's saying that I added this and I don't remember adding this and looked

• 37:02Speaker 00

  back what it was. So I had, I had created a PR as one of the porting efforts and it wasn't in the initial PR. And then some feedback from a peer or a peer's agent came in recommending doing something. And the agentic response to that was to basically address this security feedback by locking something down. But it locked it down a little bit tighter than it needed to be locked down. And that resulted in some things that had been working before the port, you know, no longer working after the port. And

• 37:46Speaker 00

  Those are really interesting things to go and figure out how they happened.

• 37:52Ryan Peterman?

  That's interesting. So your agent will auto apply and auto listen on the PR. And if it seems reasonable, it may take action without your input. Is that right?

• 38:05Speaker 00

  So for the most part, I use the GitHub Copilot app. And it's got this agent merge functionality where you can basically have it automatically do various things watching a PR. It can automatically rebase if there are conflicts or merge if there are conflicts. It can automatically drive CI to green. So if tests are failing, fix the tests. It can automatically address review feedback that comes in. And it can automatically merge once everything is green. And you get for each of those four things, you can choose which ones you want. For the most part, I leave all three of those checked, except for the last one.

• 38:46Speaker 00

  So that I can, I have a time where after everything is green and before I click the merge button, I go back and I review to see everything that it did to drive this PR to green. What tests did it fix? What feedback came in and how did it address it? To make sure that I know, again, at a high level, I'm not reviewing every single line, what it did. And this particular one, I remember actually reading, but I didn't fully comprehend how, how much it was locking things down and how much it was blocking. Because if I had, I would have said, actually, those cases are okay. Let those through. And had it back off from that.

• 39:30Ryan Peterman?

  If you had all of those checked off, I know, no one would do this because they're on your team. But imagine I was on your team, I could probably craft a PR comment that is malicious, I guess. You know, it's like, hey, it sounds reasonable, but I did request you to delete like everything or?

• 39:49Speaker 00

  Yeah, I did ask you to format your C drive. Did you not format your C drive?

• 39:52Ryan Peterman?

  Yeah, exactly. Just take everything down.

• 39:56Speaker 00

  Yeah. And, you know, certainly, different projects, I would, I would have different boxes of those checked. Like if this was an open source project, the only one I would consider leaving checked would be the CI one, you know, driving, there's no way, and maybe rebasing. But there's no way that I would allow it to automatically address arbitrary review feedback coming in on a PR and certainly not automatically merge.

• 40:26Speaker 00

  On a close, on a private closed source thing where I trust, you know, the, just the people that I know to be working on the project.

• 40:37Speaker 00

  It's, it's less risky. But even then, I still, you know, that less checkbox, you have to be very, very trusting.

• 40:49Speaker 00

  Yeah. One of the things I like about the app as well is it does, there are built in prompts as part of agent merge for what those different things are. But the app does permit me to add my own prompting for those different capabilities. And so I have additional constraints and guidance for the agent as part of doing each of those things about how it should do it. Like, you know, always rebase instead of merge for the, for one of them or for review feedback. Here are the kinds of things I want you to think critically about before you just blatantly accept it. And these are the ones where it's okay. Just go ahead and address it. You know, those sorts of things.

• 41:23Ryan Peterman?

  OpenAI, Anthropic and Cursor all use this product to make their lives easier. WorkOS is an API platform that adds enterprise features like SSO, SCIM and RBAC to your app. These are things that you need to sell to other companies, but aren't the main focus of what you're building. And now they've added more support for AI agents. For instance, Auth for agents gives each agent its own identity with short-lived scope tokens instead of a long-lived API key. So if something goes wrong, you can revoke one session and everything under it dies. Another feature they have is Airlock, which is an authorization layer. Every action agent takes is checked against rules that you can write in plain English. So it's not rubber stamping everything and it's not running unsupervised either.

• 42:05Ryan Peterman?

  And lastly, they have Pipes, which connects your agents to over 300 third-party tools without you ever needing to manage a token. So if you're building SaaS or AI agents and selling to enterprises, WorkOS fills all these gaps in a few lines of code. Check them out at workos.com to learn more and get started. And I appreciate them for sponsoring my show. We talked a bit about regressions and you mentioned that a lot of them were caught before they went out to prod, it sounds like. And I think a lot of people might hear regressions and they think, oh, you're, you're taking down the app or, you know, things are exploding all the time or their incidents. Could you explain maybe the, the further protections after end-to-end testing that made it so these aren't bad regressions?

• 42:51Speaker 00

  I would say probably two thirds of these things that I'm calling regressions were caught very soon after merging to main and before we shipped any pre-release. Even though we're shipping, at this point, we're shipping, I think, three pre-releases a day. So they were caught like very quickly.

• 43:10Speaker 00

  Then there was, there were some that made it into a pre-release and, you know, our form of ring deployments with the CLI is basically pre-release and stable, where pre-release on any given day is probably hitting five to 10% of the, the audience of the customer base. And then, you know, 90 or 95% is only on stable. And so a reasonable number of these issues, I don't know exactly what percentage, 10, 20%, found their way into a pre-release. And then pretty quickly, we'd get an issue opened or someone saying, hey, like, I'm seeing this difference in behavior. Was this by design or not? And so a lot of these were things that were different, not necessarily bad, but different enough that someone noticed

• 43:55Speaker 00

  and then asked us, like, was this on purpose?

• 44:00Speaker 00

  And some of those cases were ones where I'd failed to prevent the agent from fixing bugs as part of the port. And so it, some of them are actually desirable changes, but we hadn't intended to, you know, they weren't purposeful in that regard.

• 44:18Speaker 00

  And then there, there were a number that made it into stable releases. And the ones that made it into stable releases, because we hadn't noticed them in either main or pre-release, they often ended up staying and being unnoticed in stable releases for quite a while. And this is mostly the trickle that we had been seeing over the last, you know, well, we don't really see much anymore. But a month ago would have been the trickle that we were seeing, you know, over a few weeks, where some super corner case thing just behaved slightly differently. And then we get an issue open like, hey, this, this isn't working the way I expected, you know, is this on purpose or not? And oftentimes they weren't and we'd fix them as soon as we heard about them.

• 45:01Speaker 00

  There were also ones that we would see from various telemetry systems, you know, monitoring for exceptions, crashes, things like that. And obviously those are unintentional when we fix those.

• 45:16Ryan Peterman?

  In the post, you, you shared this graphic of which models you used throughout the course of the project. So you have this nice, I guess, bar chart that's showing what portion of each week's, I guess, tokens are consumed by either Claude or GPT and in rare cases, Gemini. At the beginning of the project, it was almost entirely Claude. And then by the end of the project, it was almost entirely GPT. So what made you shift your model of choice in the middle of the project to GPT?

• 45:52Speaker 00

  For the most part, I was constantly trying different ones. And, because, you know, over that period, like both labs were releasing new models at a very fast pace. And so I was experimenting. And when, when I would move to one and I found it to work well, I would stick with it until something bothered me and then I would try a different one. And then, you know, so it doesn't in any way reflect sort of.

• 46:23Speaker 00

  A long-term preference. It was more of at that moment, what was working best for me then? And so at the beginning of it, I think I was on mostly Opus four, six or Opus four, eight. And it was working great. And then GPT, uh, what was it? Five point five, five point six, one of those. And that started working even better for the, these particular workouts. So then I jumped to that. And then Opus five came out and was all of this is cool and jumped to that one. Right? And then, um, recently I'd been having a lot of luck with, uh, six, one, Saul.

• 47:05Speaker 00

  And I'll use, uh, I'll use Opus five, five for code review, uh, for the work that I've done with six, one, Saul. Um, and so I'll like, I'll sort of split the balance there. Sometimes if I get really frustrated with something like both over the course of the port, both models, both the GPT and Claude would exhibit some very frustrating traits at certain points like, um, a particular point, or if I was working on it for some period of time, one of the models might get stuck and kind of like, no matter what I did, it kept kind of refocusing on a particular thing and not stepping out and looking at the forest for the trees.

• 47:48Speaker 00

  And there are various things I would do in those situations. Sometimes I would start up like, keep the code changes, but start over with a clear session and try and use that to, you know, clear context window and try and use that to get it to reset itself. But often it would find itself back into a rut. And so in those situations, I would switch models, um, and then keep going with the new model until it found itself exhibiting some frustrating behavior. And then I would, you know, switch to something else.

• 48:16Speaker 00

  I did experiment using Gemini and Grok as my main drivers, um, but I would often leave those to be, as part of like a multi-model code review. So a lot of the, the reviewing that I had, that I was doing, I would be using either GPT or Claude as for the main work. And then use three or four different models from different families to do a line by line.

• 48:48Speaker 00

  Uh, you know, review of the, the porting that was done. And I would do that very frequently.

• 48:55Speaker 00

  At, at least as frequently as every time that I was rebasing to bring in additional functionality from, uh, for your changes that had been coming into main, but often, uh, even more frequently than that.

• 49:07Ryan Peterman?

  Yeah, I did see that you had a prompt that was, I guess, auto-applied on rebase or something like that, where you requested it to be reviewed, uh, I guess adversarially or, you know, all, all of them at the same time. What made you come to that conclusion rather than, I guess, code review with the same model? But I don't know, different focuses for each thread?

• 49:32Speaker 00

  Yeah, it's, it's basically to get different, effectively different perspectives. You know, every, every little change that you make to a non-deterministic system ends up sending it down, you know, different paths and looking at different things. And one of the biggest changes you can make is completely changing the model. Um, and so the best way to get different opinions and focusing on different things I, that I found is by having, um, different models, especially than the one that I used initially.

• 50:04Ryan Peterman?

  So, um, you know, after you finish the rewrite, what was the concrete impact and the cost? So if you look at it from a business perspective, what was the benefit and the cost of this project?

• 50:19Speaker 00

  So the benefits are stark. I mean, the benefits are

• 50:24Speaker 00

  the ability for all these products and services and teams across GitHub and Microsoft to be able to rely on a shared harness that is constantly sort of, you know, best of breed and getting better and better every day and not have to spend their own resources trying to do the same thing. They can focus on their business value and, and reuse this component. It's the same as with any library that you want to rely on. And this library is a particularly large library. Right? You want to be able to focus on your business value and reuse the constantly evolving work for someone that is spending all their time focusing on, on that thing.

• 51:06Speaker 00

  And so from a business value perspective, it's, it's like incalculable across GitHub and Microsoft, being able to leverage this shared harness. Whereas before the port, a lot of that would have been a harder sell and harder to justify. And, um, part of that is because of very real, uh, costs, uh, like, uh, performance costs. So, you know, something that was taking 10 times as much working set, waving my hands means 10 times less density, which means 10 times as many servers, you know, those sorts of things.

• 51:50Speaker 00

  And, uh, if you can, you know, make the, um, if you can scale the number of users that are able to be on the same machine running concurrently, that has impacts on your, your bottom line. If you can scale the, you know, the throughput and reduce the latency by five or six X or whatever it is, that has real benefits. Um, one of the, the biggest impacts from a perf perspective,

• 52:20Speaker 00

  a, a lot of these systems, especially services will do, um, you know, a, a request will come in and the, the agent loop needs to be started to make one more turn, which could be a very short thing. And then the request is done. And, um, so for these things, the startup cost ends up being one of the dominant factors. And the startup of a node application versus the startup of a, a native Rust or C application is, there's a big difference there. We're talking hundreds and hundreds of milliseconds. Uh, difference, or in some cases, seconds.

• 53:02Speaker 00

  On a per request basis, that latency can be extremely measurable and extremely impactful for bottom line, if you're even able to adopt it. Because you might not be able to, and then you're back in the previous world that I was talking about. So from a business perspective, it's worth a lot to, to do this port. In terms of what it actually cost us, um, it was about, if I, so I, I stored all of my session logs or 99% of my session logs from this effort. And from those session logs, I was then able to sort of deduce the token counts for fresh tokens, cache writes, cache reads, uh, output tokens.

• 53:47Speaker 00

  And all told, it was about 136 billion tokens, broken up in such a way that if you were to use the public pricing, um, of those models at those times, it came out to approximately $120,000.

• 54:08Speaker 00

  This is not factoring any discounts or things that companies might have for what those tokens cost. And, uh, this is just sort of what's on the website of, you know, OpenAI and Anthropic plus my time, which, you know, the, it took, it was about 14, 15 weeks. But I was doing other things at the same time, you know, a lot of these, uh, agentic projects. If I, if I look at my, my copilot app that I currently have on my screen, I've got about 35 sessions right now for things that I'm working on. Some of them in the same area. Some of them are, uh, across different projects.

• 54:50Speaker 00

  And so I'm not, I'm, developers are no longer working on one thing at a time. So if you look at the percentage, you, just as a rough estimate for my time, if you look at the percentage of PRs that I was submitting over that same period, about a fifth of them, give or take, was to do with this, this project and about 80% was for other things. And so if you figure, you know, about 20% of about 15 weeks, you end up with about three weeks of my time spread out over three months. Plus the, the token costs. And it's only getting cheaper. The, the, the models are becoming like at

• 55:35Speaker 00

  cheaper at an, an astronomical rate. I think like when OpenAI released, I'm not going to remember the exact years, but I think like six, one Saul is half the price of six Saul and six was half the price of five, six Saul. Like the, it's just getting cheaper and cheaper and cheaper. And the models are also getting better and better and better, which means that the amount of time that I need to put in to steer the models to do the right thing is also decreasing. And so the costs of these ports, the cost of these migrations is only getting cheaper over time.

• 56:09Ryan Peterman?

  Wow. So three weeks of, uh, I guess of developers' time plus $120,000, you're able to rewrite the entire code base.

• 56:20Ryan Peterman?

  If you were to scope this out pre-AI, let's say you wanted to undertake this, what would that look like?

• 56:29Speaker 00

  First, it wouldn't have happened.

• 56:32Speaker 00

  Um, for, and I'm, I'm actually serious, like for this particular thing, the- It wouldn't have happened because by the time we were done, we would have ported the wrong thing. Like the, this space is moving so fast that we would have basically never finished, I think. In fact, you can kind of see that in the charts in the blog post in that for a good portion of the porting effort, the amount of rust is increasing and the amount of TypeScript is largely remaining the same, but increasing a little bit. And that's because as the port was happening, there was just a huge amount of new functionality coming into the repo, right? And at the beginning of the porting effort, because

• 57:15Speaker 00

  the majority of the codebase was TypeScript, the majority of the new incoming code was also TypeScript. By the end of the project where the majority of the codebase was Rust, the majority of the new incoming code was Rust. But if you look at like the first half of the graph where we're generally 50% or less Rust, the amount of TypeScript was doing this even as the amount of Rust was, you know, doing that. You know, even if that wasn't the case, this, we're talking, it was about 450,000 lines of TypeScript, just production source, never mind tests that were ported. I think there was a few hundred thousand lines of unit tests there in TypeScript as well that also needed to be ported to

• 57:56Speaker 00

  Rust. Just think about the productivity of an individual developer and how much they can reasonably do in a given day if they're top notch. And we're talking a whole team of developers for a year or two to do something like this, right? It is, it is a, if you're able to do it at all, it is, it would have been a monumental investment. And like I, I struggled to think about even writing that proposal because I know that it would be sort of, I would look bad. Like I would be dismissed immediately as of course, we're not doing that. Whereas this was a proposal, it's like, I'm going to take, you know, a

• 58:41Speaker 00

  20% of my time over three months and by the end, this whole thing is going to be moved over. And for the most part, no one else on the team is going to be impacted. And you know, that's not a hundred percent true, but like waving my hands and we'll have all these business benefits in a few months. And it, you know, this will be the bill. That's a much easier proposal to write. And even just the, even if you were paying that actual, you know, non-discounted, public-facing token rates of $120,000, you compare that

• 59:21Speaker 00

  to the cost of a senior developer, you know, for a year with all their benefits and their bonus, like even that is a good deal.

• 59:35Ryan Peterman?

  It sounds like, at least for this type of work where it's super, what we want is very obvious and encoded somewhere in the existing functionality. So like this type of work is at least one order of magnitude cheaper, maybe even more. I mean, if you had a team of engineers and they're, you know, engineers, maybe 200 to 400K or something like that per, this is a drop in the bucket, the 120K. When you were proposing the project, did you have a sense of how much this would cost in tokens before you went and did it?

• 1:00:13Speaker 00

  Yes. And the bill ended up being larger than I expected. Mostly because the line count ended up being larger than I expected. I hadn't fully factored in the growth rate of how much incoming code there would be. So this was, you know, there were, there were a few things. I think I did a lot of things right. I did a few things wrong and you know, 2020 hindsight and all that. One of them I already mentioned, I didn't spend enough time really fleshing out the end-to-end tests to make sure that they were as comprehensive as they could have been. I could have done that better. The other thing I could have

• 1:00:53Speaker 00

  done better was really factor in the growth rate of code that would need to be ported. So when I looked at the, the codebase as part of writing this proposal, it was somewhere on the order of 150,000 lines of production TypeScript, 130, something like that. I don't remember the exact number. And based on that and some back of the napkin calculations and looking at how long it took to create the prototypes that I'd alluded to earlier for, you know, in the different languages to sort of measure perf and things like that. I think I'd estimated about six to eight weeks to get it done. But when you

• 1:01:36Speaker 00

  look at what we actually need to port, I didn't factor in a few things. One, that there were, you know, tens of developers using agents, very quickly augmenting the codebase with lots of a different additional functionality. And, you know, this codebase is one that's, it's not public open source, but it's effectively internal open source. So there's a bunch of folks that come in and add the things that they need. And then the core team is helping to review all that. And so there's this very significant influx of functionality that I hadn't accounted for. I also hadn't accounted for the fact

• 1:02:17Speaker 00

  that the, because of how the codebase came about, the CLI, the copied CLI and the runtime were kind of very intertwined, more so than they ideally should have been. And so in order to port just the runtime, and you know, the CLI can be ported separately if we wanted to, but to actually tease them apart to the point where you don't need the CLI, you don't need to deploy the CLI as part of your application that's using the runtime. There was this effort of teasing them apart, but then there's also an effort of moving functionality that really doesn't need to be in the TUI and that could be shared in the runtime down. And I failed to account for how much of the code that was currently

• 1:03:00Speaker 00

  marked as CLI would actually be beneficial to move down. And so over this effort, there was also code that was coming down. So by the end, we ported somewhere between four and I think 450,000 lines of TypeScript. So about three times what I'd estimated, which impacted both the end-to-end time that it took to do it and then also sort of token counts.

• 1:03:25Ryan Peterman?

  So after having gone through this process, if there was a software engineer out there on some team considering doing some sort of rewrite, what criteria should they look for to decide if it's worth undertaking it or not?

• 1:03:41Speaker 00

  It really comes down to what is the business value? I mean, there's, there's the ones you can do for fun where it's your dime and you get to decide, you know, what fun is worth. Just like, you know, I, I really don't like going to casinos because I really don't like what I consider to be losing money gambling. But friends of mine love it and they consider it paying for a nice worth of entertainment. They're not losing money, they're paying for fun. Right? And so if you're, if you've got your own personal development project and you want to spend tokens on porting that to something else, great, have fun. Like that sounds awesome. But you, but the benefit, you need to understand what the benefits are. The benefits are your enjoyment. When it comes to a business, it's what is

• 1:04:26Speaker 00

  the business value that this is accruing to? And then like anything else, what is, what is it going to cost? And what is the return on that investment? And that needs to be very clear. And there are lots of these migration projects that are people all over the world and businesses all over the world are doing even before agents, where they have done that cost assessment and they have decided that the return on investment is very much worthwhile. I saw this a lot with .NET, with, you know, people moving from .NET framework to .NET Core. And we invested, you know, a lot in making .NET Core incredibly

• 1:05:10Speaker 00

  fast. And so there would be very real performance benefits that would come from doing that migration. And they could figure out the cost associated with, well, we're spending X millions of dollars running our services today and we would save $3 million a year by moving, you know, by doing this port. And this port is going to take a year and it's going to cost us a developer for that year. So they do the math and you know, this is, this is absolutely worthwhile. Even at a more micro level, not even for ports, like what features do you invest in? When I was spending a lot of my time working, for example, on the .NET

• 1:05:52Speaker 00

  Regex engine. I could look to see how much are we spending we as a company? Or how much is this particular service spending on running on regular expressions every year? If I invest two weeks in making these changes to the .net reg ex engine and it makes the the patterns that they're using four times faster. What is the return on investment for my two weeks and how much they save as a service over the course of the year? These kinds of trade offs are everywhere. And you just need to be very clear about why you're doing it, what it will cost and what you get for it. And if you can write all that down and you can justify it, then it's a good project to do.

• 1:06:36Speaker 00

  If you can't justify it, it's very suspect.

• 1:06:40Ryan Peterman?

  When I was doing research for this interview, I looked into your career. Well, on your LinkedIn, there's no information, but I found an interview you did.

• 1:06:48Speaker 00

  Yeah, I'm very, I'm obviously, I've been at Microsoft for 25 years. I love it. I've jumped around a lot internally, but I've had no interest in going to different companies. And so I haven't really invested in keeping my LinkedIn profile up to date.

• 1:07:05Ryan Peterman?

  But so when I, when I did the digging, though, or like, listen to an interview that you had done about the different roles, your path to distinguished engineer is really interesting because you did a bunch of different roles and it wasn't just sui on team A and then sui on team B. You held a variety of ones, starting as almost a, like a forward deployed engineer, consultant engineer. And then you did technical writing at some point. And then you were an advisor to a, a technical advisor to a VP. Oh, and you also jumped to a PM at some point. From all those different roles that you tried, which of them would you say was the most,

• 1:07:50Ryan Peterman?

  like you would recommend someone dabble in that direction if they wanted to try something that wasn't the pure sui role?

• 1:07:57Speaker 00

  Yeah, every role that I did taught me something different. And so from that perspective, like, I have great respect for people that, you know, are come out of college and want to be developers and become an engineer. And that's, and that's what they do. And they just stay on that path. Like, that's a very valid path. For me, I think I picked up a whole bunch of skills by having more of a circuitous route. So like, you, you mentioned as a forward deployed engineer, it was more, at the time was Microsoft's consulting arm. And so I would spend anywhere from three weeks to three months at a particular customer site, helping them solve their problems. And I would learn a whole lot about their industry. I would learn a whole about a bunch, a whole, a whole lot about their particular breed of problem.

• 1:08:40Speaker 00

  I'd learn about Microsoft's offerings from the outside and places that it was great and places that it wasn't so great.

• 1:08:49Speaker 00

  And learn how our customers, like their, their real struggles and what they were dealing with and being able to bring that back to then later building solutions for them was incredibly valuable. When I then worked on like MSDN magazine and was doing a bunch of technical writing and technical editing and figuring out what I wanted to share with people and how I wanted to share it and some amount of development, building things to then talk about, like, I, I gathered a set of skills around how to think through what was really interesting, what was really relevant, how to communicate it in a clear fashion, what didn't need it

• 1:09:34Speaker 00

  to be said, what did need to be said, how to tell a story that would really resonate. From a PM perspective, it became, you know, how do I?

• 1:09:45Speaker 00

  How do we figure out what actually needs to be built? And what are all the pieces that need to come together to really deliver this thing? Obviously, from a dev perspective, all the coding and the design and the, and the architectural and all that kind of stuff. The most interesting role for me, I think, which I am incredibly thankful and glad that I did and will never do again, was being sort of a technical advisor for a VP at Microsoft. It was an amazing experience. It was a couple of years where I was their right hand, sort of like invisible partner,

• 1:10:28Speaker 00

  shadowing everything they were doing and ghostwriting. And after every meeting, having a little, you know, review, kind of going through what we heard and what we think we need to do about it and bouncing ideas back and forth. It was incredibly insightful into how things actually work, how decisions actually get made. Being a part of things that I wouldn't otherwise have been a part of, at least not for many, many years later.

• 1:10:59Speaker 00

  It was awesome. It was also incredibly draining, incredibly selfless.

• 1:11:06Speaker 00

  And it, it took a lot out of me. And so I cherish those experiences and what I learned from them. And I would highly encourage anyone who has the opportunity to do so, to do that kind of thing for a period of time.

• 1:11:24Ryan Peterman?

  That role is really interesting because it sounds like you're, you're basically a chief of staff to someone super high up, but technical. So you're not like an executive assistant. You're like the, you're pulled into all the really, I guess, privileged conversations and you're providing that, that information that they need to make the calls. You mentioned that you learned how things are like really done. Do you have some examples of stuff that someone might learn if they were kind of in that, that role or something that was really helpful for you?

• 1:11:57Speaker 00

  There are lots of times where if you're sort of rank and file engineer at a company and you see these decisions that get made, an email gets sent out about some new path we're following, or some partnership we're forming, or some investment that we're making, some acquisition that is being done, winding down a certain product, investing in a new product, spinning up a new team, like all of those things look like they just sort of come out of thin air and they're, you know, they're just sort of handed down as this is the finished decision and it's fully formed.

• 1:12:39Speaker 00

  You don't really get insights into everything that led up to that point and all of the alternatives that were considered and dismissed and why? And what the trade offs were that were factored in and how experienced people in the industry are thinking through those alternatives and those problems and how, which, which things they're choosing to focus on, which things they consider to be irrelevant.

• 1:13:07Speaker 00

  All of that is stuff that I didn't have exposure to. And being able to be a fly on the wall, just soaking in, not to mix metaphors, but, you know, soaking in all of that. And then being able to have conversations about it to, you know, we'd get out of a meeting and he'd basically say, so what did you hear? What do you think? Like, help, you know, help reason through it. And sometimes I would have valuable insights and sometimes he would say, no. That's, that's not how, how we think about these things. And every one of those was a learning experience for me.

• 1:13:43Ryan Peterman?

  Yeah, I've heard engineers say similarly about the value of trying management, even if they ultimately decide they hate it and they don't want to do it again. As you, yeah, you get to see how the sausage is made a little bit more as a manager. You see all those decisions in managing the org that can be helpful in operating just in your, your career in general.

• 1:14:07Speaker 00

  Yeah. And, you know, I also, I, I'm, I didn't mention it before, but I also did that. Like, I, I thought that what I really wanted to be was a manager.

• 1:14:17Speaker 00

  And so I basically set things up where at a certain point, I trade, you know, I jumped around a lot internally at Microsoft and I changed to a role where I had sort of handshake agreement that six months later, I would be able to sort of shift into a lead position and have a team of people. And I arranged all that and it all worked out and I became a lead. And six months into being a lead, I was like, hmm, I'm not sure if this is actually what I thought it was. And six months after, I was like, nope, it's definitely not. This is not for me.

• 1:14:54Speaker 00

  And then it took me about a year after that further to sort of safely work my way back to an IC role where I've been since. And again, to your point, like, super grateful for the experience, incredibly grateful to Microsoft for having the, for having an individual contributor career track that allows one to sort of progress their whole career and continue to be promoted and whatnot, but still not have to have direct reports. For me, what I realized was the thing that I loved about, or that I thought, the reason I wanted to be a manager was because I thought it was how I could help others and how I

• 1:15:36Speaker 00

  could kind of provide a career guidance or help other people solve problems or whatever it was. And that was misguided on my part. And it took becoming a manager to realize that I could do all of that as an individual contributor. And the things that I, that I loved about being a manager were things that I didn't have to be a manager to do.

• 1:16:01Ryan Peterman?

  Do you have any top book recommendation that maybe impacted you?

• 1:16:05Speaker 00

  Yeah, I mean, it's old at this point, but I've read it several times and I absolutely love it. There's a book by Charles Petzold called Code. It's probably from 2000.

• 1:16:20Speaker 00

  Free, maybe? I don't know, something in that timeframe. But I love it. And it is, one of the things that I really value is being able to explain complicated concepts in a way that doesn't require the person to know anything about what you're talking about. I think that is a valuable skill, even if you're talking to people that do know what you're talking about. Because being able to break down complicated concepts into very clear, concise pieces and communicate that clearly. And then at a 30,000 foot level, it is a, is a great way of getting everyone onto the same page and sort of arguing your

• 1:17:02Speaker 00

  case and pushing things in a direction that you want them to go. And code is a book where Charles basically does that, where he talks about how computers work from the ground up, teaching people that know nothing about computers, that aren't technically experienced, about binary, about logic gates, about circuits, about programming, about computers, about the internet, and kind of building things up from basic principles, from nothing, to a reasonable understanding of how everything works. And I was blown away by the book. And I've since reread it on multiple occasions, even though I know everything that's in there,

• 1:17:44Speaker 00

  I just love the way that it communicates things. So if I had one recommendation, that's it.

• 1:17:49Ryan Peterman?

  Looking back on all your experience, you've kind of been working in the industry for a while. What professional advice would you give a young engineer just entering the industry today that you believe could be helpful?

• 1:18:01Speaker 00

  That's a great question, and it's one I've been struggling with. A couple of years ago, I would have had a much more clear answer. But with AI, I still don't know what the right answer is. A lot of the times that I spend working with AI agents, I'm bringing to bear 25 years of experience, for better or worse, in being able to

• 1:18:34Speaker 00

  very quickly sniff out things when they're doing the right thing, or they're doing the wrong thing, or how I want to guide them. Or an agent says, I'm 100% confident that the right answer is A. And I'll say, yeah, but have you also considered B and C and D? Because these trade offs might not be there, right? And then, oh, of course, like, just those little things, being able to guide them in the right way, requires you to have gone through it. And I'm concerned that because of what AI, how AI agents are progressing, that they're taking away the opportunities to have gone through it. And I don't yet know what the

• 1:19:10Speaker 00

  answer then is. I'm a very firm believer in really learning kind of the basics and building everything up from core principles. And so, like, when, you know, learning a language, I still think it's valuable to learn and to start out with a language like C, because you're understanding how computers actually work, and you understand what memory management means, and you learn about all the various trade offs between this algorithm and that algorithm, rather than working at a layer where a lot of that is obscured from you, where the language and environment are providing a ton of services to you, but you don't necessarily

• 1:19:52Speaker 00

  understand everything that's happening. And so from that perspective, I still think that there's value in focusing on those core concepts, and really internalizing algorithms and how machines actually work, and being, being able to reason and extrapolate from that. And I suspect that that will, in whatever domain you're in, that kind of core primitive knowledge about distributed systems, or about databases, like that foundational stuff will help to be successful in a world where the vast, vast, vast majority of the code is actually being

• 1:20:35Speaker 00

  written by agents and you're helping to guide them. But I don't know that for sure. And at the rate that these agents are learning and evolving, I don't know what the world's going to look like in two or three or four years. I suspect it'll look vastly different than it does today. I think we're in for a wild ride.

• 1:20:54Ryan Peterman?

  Awesome. Well, yeah, thank you so much for your time today, Stephen. I really appreciate it.

• 1:20:58Speaker 00

  Pleasure. This has been a fun conversation. Thank you.

• 1:21:00Ryan Peterman?

  Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Lizkoff, Mike Stonebreaker, Mark Booker, these were all people that I brought on because someone left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard, so there's two sides. This is in the case. But yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units. We're now working on the long journey of

• 1:21:43Ryan Peterman?

  building the tooling now. And so if you still want to pick one up, I've left the late pledges open on Kickstarter, so you can grab one there. I'll put a link in the description. Thank you again for watching the podcast, and I'll see you in the next episode.