How to measure and improve developer productivity | Nicole Forsgren (Microsoft Research, Github)

Dr. Nicole Forsgren is a developer productivity and DevOps expert who works with engineering organizations to make work better. Best known as co-author of the Shingo Publication Award-winning book Accelerate and the DevOps Handbook, 2nd edition and author of the State of DevOps Reports, she has helped some of the biggest companies in the world transform their culture, processes, tech, and architecture. Nicole is currently a Partner at Microsoft Research, leading developer productivity research and strategy, and a technical founder/CEO with a successful exit to Google. In a previous life, she was a software engineer, sysadmin, hardware performance engineer, and professor. She has published several peer-reviewed journal papers, has been awarded public and private research grants (funders include NASA and the NSF), and has been featured in the Wall Street Journal, Forbes, Computerworld, and InformationWeek. In today’s podcast, we discuss: • Two frameworks for measuring developer productivity: DORA and SPACE • Benchmarks for what good and great look like • Common mistakes to avoid when measuring developer productivity • Resources and tools for improving your metrics • Signs your developer experience needs attention • How to improve your developer experience • Nicole’s Four-Box framework for thinking about data and relationships — This episode is brought to you by DX—a platform for measuring and improving developer productivity: https://getdx.com/lenny Find the full transcript at: https://www.lennysnewsletter.com/p/how-to-measure-and-improve-developer Where to find Nicole Forsgren: • Twitter: https://twitter.com/nicolefv • LinkedIn: https://www.linkedin.com/in/nicolefv/ • Website: https://nicolefv.com/ Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • Twitter: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ In this episode, we cover: (00:00) Nicole’s background (07:55) Unpacking the terms “developer productivity,” “developer experience,” and “DevOps” (10:06) How to move faster and improve practices across the board (13:43) The DORA framework (18:54) Benchmarks for success (22:33) Why company size doesn’t matter (24:54) How to improve DevOps capabilities by working backward (29:23) The SPACE framework and choosing metrics (32:51) How SPACE and DORA work together (35:39) Measuring satisfaction (37:52) Resources and tools for optimizing metrics (41:29) Nicole’s current book project (45:43) Common pitfalls companies run into when rolling out developer productivity/optimizations (47:42) How the DevOps space has progressed (50:07) The impact of AI on the developer experience and productivity (54:04) First steps to take if you’re trying to improve the developer experience (55:15) Why Google is an example of a company implementing DevOps solutions well (56:11) The importance of clear communication (57:32) Nicole’s Four-Box framework (1:05:15) Advice on making decisions (1:08:56) Lightning round Referenced: • Chef: https://www.chef.io/ • DORA: https://dora.dev/ • GitHub: https://github.com/ • Microsoft Research: https://www.microsoft.com/en-us/research/ • What is DORA?: https://devops.com/what-is-dora-and-why-you-should-care/ • Dustin Smith on LinkedIn: https://www.linkedin.com/in/dustin-smith-b0525458/ • Nathen Harvey on LinkedIn: https://www.linkedin.com/in/nathen/ • What is CI/CD?: https://about.gitlab.com/topics/ci-cd/ • Trunk-based development: https://cloud.google.com/architecture/devops/devops-tech-trunk-based-development • DORA DevOps Quick Check: https://dora.dev/quickcheck/ • Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations: https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339 • The SPACE of Developer Productivity: https://queue.acm.org/detail.cfm?id=3454124 • DevOps Metrics: Nicole Forsgren and Mik Kersten: https://queue.acm.org/detail.cfm?id=3182626 • How to Measure Anything: Finding the Value of Intangibles in Business: https://www.amazon.com/How-Measure-Anything-Intangibles-Business/dp/1118539273/ • GitHub Copilot: https://github.com/features/copilot • Tabnine: https://www.tabnine.com/the-leading-ai-assistant-for-software-development • Nicole’s Decision-Making Spreadsheet: https://docs.google.com/spreadsheets/d/1wItAODkhZ-zKnnFbyDERCd8Hq2NQ03WPvCfigBQ5vpc/edit?usp=sharing • How to do linear regression and correlation analysis: https://www.lennysnewsletter.com/p/linear-regression-and-correlation-analysis Find all lightning round references at: https://www.lennyspodcast.com/ Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected]. Lenny may be an investor in the companies discussed.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Starting with what is your problem or what is your goal? I would say this is a bigger challenge than most people recognize or realize. Eighty percent of the folks that I work with, this is their biggest problem. Even at like executive levels, teams will have gone off for several months and they're tackling something and they'll come back with uncertainty and they'll say like, well, you told me to improve developer experience. I'm like, okay, what do you mean by this? Are you talking about inner and outer loop? Are you talking about friction? Are you talking about culture? But if you're talking about culture, this is totally different than if you're talking about friction in tool chains. If you're on different pages, you're heading in completely different directions.

• 0:37Lenny Rachitsky?

  Welcome to Lenny's Podcast, where I interview world-class product leaders and growth experts to learn from their hardware experiences building and growing today's most successful products. Today, my guest is Nicole Forsgren. This is actually my first recording back since going on Pat Lee for the past couple of months. And what an awesome episode to get back into the swing of things. Nicole is the developer productivity expert, having written the award-winning book Accelerate. And she's been the co-author of the State of DevOps Report year after year. She's currently a partner at Microsoft Research, leading developer productivity, research and strategy. And she's helped some of the biggest companies in the world move faster, improve product quality, and transform their cultures. In our conversation, we get into the weeds of how to go about measuring and improving

• 1:22Lenny Rachitsky?

  your engineering team's productivity and experience. We talk about the DORA framework and the SPACE framework and how to actually implement them to understand how your engineering team is doing. Nicole also shares benchmarks for what elite companies are at. We talk about why moving faster turns out to be one of the best ways to improve quality and stability, plus pitfalls you want to avoid, and also a preview of a new book that she's working on, and so much more. Enjoy this episode with Nicole Forsgren after a short word from our sponsors. Today's entire episode is brought to you by DX, a platform for measuring and improving developer productivity. DX is designed by the researchers behind frameworks such as DORA, SPACE, and DevX, including Nicole Forsgren,

• 2:06Lenny Rachitsky?

  who is my guest for this very episode. If you've tried measuring developer productivity, you know that there are a lot of basic metrics out there and a lot of ways to do this wrong. And getting that full view of productivity is still really hard. DX tackles this problem by combining qualitative and quantitative insights based on the very research Nicole and her team have done, giving you full clarity into how your developers are doing. DX is used by both startups and Fortune 500 companies, including companies like Twilio, Amplitude, eBay, Brex, Toast, Pfizer, and Procter & Gamble. To learn more about DX and get a demo of their product, visit their website at getdx.com/lenny. That's getdx.com/lenny.

• 2:51Lenny Rachitsky?

  Nicole, welcome to the podcast.

• 2:54Speaker 00

  Thank you so much. I'm excited to be here.

• 2:56Lenny Rachitsky?

  I'm excited to have you here. I actually skip this question usually with guests, but I thought it'd be actually really valuable to spend a little time on your background. You have such a unique role and unique set of experiences. Could you just talk briefly about the things you've been up to in your career, where you've worked, and then what you're up to now and what you focus on these days?

• 3:15Speaker 00

  Sure, and I appreciate the question because you're right. I sort of had this like choose your own adventure background. So I started as a software engineer at IBM. I was writing software for like large enterprise systems, which meant I ended up running them. So I was also a sysadmin, I was tracking the stacking, I was running these really, really large labs. And then I kind of stumbled into this seven-day march for several years. And I was like, there has to be a better way. We're hearing rumors of it, but like management was not buying in. And so I decided to win this battle with data. And I was like, I should go do a PhD.

• 3:55Speaker 00

  And so ended up kind of taking a slight pivot into PhD in management information systems, which some people are less familiar with, but it's basically a cross between tech and business. And I ended up getting a fairly technical PhD. So I went to a school, I went to University of Arizona, which has a very, very technical degree. But I liked that it crossed with business because then I had the ability to make these strong business case statements. Right? So it was like, how is or is the way that we develop and deliver software tied to outcomes at the individual level? Right? Can I be more productive? Can I have better work-life balance? And the team level is the team more productive, is

• 4:40Speaker 00

  the team more efficient? And the organizational level? Right? This is what I was really interested in originally. Do I see better ROI? Do I see better efficiency? Because then I could sell it to people. Right? And so that was really kind of what I originally went into. And then I was a professor for a handful of years because like, if you're doing research, like traditionally, that's, that's the job in academia. I also had a master's in accounting because that really helped me make that kind of like financial tie and understanding financial statements. And then after, you know, a handful of years, kind of walked away from tenure because academia was not convinced that DevOps was a thing. Right? The DevOps wasn't real. And state a DevOps report who we, you know, I was doing with DORA, DevOps Research and Assessment in collaboration with Jez Humble and Jean Kim.

• 5:25Speaker 00

  And we started that work with Puppet. So shout out to Alana Brown for starting that and Nigel Kirsten and the team there. We kind of pivoted away and, you know, Chef, the little configuration management startup at the time, hired me and they're like, we'll give you half time to do research and half time to help our engineering practices improve. That's cool. Yeah. I mean, they were incredible because like, what startup is going to be like, yeah, do research. So I was there for a year and a half and then left to, you know, do DORA full time. We had a, we actually had a SaaS offering. So we continued this data DevOps report just to do the DORA banner. And we had a SaaS offering because so many large companies were like, I want my own customized. Measurement, reading and report.

• 6:10Speaker 00

  And then the joke there, you know, when we met with Gartner, they were like, you know, your superpower here was that you tricked people into strategy, which was not only, how do I benchmark? That was kind of our top of the funnel because everyone wants to know how they compare. But the important thing is what should you do next? What's the most important next step? So it's how do I measure? What do I do next? And that gave me this incredible view into advising large organizations into this transformation journey. And then we built out this amazing partner network because we weren't actually consulting. We just had this SaaS piece, but then how do you act on it? We were then acquired by Google, so I was CEO and co-founder. So I kind of led that acquisition and then the integration and building out these teams in Google.

• 6:54Speaker 00

  And after that point, I joined GitHub, which is, you know, this, the largest developer network. So I had this amazing opportunity to do more grounded and applied research again. I was VP of Research and Strategy. And then I went over to MSR where I kind of wear a couple of hats. So right now I have a research lab there with an incredible team. It's the Developer Experience Lab where we do a bunch of work across productivity, community and wellbeing. And then I also help with Microsoft's, you know, kind of cross-company effort to improve their developer infrastructure. So it's like sort of kind of this, this, this round effort into like, how do I really

• 7:36Speaker 00

  remain engaged in measuring, applying, thinking about this work, both in like very applied concrete pieces and incredibly forward-looking work with MSR?

• 7:48Lenny Rachitsky?

  Amazing. And just to clarify, MSR, is that Microsoft?

• 7:51Speaker 00

  Ah, yeah, thank you. Yeah, MSR is Microsoft Research.

• 7:54Lenny Rachitsky?

  Okay, cool. So you've shared a couple of these terms, DevOps, developer productivity. I'm curious what the term you like to use for this area you focus on? Developer productivity, developer experience, DevOps? What's kind of the way, the best way to think about this?

• 8:09Speaker 00

  I really love that you asked this question because I think they're very related concepts that people sometimes, you know, conflate, but, but I see them as being different. So related but different. So productivity, I think, is, you know, basically how much we can get done and how much we can do over time. And I think that's why it's so important to have this holistic measure, right? Because we can't just brute force it, right? And so that's why when my team and I and a bunch of my peers study productivity, we include this community effect, right? Because software is a team sport, we joke, right? And also why well being is so important, right? Because we see that when you do productivity the right way, we see sustainability, we see well being, we see reductions in burnout.

• 8:50Speaker 00

  Now, developer experience is very related and very tied to this, and it contributes to productivity. But developer experiences, if you think about who your users are, right? Developers really are your users in this software engineering, in the software development piece. And so it's, you know, developer experience is sort of like, what is it like to write software? Is this a friction free process? Is this a very predictable and certain experience, right? Can we reduce this uncertainty and increase the predictability here to contribute to productivity?

• 9:26Lenny Rachitsky?

  And then how does DevOps fit into that? Just so that we kind of have the mental model of these terms?

• 9:31Speaker 00

  People have sort of co-opted the terms. And some people name their tools DevOps. I'm maybe a little more old school. So when I was doing a bunch of my DevOps research, it was the capabilities and tools and processes that we can use to improve our software development and delivery end to end so that it's faster and it is more reliable. So DevOps was kind of this technical, architectural, cultural practices that enable us to do this work better so that it is, yes, like more productive. We have a better developer experience. It was kind of this, like, again, this very holistic picture.

• 10:06Lenny Rachitsky?

  So what I love about this topic is that I've never met a founder or a leader who is not thinking, we need to move faster. We need our engineers to be more productive. We need to get things out the door quicker. We want engineers to be happier. Like, nobody doesn't want that. And so that's why I'm excited to dig into a lot of these things. Is that roughly what you find as well? That nobody's ever like, we're good. We don't need any of this. We don't need to focus on this area.

• 10:32Speaker 00

  You know what? I'll say yes and, right? So.

• 10:36Speaker 00

  And it kind of goes back to like, why I got into this because on the one hand, you won't say anyone who's saying, uh, like we're, we don't really need to go faster. Everything's fine. But at the same time, very often I will come into scenarios or I'll find myself in scenarios where people are like, I mean, it would be nice if we were going faster, but do we really need to? Show me the business case. What's the ROI? Or if we go too fast, we'll have an instability, right? What are our safety measures? Are we going to lose reliability? What is happening? Right? And when I first started like ITIL and ITSM, right? The, the old school kind of change management processes.

• 11:19Speaker 00

  The common knowledge was that you had to have at least a two week wait for change approvals in order to get that stability. Turns out that's not right, right? It was just kind of an old wives tale, right?

• 11:36Speaker 00

  And so we kind of have this weird balance of I want to move faster, but is it worth the investment? What am I going to get for it? Are you sure this is the priority? Or I've been in, I've been in meetings where it's like, oh, yes, absolutely. Right? Like this is, this is a priority, but it's the lowest priority. And I'm like, right? So, so then what we want to do is we want to have these, these kind of pointed conversations or these kind of like Socratic type questions and conversations where it's like, help me understand more what your concerns are. Are your concerns around reliability when you move faster?

• 12:19Speaker 00

  We're not just trying to like all the guardrails down and sprint for no purpose of sprinting. And this is where kind of the DORA and DevOps research program comes into play, where it's, we don't just want to move fast and take all guardrails down. We want to implement good technical practices like automated testing, good architectural practices, so that when you move fast, you are also more stable. Right? We want to be thinking about improving the developer experience so that when we are faster, we are also saving time, right? And then we can highlight a handful of statistics like what is your? Typical time for feature delivery? What is your typical time to first PR? What is your typical time to steady state productivity?

• 13:04Speaker 00

  What is your typical time for code review and PR process? And if we are to do like back of the napkin math, what sorts of time are you spending here? And if we do a rough look at industry, what are your peers spending here? And are we losing time, right? And if we can turn this into a value calculation, what does that look like? So that we can think about the priority and the strategy here. And, and I think that's where it becomes a more focused conversation.

• 13:43Lenny Rachitsky?

  This is a great segue too. Something I was going to get to a little bit later, but let's just get into it, which is the DORA framework. And then there's also the SPACE framework. Can you just talk about what these two are? When you use one versus the other? And then how that essentially helps you measure and then improve productivity and, and developer experience?

• 14:01Speaker 00

  Sure, sure, absolutely. And I'm so glad you brought this up. So DORA is, it's an entire research program. Now, many people, when they hear DORA now, they think of the four keys or the DORA four or the four metrics. And I think that's what the research program and, and the company ended up becoming most known for. And so that was the software delivery performance metrics. And those are, there's two speed and two stability metrics. So the speed metrics are lead time. So how long does it take to get from code committed to code running in production? Deployment frequency? How often do you deploy code? And then the stability metrics are MTTR, mean time to restore.

• 14:45Speaker 00

  So if something happens, how long does it take you to come back? And then change fail rate for every change that is pushed? You know, what's the rough percentage of incidents? Or like that require human intervention, right? Now, the thing that was really interesting is when we started measuring these, we found that they move in tandem. Now, like with very strong significance, right? From a statistical standpoint. Now, what this means is now we say speed and stability move together. Most people only think about this from the speed standpoint, which means when you move faster, you are more stable, which means you're pushing smaller changes more often, right? Because if you're pushing all the time, it's going to be very, very small changes.

• 15:29Speaker 00

  Which means you have a smaller blast radius, which means when you push, you have an error in production, it's going to be easier to debug, right? It's going to be much easier to figure all of that out. Your mean time to restore and mitigate is going to be much faster. But that also means is the reverse. When you push changes less frequently, you will have more unstable systems. Because when you push less frequency, you will have very, very large batch changes, which means you will have a very high, very large blast radius. Which means when you do have a resulting bug error, you will have to disentangle this big, you know, ball of mud, right?

• 16:13Speaker 00

  And figure out which piece actually caused the error, figure all of that out. That ended up being a big surprise, right? Because refer to my prior comment about, you know, ITIL and ITSM. If you're forcing a two-week pause for change approvals, you're causing this batching up of changes. And sometimes people were rating with two weeks is good, a month must be better, or three months must be better, or six months must be better. And I mean, just think about the merge conflicts you're causing, right? You're just causing so many challenges in figuring out how to push this code into production. So many people think of those four metrics. One, because we found that speed and stability move together.

• 16:56Speaker 00

  And two, because we started publishing benchmarks on what this looks like for low, medium, high, and then elite performers for many times. This, I believe, may have been interesting. I'm not sure if it was useful or helpful, but I think it was interesting because it gave people at least something to shoot for, something to aim for. I will definitely say what's most important is knowing where you are and the progress that you're making, right? It doesn't matter if frankly, you're a high performer or you're an elite performer. It matters that you know where you are and you're making progress, right? You know, can you push daily or on demand? Or is your only technical capability that you can push twice a year? Right?

• 17:41Speaker 00

  Just know where you are. And is it a business decision or a technical capability? That's basically what it comes down to.

• 17:46Lenny Rachitsky?

  I'm going to jump in real quick, just to highlight what you're talking with, what you just said, which I think is extremely important and powerful. And people might kind of move on too quickly. I also want to ask you what the actual benchmarks are. If you can share those, whatever you want to share there. But before I ask that, essentially what you're sharing right now is just, I feel like the $64,000 question of this episode is just, how do I move faster as a team? And what I'm hearing is essentially it's ship smaller things is kind of at the core of it. And also, if we're, if quality is low, you're also saying the answer is ship more often, ship smaller things. Is that roughly the message?

• 18:26Speaker 00

  Yes, absolutely. It ends up being much, much safer.

• 18:29Lenny Rachitsky?

  Amazing. So I think that's an extremely important takeaway that I think people would, I don't know. That's surprising to me to hear that it's quality comes from shipping faster. And then also to ship faster and move, help your team move faster. It's ship smaller things and, and just deploy more often.

• 18:47Speaker 00

  Yep.

• 18:47Lenny Rachitsky?

  Amazing. Okay. Great. I know we'll talk more about this, but let me go back to the question I was going to ask is,

• 18:53Speaker 00

  Yeah.

• 18:54Lenny Rachitsky?

  Are you, are there benchmarks you can share just right now that you think would be useful to people? I know you said it was interesting and maybe not as useful to people as you imagine.

• 19:00Speaker 00

  Yeah. So I will admit, I, I only have the 2019 benchmarks top of mind. The team at Google has continued that work since I left. It's been led by Dr. Dustin Smith. Nathan Harvey continues the work. So huge shout out to that team. Many others participate. You can go to dora.dev and find all of the continued reports. They've integrated all of this work, but I will say they've, they've remained fairly consistent. So, so really quickly, I'll share the elite performance. So deployment frequency, you can deploy on demand. Lead time for changes takes less than a day. Time to restore is less than an hour and your change fail rate is between zero and 15%.

• 19:44Lenny Rachitsky?

  Amazing. Okay. I'm writing these down. These are extremely valuable.

• 19:48Speaker 00

  And I will mention, people will say, well, this is kind of like a chunk of time, right? It's not super precise. Precision isn't really super important here, right? Like, I don't, it doesn't really matter if you can, like, if your lead time is, if it's less than a day, it's less than a day, right? Like, that's fine from a business perspective. It doesn't matter if it's like four hours or like four hours and two minutes, right? General categories are fine. Now, I will say like the next category for lead time for changes by the day is if lead time is between a day and a week.

• 20:28Lenny Rachitsky?

  And this is for, for good?

• 20:29Speaker 00

  For high, yeah. Between elite and high. Elite is less than a day and high is between a day and a week. And then it goes between a week and a month and between a month and six months, right? So, so you can ask people and they can tell you, right? They can kind of hunch it.

• 20:44Lenny Rachitsky?

  Mm hmm. And this is from committing code into the repo and then going out into production?

• 20:50Speaker 00

  Mm hmm. To it, to about like ring zero. So you don't, like, don't worry if it's like, oh, well, now we need to think about like the global deploy and like which is the final endpoint. It's like, how long does it take to get through your deployment pipeline? Because are you going to be surprised? Do you have fast feedback loops? How does your deployment pipeline work? Does your deployment pipeline work, right? Or are you going to commit code? Are you going to wait for that final review for about three months? Is something going to happen or break? And when it comes back to the developer because something happened or broke? Because that kind of happens, right?

• 21:34Speaker 00

  Are they going to have to insert themselves back in the code, re-review all the things that happened three months ago? All, so many other things happened. That's incredibly difficult, which, to your prior question, this is how it relates to the developer experience. If something happened less than a day and like it's a surprise and it's not great, but like whatever, right? Something happened downstream and I got to fix it. I'm still sitting in my code, right in my head. I've got that mental model. I know what happened. Maybe it's not great, but it's fine. If it happened three months ago and I get interrupted, first of all, interruptions like suck. That's not fun. Second of all, now I've

• 22:15Speaker 00

  got to like re-remember, reread all of this code, maybe reload an entire new workspace instead of libraries and everything. Cause I, maybe it's a whole quarter ago and like, we thought we were done. And I gotta do the whole thing all over.

• 22:33Lenny Rachitsky?

  If a listener is working at a startup, I imagine they're hearing this and they're like, it takes a day to ship that. We ship it all day, a thousand times a day. I imagine these benchmarks are more valuable for larger companies. Is there a kind of buckets you think about for like, here's the size of company this is meant for? And then do you think about anything differently for a startup? Say, I don't know, 10 people.

• 22:55Speaker 00

  If anyone is only listening to this, I just got the biggest smile because we saw no statistical significance between small companies and large companies. The only statistical significant, statistically significant difference was with retail. I'll come back to that. It's so funny because large companies would say, oh, but this isn't fair for us. We have more complex code bases. We have so many things to do. Small companies just don't have to deal with this. Small companies would come to me and they would say, oh, but this isn't fair. Large companies have so much money. They have so many resources. They don't have to deal with all the things. This doesn't apply to me. So it's like, either way, and on days when I was feeling real snarky, I'd be like, pick your excuse. You've got your

• 23:38Speaker 00

  like dropdown. Now, when I say retail was a bit of an outlier, they had a statistically significant difference. Their difference was that they were actually better. Why? Now, I can't tell you why. I can, in a research paper, we'd have a discussion section and this is where you like get to guess, right? I would surmise, and we do this in the report, that it's probably because retail went through the retail apocalypse, right? If you didn't survive, like if you weren't just killing it, you did not survive. So many retail firms just did not make it through. You had to be at the top of your game. There was no such like Black Friday.

• 24:24Speaker 00

  There's no such thing as not having systems that are performing incredibly well. There's no such thing as not being in the cloud. Because if you cannot make it through, you know, bursting on demand, bursting like magic, sometimes I joke, right? You're not going to make it. And so I suspect if I were to guess, if you're not already a high performer in the retail space,

• 24:51Speaker 00

  natural selection got rid of you.

• 24:53Lenny Rachitsky?

  That is really interesting. That makes a lot of sense. So I'm looking at these thresholds again, and I'm thinking from the perspective of a founder who's just like, I wish my engineering team moved faster. Essentially, you're saying if deploy times, if they deploy more than once a day, if their deploy frequency is on demand, or I think it was hourly was kind of the other bucket. Was that part of it? Yeah. And then their meantime, their failure rate is like less than 10% and their mean time to recovery is less than an hour. Basically, you're doing great. That's kind of the message of this framework at least.

• 25:26Speaker 00

  And if you're not doing it through like brute force and killing yourself. Now, can I, can I jump in here? Because then people are like, but how do I do this? So let's say that like you're not in that category. And you're like, because this is the next, this is the piece of criticism I'll get about Dora, right? People are like,

• 25:47Speaker 00

  well, all you've done is make me feel bad. You gave me these metrics. You've judged me. Now I feel bad. And then I'm like, so there's Dora there. I wrote a book called Accelerate, right? Which is like the first four years of the research. Compiled and put together and expanded and a few things. And I'll joke, there's a whole rest of the book, right? Dora is best known for the four metrics, but there's an entire research program supporting it. So it's not just these four metrics. What we find is that if you improve a set of capabilities, I loved your question around what is DevOps? DevOps is not a tool chain you buy. Marketing teams labeled tool chains DevOps, they wanted your money. DevOps is a set of capabilities. There are technical capabilities, there are

• 26:32Speaker 00

  architectural capabilities, there are cultural capabilities, they are lean management practices that predict speed and stability. And then, speed and stability gives you money, right? Because like, it's your ability to create these features that give you money. So when you work backwards, if you want money, like you get the features fast. If you want the features fast and stable, you do the things. And the things are technical capabilities like automated testing, CICD.

• 27:04Lenny Rachitsky?

  And CICD is Continuous Integration, Continuous Deployment. Is that right?

• 27:07Speaker 00

  Yes. Trunk-based development, using a version control system, right? So, do you have good technical practices? Do you have good architectural practices? Like, do you have a loosely coupled system? Are you using the cloud? Or like, if you're not in the cloud, for whatever reason, are you using the underlying architectural pieces that enable good cloud to do the cloud right? Or if you're in the cloud and you're not realizing benefits, is it because you're doing the cloud wrong? Right? Do you have a good culture?

• 27:48Speaker 00

  So, you know, you don't, you don't just like magically go fast and have stability, right? So, working backwards, which pieces are you struggling on? Now, you kind of noted down the benchmarks. If you go to dora.dev, the team at Google was lovely. We, we worked really closely with the team and they're keeping this updated. You can take a quick check. There's a button there that says quick check. You can plug in where you kind of think you are. Like I said, you can hunch it. And it'll tell you where you are in the benchmarks today and what industry you're in. And then the cool part is it'll say now, like, you'll want to ask yourself, like, where am I struggling? But it'll say for your performance profile and for the industry that you're in,

• 28:31Speaker 00

  statistically, over the last several years, these are probably your constraints, AKA, these are probably the things that you're struggling in right now. Right? Like, for people in finance who are high performers, they tend to struggle with these four things, right? Whether it's like culture or continuous integration or whatever.

• 28:54Lenny Rachitsky?

  I love that you're getting tactical with how to actually improve these already, which is the bread and butter of this podcast. And so we'll link to this quick check. It's just dora.dev/quickcheck.

• 29:03Speaker 00

  And by the way, they do not collect your name. They do not collect your info. There is no, there's no lead, any lead gen, anything. Everything is just there. And then there's deep dives into every single one of the capabilities.

• 29:14Lenny Rachitsky?

  Amazing. And also, your book talks about all these things. So people should go check out the book. Obviously, it's on Amazon. Search Accelerate. Is that right?

• 29:22Speaker 00

  Yep.

• 29:22Lenny Rachitsky?

  Okay, so we were talking about Dora. This may be a good time to talk about space, which I think is a different framework you recommend. What is that all about?

• 29:30Speaker 00

  Okay, so space is a way to measure, we say productivity, developer productivity, but it's a little bit more than that. Space is a good way to measure any type of complex creative work. Now, how do they relate? Let's say you go through the quick check. It points out like four things and you decide you want to improve, continuous integration and culture, right? Well, now you're like, cool, but how am I going to actually measure them? This is where space comes in, because space helps you figure out, space gives you a framework to pick the right metrics.

• 30:12Speaker 00

  Now, some people are like, well, space, you didn't give me the exact metrics. People love DORA because it's like, here's the exact four you need. Well, space is like, when you want to measure something that's complex, creative work, maybe like developer productivity, there's also an example at the bottom for incident management. When you have something you want to measure, it says, within your context, within the metrics you have available to you, here's how to pick. That's what space is good for. Now, we called it space because it stands for the five dimensions that you want to measure. So S is satisfaction and wellbeing.

• 30:53Speaker 00

  So satisfaction and wellbeing is kind of self-explanatory. Now, some people might jump in here and say, oh, well, you're just, you know, touchy feeling. This actually matters because we find that satisfaction and wellbeing ends up being incredibly highly correlated with all of the other dimensions of productivity and doing things well. And as soon as satisfaction and wellbeing, things like sustainability, if you're satisfied, as soon as that starts falling off, other things start to break. So this can be an incredibly strong and important signal. P is performance. This is going to be the outcome of a process. So reliability within DORA, the MTTM or change fail rate, right? Those are both performance metrics.

• 31:34Lenny Rachitsky?

  And so you pick one to kind of measure as performance?

• 31:36Speaker 00

  Yep. A is activity. Anytime you have a count or a number of something, these we see all the time because they're super easy to instrument and automate, right? Number of pull requests, number of check-ins, number of something, that's A. C is communication and collaboration. This can be how people work and talk together. It can be meetings, it can be collaboration. It can also be how our systems communicate together. It can be the searchability of a code base. And then E is efficiency and flow. So this is going to be the flow through the system. It can be the time through the system. If we think about SRE or incident management, it can be the number of hops a ticket takes until it reaches the right person.

• 32:21Speaker 00

  Now, to use space correctly, we want to use at least three dimensions at a time. Because that helps us balance. Turns out Dora is actually an implementation of space. So Dora would be space for mostly that outer loop.

• 32:37Lenny Rachitsky?

  Hmm.

• 32:38Speaker 00

  So again, once you've found something that you want to improve,

• 32:43Speaker 00

  find the metrics that make sense to you. Try to have them be in balance or intention. So you don't like throw something at a whack. But pick three.

• 32:51Lenny Rachitsky?

  So when you say Dora is an implementation of space, one has five buckets, one has four, how does, how do you actually think about that?

• 32:59Speaker 00

  So space is there to help you think about how you want to pick metrics, right? So a lot of time I see people, so we have to step back. I used to advise people on how to pick metrics, right? For years, people would would pull me in to advise on Dora. Or accelerate, right? They would ask me questions, but it ended up being metrics questions a lot. How do I pick the right metrics? To improve what I'm doing, right? Like I said, they had the Dora numbers. They would pick their constraints and they wanted to improve. But how do I improve? How do I measure this? How do I show improvement? And so we would start thinking really critically about which metrics were the right metrics to pick. And I would always say, make sure you pick balanced metrics.

• 33:43Speaker 00

  Make sure you pick pick metrics that are intention.

• 33:48Speaker 00

  And I could say it, but people have a hard time wrapping that around their heads because they kept picking things like number of lines of code. Never picked number of lines of code.

• 33:57Lenny Rachitsky?

  Oh, wow.

• 33:57Speaker 00

  Number of still every month I get an email about this. Number of pull requests, number of commits. And I was like, these are all activity metrics. And so finally I pulled a few of my friends together and I was like, let's come up with a framework to help people think about it. And so there are five broad categories. Pick three, because that will help force you through the mental exercise of, what could I possibly pick? You don't need all five, right? This isn't, we're not playing bingo. We're not playing blackout bingo. You don't need all of them. But try to have at least three across different dimensions. Now, one example here, I was working with a group that wanted to improve their pull requests very generally. They just said, improve pull requests.

• 34:42Speaker 00

  So they were thinking about pinging someone every 15 minutes. And I was like, oh, this is going to be bad. Cause we know from other literature and research like nursing, you'll get alert fatigue, where people will just start tuning out alerts. Either they'll turn them off, or they will just stop hearing them. So like number of alerts, right? They're like, let's just think about number of alerts. And I said, well, but if we think about efficiency and flow, how much time do you have to work on your coding? So those two are balanced.

• 35:18Speaker 00

  So we need to protect time to work as well as code review time, right? Pull, pull request time. And so sometimes we can think about those and then, and then we, I think we added a satisfaction metric. Are you satisfied with the pull request process and the selection of the reviewer?

• 35:39Lenny Rachitsky?

  How do you go about actually capturing and measuring this, say, satisfaction?

• 35:43Speaker 00

  So for satisfaction, I would, I would generally ask, right? Go ahead and ask. Now, the ones that you instrument, you can instrument and pull out of systems all the time, right? Go ahead and grab that string. For a satisfaction metric, I would only pull that like periodically, right? Once every few months.

• 36:00Lenny Rachitsky?

  Like a survey to your engineering.

• 36:01Speaker 00

  Like a survey. Yep, absolutely.

• 36:04Lenny Rachitsky?

  Awesome.

• 36:05Speaker 00

  And don't, don't discount what people say, right? Sometimes I hear, actually not sometimes, a lot of times I hear people say, oh, but people lie. First of all, what is their incentive to lie? Why would they lie about having a bad system? Uh, cause it's bad and they want it fixed, right? If it's absolutely a hostile work environment, they might lie and then tell you it's good, then you have bigger problems, right? Also. Do we ever see bad data from our systems or incomplete data from our systems? That's a lie. But we find ways to deal with it. We see it. We acknowledge it. We look for holes in our system data. We try to deal with it. Right? That's also a lie.

• 36:51Speaker 00

  So I think there are better ways to think about and deal with that. And then try to work with it. Right? Because, and I, I wrote a paper with Mick Kirsten on this several years ago on how

• 37:06Speaker 00

  data from people and data from systems are really important complements because we can get certain insights from people that we'll never get from systems. Right? Like, let's look at lead time from changes, for example, right? From commit to deploy. The speed might be fine, but people might tell you it's taking absolute heroics. Right? It's some ridiculous Rube Goldberg machine. The system will never tell you that.

• 37:31Speaker 00

  Or you could get data on your version control system. I worked with a company several years ago and we found out that there was a significant portion of code that was just not going into any version control system. You're never going to find that out from your systems because it's not in the systems. And it was mission critical.

• 37:52Lenny Rachitsky?

  I can see why people come to you asking for advice on metrics because you have this framework of here's the type of metrics you want. And then I think, and especially from engineering team, there's going to be this like, how do I optimize and make sure I'm doing the right thing and measuring the right things? For someone that wants to do this and, you know, an hour long podcast isn't going to give them all the answers, what do you recommend they go read or go do or look at to help them figure that out?

• 38:17Speaker 00

  One, I, I hate to be this person, but I'll, I'll point to a few of my papers because I will say I write things down because I get asked them so often and I want to make sure it is broadly applicable or like broadly available, I guess. This space paper for sure. It's at ACM and I think the year we published it, it was like the most read paper at ACMQ. Yeah. So we tried to make it as.

• 38:43Speaker 00

  Readable as possible. So the space paper is nice because it outlines this framework and it gives examples of metrics in every single category. And so hopefully people can look at this and they can say, "Okay, here's an example to use here, right? Here are some of the things that I could possibly use." And we're seeing that space is being used lots and lots of different places. Another good one could be the paper that I mentioned with Mick Kirsten. And it was about, we talked about using data from people and data from systems. We wrote it up in the DevOps context because I want to say this was written in like 2016 or 2017 or something, but it helps you think through what types of data are good in which situations, right?

• 39:27Speaker 00

  Because you will never find yourself in a situation when you don't want both types of data. Even teams that I've worked with that are the most advanced, they have absolute instrumentation in every possible scenario in the most detailed way. They will still survey their developers at least once a year because you can get new insights, right? One book that I love, it's a little dense, but it's really interesting that I love, is "How to Measure Anything." And it's by Hubbard. And there are parts of it that are like real stats heavy,

• 40:12Speaker 00

  but he has this portion in the front that's like covering intangibles. And so it's like, what happens when you don't have data? You have no data, you're starting from nothing. What are good ways to hunch data? And I really love that because he covers some really good ground there.

• 40:34Lenny Rachitsky?

  Today's entire episode is brought to you by DX, a platform for measuring and improving developer productivity. DX is designed by the researchers behind frameworks such as Dora, Space and DevX, including Nicole Forsgren, who is my guest for this very episode. If you've tried measuring developer productivity, you know that there are a lot of basic metrics out there and a lot of ways to do this wrong. And getting that full view of productivity is still really hard. DX tackles this problem by combining qualitative and quantitative insights based on the very research Nicole and her team have done, giving you full clarity into how your developers are doing. DX is used by both startups and Fortune 500 companies, including companies like Twilio, Amplitude, eBay, Brex, Toast, Pfizer and Procter & Gamble.

• 41:19Lenny Rachitsky?

  To learn more about DX and get a demo of their product, visit their website at getdx.com/leni. That's getdx.com/leni.

• 41:30Lenny Rachitsky?

  You also mentioned offline that you might be working on a book that will answer a lot of these questions. Is that something you're up for chatting about?

• 41:36Speaker 00

  Yeah, absolutely. So as I mentioned, I tend to write, write things down when I get asked questions on it a lot. And so this is one in particular. So we'll be covering, you know, I'm, I'm starting to go through, and I'm covering some of these. And I think some of the important topics in particular are, you know, starting with what is your problem or what is your goal? And being super, super crisp on it, right? Like, what is it that we're trying to answer? And I would say this is a bigger challenge than most people recognize or realize. Like, I'm making this setup, right? Like, 80% of the folks that I work with, this, this is their biggest problem. Even at like executive levels,

• 42:21Speaker 00

  they'll, they'll ask their team or teams will come back with uncertainty and they'll say like, well, you told me to improve developer experience. And like, okay, great. What do you mean by that? And then teams will have gone off for several months and they're tackling something and they'll come back and they'll be like, oh, that wasn't what I meant. And I'm like, okay, what do you mean by this? Are you talking about?

• 42:43Speaker 00

  Inner and outer loop, are you talking about friction? Are you talking about culture? Cause sometimes they're talking about culture. And if you're talking about culture, this is an incredibly valid answer. But if you're talking about culture, this is totally different than if you're talking about friction in tool chains. Right? And if you're on different pages, you're heading in completely different directions. So like that, that's one thing we cover, which seems obvious, but trust me, it is not. And then even like, how do you, we're going to do kind of a rough version of like, how do you start measuring from nothing? And also the measurement journey. Right? Like, how do you think about the trade offs between and the proportion of measurement between

• 43:28Speaker 00

  subjective data, right? Data from people. So you have as your interviews and you have surveys and objective data, stuff you get from systems. Because when you first start off, you'll be relying much more on data from people because you can get it relatively quickly. But as you kind of transition through this measurement journey, you'll get more and more data from your systems because it's scalable. It can be engineered, you can be doing, you know, much more with it. And also you should be thinking about, you know, don't let the perfect be the enemy of the good. Right? So like, how do we think about this very, very strategically? How do we transition through this? How do we think about what each piece of data is for? And also like lots and lots of examples, right?

• 44:11Speaker 00

  So I have included like example, interview scripts. How do you select people? How do you screen people? Example, survey scripts. What are some of the analyses we should do? And trying to make this incredibly accessible. So like basically anyone can do this. So you do not need to be a data scientist, but if you have one on staff, like you could hand them some of this and just like let them run.

• 44:32Lenny Rachitsky?

  I think this book is going to do extremely well. Definitely come back on when it is out. I think you said maybe yearish kind of timeframe.

• 44:38Speaker 00

  Yeah, probably about a year by the time we get all the way through.

• 44:41Lenny Rachitsky?

  If people want to be notified when it's out, can they sign up on your site for a newsletter or anything like that? Is there any way to kind of be in the loop as it approaches?

• 44:49Speaker 00

  Oh, yeah, absolutely. Yeah, I'll add a link for that. Also, if anyone is doing some of this work now, if they have, you know, major questions that they would love to have me to answer, if they have success stories, if they have case studies, if they have anything that they, you know, would love to be included, you know, I remember when I wrote Accelerate before, there were a couple of folks that reached out after and they would like, "Oh, I wanted to have something included." Now I've learned, right? If there's anything that folks would love to, you know, be, you know, in discussion with me about, I'm always eager to chat and nerd out about, you know, DevX and especially measurement and measurement journeys, so.

• 45:28Lenny Rachitsky?

  Awesome. I usually ask this at the end and I have more questions, but while we're here, how would people reach out to you? What's the best way to contact?

• 45:35Speaker 00

  On my website, I've got info.nicolafie@gmail.

• 45:39Lenny Rachitsky?

  Okay, awesome. Okay, a few more questions.

• 45:42Speaker 00

  Awesome, thanks.

• 45:43Lenny Rachitsky?

  What are the most common pitfalls that companies run into when they're trying to roll out any sort of developer experience, developer productivity, system measurements, improvements?

• 45:53Speaker 00

  I think one I just mentioned, right, like not being clear or not understanding what it is that they're looking for, because then you can have, you know, a thousand flowers bloom and everyone's kind of running in a different direction. I think another one is not pursuing this in both a top-down and a bottom-up structure, right? And I think that can really help drive success. And having, you know, good communication throughout is super, super important, right? So getting your ICs bought in and helping them understand that this is for them, we want to understand what they're doing, knowing what vocabulary

• 46:38Speaker 00

  they use, you know, what terminology they use is super important. And then chatting with leaders, right? And understanding like what their motivations are or helping them understand what the motivations could be. You know, this kind of hearkens back to one of our earliest chats on why I even got into this and how I see two different sides to the conversation on like, why is DevOps even a thing? Why should we even ship faster?

• 47:03Speaker 00

  There are so many people that I talk to that are super passionate about DevX right now. And they're like, how can I convince my executive team this is important? Because their developers are just completely burning out, or they use computers in anger every day. And so it's like, how can we have the right tools to socialize this to our leaders as well? Right? Because this should be a priority. This needs to be a strategic piece. And how can we help pull together the right value points to communicate this? And to understand what their priorities are so that we can see how this fits in, right?

• 47:42Lenny Rachitsky?

  You've been working in this space for a long time, probably longer than anyone that has ever worked on this area of developer experience productivity. What have you seen change most from the time you started working in the space to today? What kind of progress has been made?

• 47:58Speaker 00

  We have these increasingly large, complex systems, right? So like 10 or 15 years ago, like the internet was around, but like, things were really different. Now, almost every company has a really large, complex system, right? We also have a shortage of developers, or at least a reported, perceived shortage of developers. More companies are technology-driven, or at least they understand they're technology-driven. It's like, I understand a handful. I remember a handful of years ago when I met with a financial institution whose CTO insisted to me that he was not a tech company. Like, that, that's not real anymore. That doesn't happen anymore, at least like, very, very rarely.

• 48:39Speaker 00

  So all of these things come together, and suddenly, many more companies are like, we have to be better at this. And that was not always the case, like five to 10 years ago. I used to have to really explain why this was a pressing concern, and why it would continue to be a pressing concern. And now, in the last six to nine to 12 months, we have this AI moment happening. And it just poured gas on top of everything. Because now, what's important, like, we've always said that like, ideas aren't important, execution's important, but now, this is absolutely true. Because it's not just about what it is that you build, it's about...

• 49:25Speaker 00

  Creating absolutely novel, incredibly new experiences and doing them at a speed that no one has seen before. And the only way to do this is to have this software pipeline that is fast and is safe and is stable and is reliable. And that's where we're seeing this really interesting convergence and pressure isn't quite the right word, but it's really forcing the discussion and strategy and prioritization, right?

• 50:07Lenny Rachitsky?

  I'm glad you touched on AI. That was actually exactly where I was going to go next.

• 50:10Speaker 00

  Perfect.

• 50:11Lenny Rachitsky?

  Yeah, obviously productivity, AI engineering, something that's top of mind for a lot of people. There's a lot of layoffs that have been happening. There's a lot of talk of we don't need as many engineers. I actually had dinner not too long ago with a few, I'd say 10x engineers. And those are folks that people sometimes say like, they don't need Copilot. They're not going to use any of these tools. They're already amazing. And they were the opposite. They're like, this is making me 100% more effective and efficient and I love it. So clearly, good things are happening there. I don't know what the question is specifically, but I guess, have you seen the impact of AI on engineering productivity? And has that shifted how you think about developer experience and productivity beyond what you already just shared?

• 50:54Speaker 00

  Absolutely. So yes, and, right? I think this is a super interesting, open question. So can I answer it just with a whole bunch of questions?

• 51:03Lenny Rachitsky?

  Absolutely.

• 51:04Speaker 00

  We're absolutely seeing an impact and we continue to explore this. So like, I have an interesting question to see like, how it'll change the space framework. What's open here? I think a few things will remain, right? Satisfaction is still going to be there. Performance is still going to be there. Activity is still going to be there. How you communicate with people and with the tool, efficiency and flow is still going to be there. I believe it will change and add a dimension like trust or reliability, right? How do I rely? Can I rely on it? Well, I have an over reliance on it. And what we're seeing is that probably unsurprisingly, people really fundamentally shift the way they

• 51:47Speaker 00

  Work when they work with an AI-enabled tool like GitHub Copilot or like, you know, Tab9 or others, because now instead of just writing code or like having a short like autocomplete, you spend more time reviewing code than writing code, right? There's this wonderful paper out, uh, this is the CUP's model. Uh, I'll, I'll share it with you. A team at MSR did it, defines that, you know, about 50% of your time now is spent reviewing versus writing. But it'll be interesting to see how that changes things longitudinally, right? Cause, cause other, you know, some of my colleagues also did a paper that showed that, you know, you can do certain tasks like building HTTP server, you know, 50% faster.

• 52:29Speaker 00

  But I don't think that's what productivity is about when you're using an AI tool, frankly, right? Anyone who's looking at that and like, dear CEOs or whoever who are like, now I can lay off half my workforce. That's not what this is about, right? It's not about taking a task and cutting your time in half because now what we've enabled is your ability to do certain things faster and then free up some of your cognitive space so that you can do harder things with this new Copilot sidecar or something, right? But also because now you're accepting text and then reviewing it, we've changed what your mental model is. So we've changed the friction model that you expect.

• 53:14Speaker 00

  We've changed the cognitive load of what you expect. We're changing reliance on code. So what does this mean for reliance or overreliance? What does this mean for learning? What does this mean for novices versus experts? How do we measure productivity, right? There are a handful of us that are having these discussions on what does this mean? And how do we communicate it thoughtfully? Again, we really need to have these kind of holistic balance metrics because if we, if it's an oversimplification, we really risk losing the forest for the trees, right? But it's also like super interesting and super compelling, I think. How can we think about, you know, learning or like onboarding to new code bases or new languages for folks who already know computational learning?

• 53:57Speaker 00

  I, I think it's also very different for folks who are just learning programming languages and don't already know things like computational thinking.

• 54:05Lenny Rachitsky?

  If someone was excited to kind of go down this road of we're going to focus on developer experience, we're going to focus on helping engineers be more productive, what are the next step or two that they should take in your opinion, just broadly, knowing that you don't know any specifics about the say the company that's thinking about this right now?

• 54:22Speaker 00

  I think if you're like walking away from this podcast and you're like, you know, I'm already working on this, or I think this is a thing that's that's happening, right? I would say just like, go check your work, basically, right? Like, has this been written down? Is there a clearly defined challenge, problem, something? Start there. Absolutely, right? Because that is going to be the thing that reduces confusion the best, right? Absolutely. And then see if there's any data. And data can be very loosely defined, right? Is there any signal that is related to the problem? Like, I'd start there. And

• 55:05Speaker 00

  you can do that. You can do that in a week. You can hunt something down.

• 55:08Lenny Rachitsky?

  Sounds like something you could do in a day.

• 55:10Speaker 00

  Yeah. Well, depending, depending on how scattered things are.

• 55:16Lenny Rachitsky?

  Are there any companies that you look at as good models of they do this really well?

• 55:20Speaker 00

  I think Google does this incredibly well. And and sometimes I hesitate to mention Google because they're like, you know, some people like, well, like, we can't be Google, and we aren't really advanced. But the thing I love about Google's approach is that they've really taken kind of this measurement phase approach to things, right? Even when they roll it out in new places. They're very systematic in how they measure things. They have incredible telemetry and tooling and instrumentation. And they continue to invest time in developer experience surveys, and they triangulate them. And one thing that I also love being able to point out here is if there is ever a disagreement between the surveys and the

• 56:02Speaker 00

  instrumentation, which is incredibly advanced, almost every time, every time that I've ever heard of, the surveys are correct, and not the instrumentation.

• 56:11Lenny Rachitsky?

  Amazing. I have just a couple more questions unrelated to this topic. Is there anything else that you thought you think would be useful to share or leave people with around this general space?

• 56:22Speaker 00

  I would say that

• 56:26Speaker 00

  Like thinking about what it is you want to do is always important, right? Like getting crisp, the ability to communicate clearly is always one of the best things. One of, I think, one of my superpowers and one of the things that I've been working with my teams on doing and kind of teaching them is, and one of the things that's really leveled up our work in general is making your work incredibly accessible. And accessible, not necessarily like the accessibility definition of the word, but making it very easy to understand what you're doing for your key audiences. And so thinking about doing that for anything that, you know, anyone who's listening for all of your work is super important. Right? So who is it that your audience is? What's their role?

• 57:12Speaker 00

  What words resonate with them? And then always being able to translate your work into a few sentences or a paragraph or less or less.

• 57:21Lenny Rachitsky?

  I love it. A lot of the listeners of this podcast are product managers and this is so core to the work of a PM. So I think this is

• 57:28Speaker 00

  Perfect.

• 57:28Lenny Rachitsky?

  speaking very directly to a lot of the listeners. Okay, so just a couple more questions before this podcast asks you a few questions, including just like, what are people asking you for advice often around? And are there any other frameworks that you find really useful? And so there's a couple of things I just wanted to touch on, see if there's something interesting there. The first is you have this framework that you call the four box framework. I'm curious what that is and what it's all about.

• 57:53Speaker 00

  Yes, I love this four box framework. I've used it for years. I actually pulled it out first when I was a professor and I still to this day, get LinkedIn messages from my students saying that it's like the most useful thing they've ever used. So here's what it is. I literally pull this out on napkins at bars at conferences to this day. So here we go. Draw four boxes on a piece of paper, two on the top, two on the bottom. So they'll, they'll be kind of aligned.

• 58:23Speaker 00

  The first two, to the left of them, write the word words.

• 58:29Speaker 00

  And below them, write the word data.

• 58:34Speaker 00

  And then between the two on the top, draw an arrow between them. So it's, so it'll say words, box, arrow, box, right? Does that make it sense? Then on the bottom, it'll say data, box, arrow, box. Okay. So on the top half, this is where if you want to think about measuring something or testing something, you have to start with words. So as an example, let's just say, I think that customer satisfaction gets us more money. Or customer satisfaction gets us returned customers. Let's do customer satisfaction. So at the first box, you'll put customer satisfaction inside the box. And you'll put returned customers in the second box.

• 59:18Speaker 00

  Now, always start with words. Do not start with data. You always start with words. And then you'll go around to a couple of people, stakeholders, managers, others. And you'll say, do you agree with, do you agree with this? Is this actually what we're doing? And it can turn into a sentence.

• 59:36Speaker 00

  And then in the boxes below it, this is your data. How are we going to measure customer satisfaction? It could be a survey. And so like, this is where you'll go and you'll say, like, what data points do we have that could proxy for? What could be our data points for customer satisfaction? And this is where it gets tricky because you could say, well, customer satisfaction could be returned customers, but we think it leads to returned customers, so we can't use that here. But returned customers could be, so like, that's where you kind of like roll this out. So how else would we measure customer satisfaction? I made this hard on myself.

• 1:00:09Lenny Rachitsky?

  Like a CSAT score or NPS.

• 1:00:12Speaker 00

  CSAT, yeah. CSAT, NPS. We could say the amount of money that they spent. It's a stretch. Okay. Now, returned customers. Let's go to the next box. How are we going to measure returned customers? Depending on our context, let's say that this is an online business. We could say that it's returned customers as measured through the website. We could say that it's returned customers. We could just ask them, right? Maybe we have a follow-up survey. Returned customers, maybe we're going to do a stretch here. Maybe we say it's a referral link.

• 1:00:50Speaker 00

  This helps us get super clear on what it is we're going to measure. Now, the reason I like this is because if some of our data, now this data analysis, we'll just do correlations here, right? If we have longitudinal over time, that's fine. You can hand this to like a data scientist. You can hand this to someone and you can say, what data do we have? Let's go run this. If something here falls apart, now you can point to the data boxes and we can get mad about the things in the data boxes and we can say, what's wrong? Is the data poor quality? Are we missing data? Was this a bad proxy? Right? Proxy stands in for something else. Was this, was this ridiculous? Right? One of the things I made up, right? Like it was just a bad idea.

• 1:01:32Speaker 00

  Instead of getting mad at Lenny for his really stupid idea or getting mad at Nicole because this was a really bad idea, we can say, this was problematic. What's wrong here? Or we can go back up to the words at the top and we can say, this is not actually something that is probably going to hold, or this is not something we want to test right now, or this is something instead. And it, it makes things incredibly clear. It helps you communicate what it is you want to do fairly quickly.

• 1:02:04Lenny Rachitsky?

  I love it. Here's mine. Check it out. It's ugly.

• 1:02:08Speaker 00

  Nice.

• 1:02:09Lenny Rachitsky?

  Zoom in, right? Okay.

• 1:02:10Speaker 00

  Now, I will say advanced mode. You can start with the same four box framework and you can say, what data do we have available?

• 1:02:19Speaker 00

  What do we think the relationships are? But then you have to go back up to words and then say, for these data points, and we think that they like represent something and we think, uh, we think this is the relationship between them. What do they represent? If I turn this into a sentence, what do they represent? And then you want to like double check because you know, Spurries correlations, one of my favorite websites and set of charts. So you'll want to go chat with someone, interview, make sure things are actually right. But the challenge is I will see people run every correlation they could think of, but they haven't turned it into a word or a sentence that you could communicate to someone else. They don't do the check.

• 1:03:05Speaker 00

  And and they don't do that before, one, before running the correlations. And two, if it's there, right? All of our data is so interrelated, we quite often will find spurious correlations. But it can be really helpful just to have that laid out, like even if it's just on a post-it, right? To say, what are the things I expect to see? What is this actually testing? What relationship do I suspect is there?

• 1:03:27Lenny Rachitsky?

  Mm hmm. Amazing. There's actually have a newsletter post, a guest post on how to do a correlation analysis and a regression analysis. So folks can read that.

• 1:03:36Speaker 00

  Awesome. Oh, that's so great.

• 1:03:38Lenny Rachitsky?

  Plug and play, all kinds of makes it easy for you. So what I'm taking away from this is this is an awesome framework, especially for thinking about a hypothesis you may have. In this case, it's like customer satisfaction is going to lead to more return customers. Here's how we're going to measure it. And then you basically run the test and see if it's true. And if it's not, maybe you need to pick different metrics, maybe you need to pick a different conclusion.

• 1:03:59Speaker 00

  And within like the DORA framework, we would say if we want to improve our speed and stability, we think improving build time would help. And then how would I measure build time? Right? These are the data points that I have available to us. Yep, to circle back.

• 1:04:13Lenny Rachitsky?

  I love it. It's all connected. Okay. And then last question, I asked you what advice people often ask you for? And you said that it's around making decisions. And I'm curious, what advice do you give people about making decisions?

• 1:04:28Speaker 00

  Yes. So this one comes up in business, but also comes up personally and among my mentees. So many times it starts with, you know, being very critical about your objectives and definitions. But then it comes down to, you know, really clearly defining what your criteria is, what's important? And then among that criteria, what's most important? Some of my friends know I have a decision making spreadsheet.

• 1:04:57Speaker 00

  That I have shared out with a handful of friends on, you know, should you take a job? Where should you move? Uh, what are the different things? Really useful. Should do. It is. It's well, it's funny though, because what's interesting is many times I will like, I'll share it with someone and I've got a couple that are just funny, right? But walking through the spreadsheet is often all you need to do in order to know what the decision is. And by that, I mean, so we walked through the decision. I had one where I was like, where should I move next? Or like, what job should I take? Right? So when I started Dora, I, I did this, uh, starting Dora, I thought was my lowest. Once I walked through this spreadsheet, it became my high. So what you do is you outline like of, of all of your options, what do you want to do?

• 1:05:40Speaker 00

  And then you say, what are the criteria that are important to me? So if it's for a job, is it something like, uh, total comp, cash money, prestige, team, job predictability, work-life balance? Identify the criteria that are most important to you. Now it's really interesting because sometimes I will only get that far when I'm working with someone I'm mentoring or coaching and they will say, I know what my answer is. We don't even get to the next step. But just identifying the criteria that are important is it? Now, when I was thinking about where I wanted to move next, it was proximity to an airport,

• 1:06:25Speaker 00

  the relative tech scene, the food scene, that was real high for me. Uh, you know, a handful of things that was important. Now, the next thing I do is for each criteria, what's their relative weight? What's their importance? And I make it add up to a hundred percent.

• 1:06:45Speaker 00

  And then I like, this is the easy part, right? Like you just put it in a little spreadsheet and I, then I give everything to score and I just multiply it out. Now, this is where I'm data informed and I'm not data driven. There have been times I make a decision where, you know, the, the whole like flip a coin and like, whatever it's, wherever it lands on, what your reaction is, tells you what it should actually be. There have been times where like I multiply it out and then I'll actually like fudge the numbers to get what I want, but it's still slightly off. That's where your data informed. Same thing in business. There are many times where you actually run the numbers and it'll give you a class or a category of things and then you choose.

• 1:07:27Speaker 00

  Now, this is where, you know, one of my favorite quotes I heard somewhere about strategy comes into play. Right. And that's that the key to having a good strategy is knowing what not to do. And the key to executing a good strategy is actually not doing it.

• 1:07:46Speaker 00

  So you can have many options, right? As a leader and as an executive, we have many options and we only fund some of them. If you fund everything,

• 1:07:56Speaker 00

  things are going to fail. So being able to think through and identify what your criteria are,

• 1:08:04Speaker 00

  identifying that criteria, what's your selection criteria? What's your evaluative criteria? Ranking them and then deciding what the cutoff is, is important. You can't fund everything. You don't get to pick everything.

• 1:08:19Lenny Rachitsky?

  Amazing. I love the spreadsheet idea. I've made versions of it, but it's always, I think, like you said, a lot of times the exercise is just tell you what you already think and just gives you like, all right, you're right. You probably should just do that thing you already thought you should do.

• 1:08:31Speaker 00

  Yep.

• 1:08:33Lenny Rachitsky?

  Have you thought about making a public template of this spreadsheet? Even though it is simple, I bet it would be really helpful to people.

• 1:08:39Speaker 00

  I have, and this actually might be a good forcing function, maybe.

• 1:08:42Lenny Rachitsky?

  Okay, awesome. So if, if you do it, I'll put in the show notes. It'll probably be near the bottom at the end of the episode, but that'd be awesome. Is there anything else that you want to share before we get to our very exciting lightning round?

• 1:08:54Speaker 00

  Uh, no, I think that's it.

• 1:08:56Lenny Rachitsky?

  Well, welcome to our very exciting lightning round. I've got six questions for you. Are you ready?

• 1:09:01Speaker 00

  Absolutely.

• 1:09:02Lenny Rachitsky?

  All right. First question, what are two or three books that you've recommended most to other people?

• 1:09:08Speaker 00

  We actually had the perfect segue because the book I've recommended absolutely the most is called Good Strategy, Bad Strategy by Richard Rumelt. Another one is Designing Your Life by Bill Burnett and Dave Evans. And the last one is probably Ender's Game. Of course, it's got card. No comment right now on.

• 1:09:30Speaker 00

  Some of his political commentary, but I used to have extra copies in my office when I was a professor and I would just hand it out to my students. It's a fun, like, just easy nonsense read, but.

• 1:09:41Lenny Rachitsky?

  I absolutely love it. Such a, such a good pick. Haven't read in a long time. And are they making a show of that at all? That'd be, that'd be something.

• 1:09:49Speaker 00

  They made a movie and I was afraid I wasn't going to like it. So I just didn't read it because I didn't want it to ruin the book. But at least Harrison Ford was in it.

• 1:09:58Lenny Rachitsky?

  Okay, I'm not going to check it out. They're making a movie of Three Body Problem. I don't know if you've read that, but that is, I'm really excited for that.

• 1:10:03Speaker 00

  It's on my list.

• 1:10:05Lenny Rachitsky?

  Oh, man. Best, best sci fi ever. Next question, actually, very correlated. What is a favorite recent movie or TV show?

• 1:10:12Speaker 00

  I think going through like some real, just easy fun watches lately, uh, I'm rewatching Suits again, but Ted Lasso is a favorite. And I just tore through Never Have I Ever, which is fun because John McEnroe narrates it, which is hilarious.

• 1:10:30Lenny Rachitsky?

  John McEnroe, the tennis player?

• 1:10:31Speaker 00

  Yeah, it's a riot. Yeah, it's so funny.

• 1:10:35Lenny Rachitsky?

  I love it. Next question. What's a favorite interview question that you like to ask people when you're interviewing them?

• 1:10:41Speaker 00

  I love questions that I can kind of spin around hard decisions that people have had to make and how they made them. I love hearing their thought process. And I get a little nervous when people just like YOLO and shoot from the hip constantly.

• 1:10:55Lenny Rachitsky?

  Hmm. So what is it you look for there that gives you a sense that they're someone you may want to hire or work with?

• 1:11:01Speaker 00

  I just like hearing if they have some sort of process, right? If they have some kind of decision-making process, if they have criteria, if they have, like, how do they do evaluation?

• 1:11:11Lenny Rachitsky?

  What is a favorite product that you've recently discovered that you love?

• 1:11:14Speaker 00

  I have a big one and a little one. Um, my big one is probably, uh, Sleep Eight. So I live in Arizona. It gets, it gets hot here sometimes.

• 1:11:21Lenny Rachitsky?

  Eight, Eight Sleep?

• 1:11:22Speaker 00

  Or, yeah, Eight Sleep. Yeah, the other way around. Um, yeah. So that was fun because it, it makes the bed cold and also gives me some, some data, which is probably like a little bit off, but in the approximation is fine. And then Korean face masks. They're just fun. Yeah, you can get some, some pretty good ones for like just a couple of dollars and that's always fun. Self-care.

• 1:11:44Lenny Rachitsky?

  First mention of that one, of Korean face masks.

• 1:11:47Speaker 00

  Right. Listen, everyone get on board.

• 1:11:49Lenny Rachitsky?

  I just did the TikTok. There's a filter now where you could see how you look when you age. And I'm not happy with how it turned out. And so I'm, may I look into this?

• 1:12:00Speaker 00

  I had some basal cell cancer on my forehead a few years ago. And so I am much more careful with my skin. And you can get, like, one of my favorites is Cosar X. You can get 10 for like $15. So it's fun to just like chill at the end of the day with a good, with a good face mask.

• 1:12:15Lenny Rachitsky?

  I was gonna ask you for a specific pick. And so we got one.

• 1:12:18Speaker 00

  Yep.

• 1:12:19Lenny Rachitsky?

  Amazing. This next question, I ask everyone and it's specially appropriate to you, but I don't know if you'll have an answer. What's something relatively minor you've changed in your product development process that has had a big impact on your team's ability to execute? And I feel like you have a big perspective on this. So I'm curious what you have as an answer.

• 1:12:38Speaker 00

  I think I alluded to this earlier. I would say that it's, you know, helping everyone. So I've done this before, but I think it's helping everyone to ask, who's our audience and how will we share this now? And it's sort of interesting because right now I'm wearing two hats. One is at MSR, Microsoft Research. We lead very ambitious research, right? So like H2, H3.

• 1:13:03Lenny Rachitsky?

  Wait, what is H3? The third half?

• 1:13:06Speaker 00

  Oh, Horizon 3. And so it's supposed to be like five to 10 years out, which right now is like, who even knows, right?

• 1:13:14Lenny Rachitsky?

  We're gonna be in computers, living in computers.

• 1:13:16Speaker 00

  Yeah, AI has like completely upended how we kind of think of Horizons. And so when we're, when we're thinking really ambitiously and very, very, very forward looking, what's our check-in? How do we evaluate this? And then how can we easily communicate it to our core audience? And so here, who's our audience and how do we bring the far near? And then for the other hat I'm wearing, I'm working with Octo kind of across all of Microsoft to take a data-informed approach to really improve and up-level our central developer infrastructure. And so as we're thinking very, very tactically, what is our long-term vision and how do we align with several of our broad stakeholders? And so

• 1:13:59Speaker 00

  there, it's who's our audience and how do we bring the near far?

• 1:14:04Lenny Rachitsky?

  I love that. Final question. What is one tactical piece of advice that listeners can do this week to help improve their developer productivity or developer experience and move it in the right direction?

• 1:14:17Speaker 00

  You know, if you walk away from this podcast right now, you could take a look at what's happening in your org today. Is it written down? Is it clear? Do you have any existing data and efforts? And if not, go find a handful of developers and ask them how they feel about their work tools and their work process and what the biggest barriers to their productivity are.

• 1:14:41Lenny Rachitsky?

  Also, pick up a copy of Accelerate on Amazon or your local retail establishment. Nicole, this was amazing. I think we're going to help a lot of companies move faster, have better and happier engineers, which is going to create infinite value in the world. Thank you so much for being here. Two final questions. Where can folks find you online if they want to reach out? And how can listeners be useful to you?

• 1:15:02Speaker 00

  I am on Twitter and Blue Sky, Nicole F V. And my website is nicolefv.com. And my, all my contact information is there. And as we mentioned previously, I'm working on a new project and a new book, digging into exactly these ideas, right? How can we measure better? How can we improve? And what does that measurement process look like? Both for kind of one-time, really quick, you know, unofficial measurement pieces and if we want to do kind of very formal, longer-term measurement pieces. So if anyone is interested in that or has any success stories they'd love to share, I would love to hear more about it. So please reach out and share. I'd love to hear more.

• 1:15:44Lenny Rachitsky?

  Awesome, Nicole. Thank you again for being here.

• 1:15:47Speaker 00

  Hey, thank you, Lenny.

• 1:15:49Lenny Rachitsky?

  Bye, everyone.

• 1:15:52Lenny Rachitsky?

  Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.