20Product: The Five Step Process to Hiring the Best Product People, The Four Core Skills the Best PMs Need to Have, The Two Product Documents that Drive World Class Product Teams & Why the Best PMs are Writers with Scott Williamson, Former CPO @ Gitlab

Scott Williamson was most recently Chief Product Officer for GitLab, where he led a team of 65 in Product Management, Product Operations, Growth, Pricing, and Corporate Development functions. Before GitLab, Scott was VP of Product for SendGrid for over six years, where helped lead the company to a successful IPO and $3B acquisition by Twilio. In Today's Episode with Scott Williamson We Discuss: 1. From Sales to Product Leader: Why does Scott believe sales is a great starting point for product people? To what extent does an MBA help someone wanting to pursue a career in product management? What does Scott know now that he wishes he had known when he started his career in product? 2. What, Who, When: How to Build a Product Team: Is product management art or science? What is the ratio? What are the four core roles of a product manager today? When is the right time to hire your first PM? What is the ideal profile for this first PM hire? What are the single biggest mistakes founders make when hiring PMs? 3. Hiring the Best Product People: What does Scott's hiring process look like for all new product hires? How does Scott test for systematic thinking and problem-solving ability? What questions does Scott always ask in interviews? What are the best case studies to use to test a candidate's skill set? How important is it for the candidate to have domain expertise in your product category? 4. The Best Product Teams are the Best Writers: What are the two different types of documents that product teams must use? How do you know when to use a one-pager vs a six-pager? How does the discussion and planning cycle for the different documents differ? How important is it for PMs to be great writers also?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  There's four buckets for an individual PM. One is around valid, I call it validation. Customer interviewing, product usage data, kind of like gathering insights. Two is build, which is how well do you work with engineering? Can you make trade offs? Can you tee it up for them in a language they understand? Three, the business, do you know which KPI to set? What thing you're working on is going to drive? And third is communication skills. You need to man it up to execs. You need to talk to customers in their language. You need to speak to engineering in their language.

• 0:33Harry Stebbings?

  This is 20 Product with me, Harry Stebbings. Now, 20 Product is the monthly show where we sit down with one of the best product leaders to discuss how they hire, scale and maintain the best product teams in the world. Now, today, I'm thrilled to welcome Scott Williamson. Scott was most recently Chief Product Officer at GitLab, where he led a team of 65 and before that was VP of Product at SendGrid for over six years. But before we dive into this incredible and very granular show with Scott, your website is the front door of your business, but the way teams build and optimize it is broken. Stuck between inflexible templates and cumbersome code-dependent solutions. There is a better and faster way.

• 1:14Harry Stebbings?

  Enter Webflow, a visual-first platform that empowers you to create freely. Experience a CMS where you can build dynamic content without code. Native localization that lets you translate your site in one click to reach global audiences and third-party apps and integrations so you can build faster. Now, you can ship webpages in weeks instead of months and save millions in development costs. These are the real results for companies like Orange Theory, Dropbox and Idio. Get started today at webflow.com. That's webflow.com, more than a website builder. And once you have a great site or product, the next question is,

• 1:57Harry Stebbings?

  how do you get your users to do what you want them to do? Well, that's where Pendo comes in. Pendo is the only all-in-one product experience platform for any type of application. Honestly, Pendo's differentiated. Is in its platform. Every capability from analytics to in-app guidance to session replay, mobile, feedback management and roadmapping are all purpose-built to work together. But don't take my word for it. 10,000 companies, 10,000 use Pendo. And Pendo also does Mind the Product, the world's largest community of product management professionals. Check out their amazing free product today on pendo.io/20product-podcast. And speaking of amazing products like Webflow

• 2:42Harry Stebbings?

  there, incredible stat. Did you know that half of product managers say bad process and tooling are their biggest challenges? Well, with Airtable, you'll make sure you're building the right products grounded in what customers actually want. In fact, unify your entire product portfolio and workflows in one place and ship those products to customers faster than ever. One global financial services company even said it cut its time to market for new features by half using Airtable. Maximize your team's impact too with Airtable and give it a try for free at airtable.com today. That's airtable.com to get started.

• 3:18Speaker 02

  You have now arrived at your destination.

• 3:21Harry Stebbings?

  Scott, I am so excited for this. I heard so many great things from Healer. So thank you so much for joining me today.

• 3:26Speaker 00

  Thanks, Harry. I've listened to the show a bunch of times, so I'm really excited to be taking turn as a guest.

• 3:33Harry Stebbings?

  Listen, I totally get it. My accent is shit in British, but I can't change it. So you can leave a review on iTunes. But I want to start, you know, you've worked and led product teams at GitLab, at SendGrid. How did you make your way into the world of product and come to lead some of these incredible product teams?

• 3:49Speaker 00

  Well, it was actually a long road to get into product. Well, I started my tech career in sales and spent four years doing that. Really quickly into that role, learned that I was very excited about what the product people were doing. So I endeavored to position myself to get into the role. And among other things, there wasn't much product management content back in the day. So I decided to get an MBA. With a management of technology specialization to prove to hiring managers that I was serious about getting into product and gain some experience that I hadn't really had, having kind of a traditional business background. And unfortunately, when I came out of grad

• 4:29Speaker 00

  school, it was 03, sort of the nuclear winter of tech, there'd been the dot com bust. And so I was still competing with people who were trained product managers. So I took a job in alliances, which sat very close to product at this particular company. And after a few years, working very closely with the product team, was finally able to kind of get in the door, get my first PM role. And then from there, it was up to the races because I had a very broad background up until that point. It was a role I was supposed to be doing. And so from there, I have had an amazing career in product leadership at Wiley, CA, Sanyo and GitLab.

• 5:09Harry Stebbings?

  Okay, so I have to touch on a couple of things there. Number one, are MBAs still worth it today? I asked Brian Halligan at HubSpot, what do you think?

• 5:16Speaker 00

  I think it depends on where you want to end up. If you want to be a badass PM, I don't think you really need it. You can learn the skills that you need to learn through product focused training, through being in the right situation, through learning from Jedi masters at your company. If you want to be a CPO or VP, if you want to really connect the dots between product as a function and the rest of the company, I do think it can have long-term payoffs from that standpoint. So I think it can have return for people with a very long view. But in the short term, there may be a faster path to a really cool PM job.

• 5:55Harry Stebbings?

  I want to start with the hardest question of all in product, okay? And it's art or science? We have a ton of data today and it drives a lot of decision making. How do you think about whether product is art or science when done well?

• 6:09Speaker 00

  I think it's for most roles, about 50/50.

• 6:13Harry Stebbings?

  Oh, that is a cop out, Scott.

• 6:15Speaker 00

  I know. Let me explain. And there are certain roles where it's a different mix. I think for most roles, what I've come to learn is that if you have a systematic approach to how you capture inputs for product, do you do customer interviews in a systematic, thorough, professional way? Do you instrument your application in the right way to where you get the right kind of product usage data and answer real questions? Do you have the right KPI that connects what you're doing to the business outcome? These are just examples of sort of having a systematic mindset to how you go and you do the job and how you capture the inputs. But then you've got to derive insights from it. You have to figure

• 6:57Speaker 00

  out where to take that data, how to turn that data into a user experience that actually solves the problem that you're attempting to solve, that actually drives the business outcome you're trying to drive. And I feel like that piece of it is artistic, quite creative. And I think in most roles, it's about half and half. If you're a growth PM at Facebook, you've got tons of data, the path for the product is pretty clear, you're mostly just running A B tests, maybe it's more like 80, 20, 90, 10 data to art. If you're a startup, you have very little data at your disposal. You're mostly working on early qualitative feedback, really deeply understanding a problem and then trying to go iterate your way into it. And that'd be more like 20 data, 80 art. So that's how I think about that mix.

• 7:43Harry Stebbings?

  Are product leaders generally a science-led product leader or an art-led product leader? Is the unicorn the one that does both?

• 7:49Speaker 00

  Yeah. If I think about my own career, I've been more of the later stage, let's bring some science to this thing. Let's bring some rigor to how the PM job is done. We can raise the average performance level for PMs on a medium to large team. It's more of like thinking of your product team as the product because they have to go do the individual work versus if you're a very early stage, either founder or person who's come in to run it, a lot less data, maybe it's just you, you've got to be scrappier, you've got to work on less data, you've got to use your gut a lot.

• 8:25Harry Stebbings?

  When does a product need to be more science? For founders that are listening today and that are a million in ARR and they're more intuition-based, they're more creative, they're more artistic, when do they need to go, huh, we should instill more scientific structure, processes, rigor into our product org?

• 8:42Speaker 00

  Basically, it's when you've found repeatable product market fit. You know who your ideal customer profile is. You generally know the problems they need you to solve. You generally have confidence that your product experience can both attract and retain them at a reasonable, sustainable level. Once you get to that point, then you're going to need to start adding nodes on your PM machine. You're going to need to break up the product experience. You're going to need to be able to bring on individual PMs and designers and engineering managers, own pieces of it. And when you get to, maybe when there's still three PMs, there's still not a big need for big structured process. But when you get up to 10 plus, you're only as good as through the average PMs performance.

• 9:26Speaker 00

  And if you don't think about the broader system that each one of these people are plugging into, you're going to get this huge variance in performance.

• 9:33Harry Stebbings?

  We're going to talk about PM performance. First, I just want to touch on PM. It's a kind of ever-changing role. I'd love to understand, how do you define the role of PM? And how is it changing in your mind?

• 9:45Speaker 00

  Well, I define the role as a hub of function between the outside world, the market itself, the customers, the competitive moves, technology changes, and then the company. You've got engineers, marketers, salespeople. They all need, both sides of that need to interact with your product. And so PM is the function that sits in between those external and internal worlds, sets direction, makes the priority calls, decides where this thing should go and what we should invest in, and what ultimately what the user experience should be. Now, to do that well, you need about half your time to be out of the building.

• 10:24Speaker 00

  Most PMs spend about 95% of their time facing engineering and zero to fives, talking to customers and being out in the world. What that means is the work that they tear for engineers oftentimes very ill-informed, maybe based on instinct or what the sales team told them to do, or what the CEO told them to do, or what five million dollar customer told them to do. And you end up with a work product that's Ultimately misses the mark for progressive teams that do product well. They've changed in that they've given PMs more time to be customer facing. They start with the problem rather than the solution or what internal people think you need. And they routinely connect the work you're trying to do in product to the business outcome you're trying to drive.

• 11:08Speaker 00

  Which customer is going to use this? What action do we think is what's going to change? If you operate that way, you get more outcome focused and a lot less output focused. And I don't know, maybe 20% of teams work this way, still a minority.

• 11:22Harry Stebbings?

  I want to dig in after this next question into kind of how do we hire those people with those qualities? Before, how do we hire them? Just in case of when, you know, you said before we started recording that you advise sometimes founders who are focused on product as early as that. When is the right time to make your first product hires, your first PMs, your first product leader?

• 11:41Speaker 00

  When you've found repeatable product market fit where you know the ICP, you've got decent retention and you've seen a repeatable ability to go acquire that target customer.

• 11:52Harry Stebbings?

  That's quite late though.

• 11:53Speaker 00

  Yeah.

• 11:54Harry Stebbings?

  I mean, this is like two, three million an hour.

• 11:56Speaker 00

  I think if you hand it off before then, it can be very problematic because it's these decisions. Which ICP? What are we going to prioritize to build for them? What's the core user experience? These things are so central to a company, especially if it's a PLG company. Handing that off before that is fairly clear. It feels like a recipe for disaster because you're actually either going to want to still be super close to it and you're going to squeeze them up or you're going to delegate too early and you're going to have someone without the core context with founder making these critical decisions. I could get behind hiring a fair junior PM, maybe to do some of the core day to day backlog management, maybe.

• 12:39Speaker 00

  But nothing more than that until you have decently repeatable product market fit.

• 12:43Harry Stebbings?

  Okay, so decently repeatable product market fit. Now we know when we need to actually do it. I think it's really freaking hard to find these great people. And so I do want to unpack the structure of your hiring process. If we break down that structure, what does that look like? How do you structure the hiring process for these product hires that you just mentioned?

• 13:03Speaker 00

  Five interviews total. First one's 30 minute recruiter screen or hiring manager screen. If you don't have a recruiter, just check into the basics. Then the hiring manager should spend an hour with them checking on, clear the bar for the core competencies of PM. And I can get into what I think those are. An engineering manager interview for an hour or so where you're testing for clearing the bar on the domain and the minimum level of technical knowledge they're going to need and their ability to work with engineering. One peer interview, if you have other PMs, they should be doing this. I could get into tactics and techniques for that, but I typically like to see that be a little more hands on where they work on something together.

• 13:45Speaker 00

  And then one more sort of finalist interview, either with the hiring manager again or a bar raiser interview. If you have someone else in the org who's very good at interviewing where you get into more like case question.

• 13:57Harry Stebbings?

  There's so much I want to unpack there. So that's really helpful. And I write on my hand, it's a good sign. So you said there about the core competencies of being a PM. Just so we actually understand them when we are in this process. What are those core competencies that we're searching for in PMs?

• 14:11Speaker 00

  There's four buckets for an individual PM. One is around valid, I call it validation, customer interviewing, product usage data, kind of like gathering insights. Two is build, which is how well do you work with engineering? Can you make trade offs? Can you tee it up for them in a language they understand? Can you iterate? Can you work with them to break stuff down? Can you ship? Three, the business, do you know which KPI to set? Do you know what the thing you're working on is going to drive? So linking the activity to the potential outcome. Can you like, think about the business of your product and what you're trying to do? And third is communication skills. You need to man it up to execs. You need to talk to customers in their language.

• 14:55Speaker 00

  You need to speak to engineering in their language. Every one of them involves different vocabulary, different altitude. And you need to be clear to each one of those constituents. That's how I break it up down.

• 15:07Harry Stebbings?

  And then you said about peer interviews was another one where I wanted to dig in. How do we structure them in the right way? And what are the nuances and complexities there that we should know about?

• 15:16Speaker 00

  I think GitLab did this well. We had what was called the think big, think small was the exercise. And the PM would partner with the interview candidate. And ahead of time, they would ask them to come prepared with a topic. It was sort of pertinent to the area they were interviewing for. And we would ask them to write down what's sort of the vision for this thing. And then how would you iterate your way to it? Iteration was a company value at GitLab. I think it's a very underrated aspect of great PM and great product development is this act of breaking things down into small chunks. So that you don't over complicate what you ship. So this exercise tested a few things. It tested their ability to think about something on a fairly large scale.

• 16:02Speaker 00

  Tested their ability to break it down into like actionable steps that the engineering could absorb. And it tested their verbal skill and their written skill because they had to write it down. GitLab is all remote. You're dead in the water if you can't write clearly at GitLab. So it was testing all those things at once.

• 16:19Harry Stebbings?

  Is writing a core skill of PMs? I remember I had Kevin Napaka on from Twilio and Segment, but he said like writing is the most important skill. Do you agree and why so?

• 16:28Speaker 00

  I tend to agree, especially at a place like GitLab where it was all remote. PMs did very little synchronous work. If you can't rely on sort of verbal agility to get things done, you better be a good writer. And I think relying on verbal is is fine. It's helpful. Like I said before, you have to communicate clearly to different levels and different audiences. But that written artifact is just essential to driver alignment. There are two that we use at SendGrid and GitLab that I think were essential. One was a one pager called an opportunity canvas, which asked PM to outline certain concept they wanted to to build. The other one is like a two to six pager. I think Kevin might have mentioned this.

• 17:13Speaker 00

  It's sort of borrowing from Amazon, this concept of a six pager, sort of a longer arc strategy doc. Those are two that I feel like you have to write down.

• 17:22Harry Stebbings?

  Breaking those down, who do you present those to? How are they used internally? Can you just share that? If I'm a founder listening and I'm like, oh, that's a good point. One pager and a six pager.

• 17:33Speaker 00

  Well, start with the six pager because that's sort of the longer arc one that should help you drive more focus. At GitLab, we had these things at my level for the whole product line. The directors had them for product areas. So if you think about GitLab, there's Dev, Sec, and Ops. So we had a Dev strategy, a Sec strategy, and Ops strategy. And then each PM underneath those directors had their own strategy for their area. Maybe it was agile planning or roadmapping or continuous integration or whatever. And when you have these laddered strategies, it helps each level make sure that the things they're working on are connected to other levels. I don't want to overcomplicate it. You mostly deal in startups here, so you probably just need one of these.

• 18:18Speaker 00

  But the point is, you want a little bit of a longer arc than your typical planning cycle. So GitLab, we had a one-year planning cycle. Plenty of companies have quarterly planning cycles. Make sure it's two or three iterations, larger little planning cycles. So if you plan every quarter, make sure it's at least a year. If you plan every year, make sure it's two. So that you can kind of get out and over the planning cycle into like, where are we going over a longer arc? And it should make trades. It should say, hey, we're going after this segment and not this segment. We're going to focus on these use cases and not these use cases. It should narrow your focus rather than having it overly broad. And that kind of a doc and that kind of alignment is tough to get to.

• 19:01Speaker 00

  And I think the only way you get to it effectively is through long form writing.

• 19:04Harry Stebbings?

  Why is it tough to get to?

• 19:06Speaker 00

  Because words matter. And if you're projecting your strategy verbally or in bullet points, people can hear what they want to hear. They can read, they can interpret what they want to interpret. When it's written down and you actually read it and it's off 10 degrees from what you think, I think it's easier at MAP to, hey, here's what I really think we should do. And here's what this person's saying we should do. And just that fidelity alignment is so much higher in a long form doc on something like a strategy.

• 19:36Harry Stebbings?

  Let's take this further. So you, we're in the same product team. You write one of these six pages. How do we have an internal debate discussion on your six page?

• 19:45Speaker 00

  Yeah.

• 19:45Harry Stebbings?

  Is that commenting? Is that a cool?

• 19:47Speaker 00

  Yeah.

• 19:48Harry Stebbings?

  What does that discussion look like?

• 19:50Speaker 00

  I'll use GitLab as an example. It was a fairly basic process, given the all remote nature. I did it at my level in sequential kinds of concentric circles that got bigger. I reviewed it with the CEO first. I reviewed it with my product leadership team second, the executive team third, broader product team, fourth, whole company fit. Each time I took comments essentially. And each time I would refine it until that group understood it. And then I would move out from there. If this is an individual PM, maybe you start with your design and engineering manager or your boss, your boss probably, then your media peers, then the broader team, then the company.

• 20:34Harry Stebbings?

  Which stage was the most common to put a blocker on it?

• 20:38Speaker 00

  Exec team. In my own experience, the exec team's work gets more difficult because that's where you really need to make cross-company trade-offs. Good example, at SendGrid, we stayed pretty true to the mid-market as our. Target ideal customer profile. We resisted going up to enterprise. That was controversial. There were almost every sales leaders at the company wanted us to invest in more to go up, to do the tippy top complex shit that, you know, the bigger brands wanted. But we more or less resisted that over the years and I think that was to our benefit. We kept the product experience really clean, very self-serve first. I think that was the hard way that company ended up doing well.

• 21:22Speaker 00

  But, you know, you had a strategy review and they don't see that you're doing the stuff with the top 10 one and it's complicated.

• 21:29Harry Stebbings?

  It absolutely is and then you've also got misaligned incentives in some ways. Sales comp is often easier, not easier, but larger if you go after heavy enterprise. Going after a smaller contract is different. Okay, and so you have those different components. Most often it gets blocked at the exact level. When should founders start doing this? Is that when you're post 100 people? Cause there's quite a few layers there.

• 21:51Speaker 00

  Even if you're more than 10 people, this can be helpful. Maybe when you're not all in the same room. When you start adding new people at some clip. Because new people just don't have that context. They don't have the tribal knowledge. And when you add more nodes on, the quicker you can get them wrapped around who we're going after, what we're gonna do, we're not gonna do, what matters most, what business outcomes are we trying to drive. You eliminate a lot of extraneous conversations that can otherwise slow the whole place down. So I think it can be useful very early.

• 22:22Harry Stebbings?

  When do you do six pages versus one page?

• 22:25Speaker 00

  So I think a good strap can be as short as two pages. So there's no need to fill up the sixth. I think the reason why Amazon set the six page limit is because in one meeting on one topic, any more than six pages is just too much. But make it as short as you can. The one I did at GitLab for the whole product is probably a couple of pages, two, two, maybe three. It wasn't that long. It can be quite concise. That's all the strategy. When I mentioned the one pager earlier, that was on a particular product concept. I called it an opportunity canvas. And this was used early in the process. The owner of it was the product manager. The point of it was to help them think holistically about something they wanted to take on.

• 23:08Speaker 00

  You don't do this for all projects. You don't do it for iterative improvements. You don't do it for tech debt. But for something that's big, that's going to take weeks or months of engineering time. I think it's very smart to do something like this. And ask them to go out and talk to five to 10 ideal people in the target user profile. And interview them about the problem that they have. And then come back and show me an opportunity canvas. And it tells me who's the target user, what pain point do they experience? What are their workarounds? What would the upside be to us to do this? What would the first thing you would do to build against this be? What are your alternatives? What are your risks and what's your KPI? There are a couple of other things, but it could all fit in one page.

• 23:52Speaker 00

  And if you've done your homework, you can fill it out 15 minutes. And what that does is it forces the PM to think about a bunch of different angles on an idea. The problem with ideas is they can sound very good from one perspective, like customer X asked for it. Or wow, this sounds cool. Or, oh, if we did that, we could up tier them into the next tier. There's always some reason why an idea sounds good on the surface. But sometimes when you line these things up, it's like, oh shit. Our target user that really wants this, we don't actually talk to, we don't market to. Or there's only three customers that want this, and yes, it's acute for them, it's not broad enough. Forces that conversation. And it's a good time to either redirect or kill or pause.

• 24:37Speaker 00

  Cause you haven't coded anything yet. You haven't wasted the team's time yet. So I had to spend a lot of time kind of at this opportunity canvas, problem validation phase. To make sure that the things you're going to push down the cycle were well conceived.

• 24:51Harry Stebbings?

  As a product leader, you have hopefully a team that is full of ideas and does a lot of opportunity canvases. How do you ensure focus and velocity, which is speed in a given direction that's hopefully right? But also not ensure that a team doesn't feel crushed and stifled of innovation and ideas?

• 25:10Speaker 00

  It's, it's a tricky balance because this sounds a little heavy, but let me also explain. PMs always have two tracks going. There's a validation track where you're doing your homework on the things you want to pump into engineering. Or then you have a build track. The build track's always moving. You go on sprint to sprint to sprint. Every place I've ever been has been in an enormous backlog for engineering. So it's, in my opinion, if it's well-run, doing a proper validation shouldn't slow down engineering. They have more than enough to do. This is about making sure that the next thing you tee up with them is well-conceived. But to my far earlier point, PMs need half their time available to be doing this kind of validation work. So it shouldn't slow down the engineer, it should speed them up.

• 25:53Speaker 00

  Because there's less back and forth between PMs and designers and engineers if the project is well-conceived. Where you get a lot of back and forth and slowdown is when you're like, are we building for that or that? Who is this for? Who cares about what? If that shit's unclear, the product development process can get really chaotic. So in my experience, it's sped up engineering.

• 26:13Harry Stebbings?

  How do you know if you have a chaotic product development process? I think a lot of founders do, but it may not be obvious. What are the signs that it is a chaotic product development process?

• 26:23Speaker 00

  Tons of back and forth, engineers that don't get it. They're like, why are we doing this? I don't understand why this thing is more important than that thing. When you're switching a lot, oh, well, we're weekend of the sprint. We redirected to this thing that's totally different and now I gotta switch. So if there's a lot of content switching, there's a lot of confusion about why. There's a lot of back and forth between the engineers and whoever asked them to do this thing. Probably isn't a well-conceived backlog in the first place.

• 26:53Harry Stebbings?

  We're gonna bring this back to also the hiring process. Cause it's kind of aligned, but one of the other core elements you said was the case study element. And so kind of thinking about that, how do I do effective case studies in a product interview process?

• 27:06Speaker 00

  Yeah, I like to ask a non-technical, I like to pose a non-technical scenario that they haven't worked in before and actually isn't what we do either. Because I don't want them to A, rely on the jargon that they learned in their last job. And I don't want to also sort of penalize them for asking them about an area that I know really well. And boy, this you have seen this. So ask sort of a non-technical, semi-unrelated case to ask them how they would solve a problem.

• 27:36Harry Stebbings?

  What would be an example of that? Sorry. So if you were at Sangre, just is it like your PM at Uber solves this problem?

• 27:43Speaker 00

  Yeah, I'm always sort of borderline embarrassed about this example, but I've used it a ton of times. If you ask product managers at Sangre or GitLab, they'd probably chuckle about having to have gone through this. I asked them to pretend they are a product manager at a foods company and they own the milk product line. There's been a problem with the production line and they're losing X per day and it sort of set up this scenario. And the general managers asked you to dig in and figure out what to do. So what do you do? Bizarre problem statement, but you learn a lot from how they handle it. Like, do they start from first principles? Do they have sort of a methodical way of working through a fresh problem? Do they start with

• 28:25Speaker 00

  the customer or do they just start making shit up? Do they ask good questions to unpack the scenario? These are all signs of someone who thinks systematically. Ultimately, that's what that case is for, is to test systems thinking.

• 28:38Harry Stebbings?

  When we asked about kind of where does the six pages get called in the process? When you think about like candidates going through the PM process, why do they most often fail?

• 28:48Speaker 00

  I think it's when I interview a candidate, I set it up, say, tell me about the product that is the most relevant to what we do here. And I'm going to ask you a bunch of questions about it. And so I try to go a few levels deep on what they've done.

• 29:02Harry Stebbings?

  Does it matter if they haven't done anything similar? And so if I'm joining SandGrid and I used to work at Snap, seemingly uncorrelated, does that matter?

• 29:11Speaker 00

  Not really. Look, it can matter. There are certain roles where you need a certain level of technical expertise. That's a fact. There are certain roles, like, let's say I'm hiring for a billing product manager. I'd probably want someone who's been around billing before. So you need to make sure that they've had enough domain experience to survive. But mostly what I'm testing for is how you do product and how you think. And that's relatively domain agnostic. I tend not to hire for domain if I don't have to. And you can test those things from a Snap PM just fine. So how do they fail? The examples that they give don't indicate to me that they took orders. They didn't

• 29:52Speaker 00

  use their own judgment from inputs that they got. So either they didn't go out and get good inputs, or B, they didn't or weren't able to turn those into insights. What good PMs can do is take inputs and derive really good insights. And so I'll ask questions like, tell me about a time where you're doing discovery with customers and learned something that's surprising but caused you to change course. Or tell me about a time where you're looking at product usage data and found something that... Was counter to what you believed and what you do about it? If somebody's spent enough time inter-healing and enough time sifted through product usage data, they should have plenty of examples like that. If they haven't and they're just making shit out, that kind of thing turns up.

• 30:35Speaker 00

  So that's one way they fail, that depth of their examples doesn't demonstrate they actually worked in the way we wanted to work. Two, this think big, think small, maybe they can't communicate clearly in writing or verbally or have trouble iterating to a good answer. And then three, the case study, a lot of people think very haphazardly about a problem because they just have a great framework for it.

• 30:59Harry Stebbings?

  When we think about hiring in a lot of roles, we have an obviously media team here. We're hiring for social media often is one of the roles we hire for. It's very clear when someone says, well, on Instagram, I caused a 38% uplift in conversion. On TikTok, I caused this percent. Data centricity tied to their role is a core signal of A star candidate early. Is that the same for product? And what are the signals that are like, ah, Harry's got it from that answer?

• 31:26Speaker 00

  Yeah, I think someone who can clearly connect the work they were doing to the outcome they were trying to drive on can describe, okay, what did I learn from customers that led me to believe that they would take this action if X were there? How did I instrument the product to know whether the intended effect was there or not? Well, then how did I manage to that and iterate from there? Like, tell me a story about how you went from early stage to later stage. What lose did you make along the way? Because it's never about shipping one thing. It's about shipping and learning and shipping and learning. And it's a constant journey and you have to have good habits all along the way.

• 32:07Harry Stebbings?

  Speaking of shipping and learning, we're going to get to product reviews, but we are always evolving our hiring approach ourselves. I've made many mistakes hiring. When you think back, what's been your biggest hiring mistakes? And what did you learn from them? And how did it change how you think about hiring?

• 32:23Speaker 00

  Well, I think way back, I didn't have nearly as much conviction on A, what a good PM looked like. I didn't have a rubric in my head like I do now. And I hadn't made the connection on systems thinking and that being super important for PM. So I just wasn't interviewing for the right stuff is where the early stage came from.

• 32:41Harry Stebbings?

  Is that the same for founders today? When you look at the founders that you work with, is that the same as the mistakes that they make when hiring for product teams?

• 32:48Speaker 00

  Yeah, I mean, they don't know. They most of them haven't been a product manager. They have no idea what this role is about. They don't know what to look for. They don't know how to grade it. And so they go off of pedigree or they are at Google. Or B, they know the lingo, you know, they talked about validation. So must be good at it because, you know, it's like, it's logical. I get it. I sucked at interviewing for this at first too. So the more that you can do is a founder to be clear on what skills you need out of a PM and be really intentional about testing for it. Even if you don't know if function well, you're be better off.

• 33:22Harry Stebbings?

  Say we hire this person and we're excited about them joining. Onboarding is universally done terribly. How, how do you onboard PMs effectively, Scott?

• 33:33Speaker 00

  First, especially in early stage scenarios, the founder has to be crystal clear with the incoming product person about what they want to own as the founder and what they want this product person to own. Oftentimes this is murky and the PM comes in or the product leader comes in and the founder decides, oh, I actually still want to own prioritization. But the product person came in thinking they would own law. So do you want to own vision? Yes or no? Do you want to own strategy? Yes or no? Do you want to own prioritization? Yes or no? Do you want to own hiring? Yes or no? Be crystal clear about those and communicate that in the hiring process and, and document it when someone comes in so that that's clear. They know what their field is supposed to play on.

• 34:17Speaker 00

  And then two, I like to be very prescriptive about what I expect. Okay, in week one, you should meet these people. Here's a link to all the docs you should read. Like in the first two weeks, you should be helping them gain context very fast. Like roll out the red carpet. Here's who to meet. Here's what to read. Here's all the contacts you need. In weeks three and four, talk to customers. You know, I want to see 10 customer interviews or whatever. By the end of month one, you should be able to give a dental, what these deliverables are company specific, but help them get context. Tell them your expectations on what they should own when. Another good example is maybe you're the founder and you still own all the squirm proceedings. When do I want you? To start leading these things after month one, month two, like set their expectations

• 35:01Speaker 00

  and then along the way engage whether they're going to be ready on time.

• 35:04Harry Stebbings?

  How fast do you know if someone's bad?

• 35:07Speaker 00

  I think if they're really bad, you know, within the first week. If they're maybe good, but not good enough, you're going to know within the first quarter.

• 35:15Harry Stebbings?

  Have you ever had someone being really bad? And if so, what did you not see? How did they get through that process?

• 35:20Speaker 00

  Later on, when I got better at hiring, there were relatively few of these where it was just, you know, they're bail on. We wanted just a total disaster because we got better for screening, but we did have a fair number that just didn't make a cut at month three. I think what you find is, hey, they don't take ownership. Either they don't want to or can't. By month three, you should really see them start leaning in and have a strong point of view, take ownership, have some conviction and priority. And they're just not there for whatever reason. Maybe they didn't have the base skills to form a point of view. Maybe they're a little timid about communicating a point of view. Maybe they don't collaborate. Well, these things can be a little harder to test for and show up in, within the first few months.

• 36:05Harry Stebbings?

  Once in those first few months, managing performance is so important. How do you think about managing PM performance? We said it earlier, but how do you actually do that? And is there a framework or structure that's effective in doing so?

• 36:19Speaker 00

  Yeah, I have this thing called a career development framework. I mentioned those four buckets, so validation, bill, business and communication. And then there's a breakdown for PM, senior PM, principal PM. And it tried to outline with as much specificity as possible what behavior and outcome expectations are at each level for each bucket. And I would have conversations with my directs and I would ask any other people leaders to do the same, to have a check in every two to three months on that. What I would do is very simple. I'd create a new line below their level and I would write down, what did I see that was either really good? Or constructive feedback on each of these four buckets.

• 37:04Speaker 00

  And then I would also tell them where I think they stood at large against that skill set. Do they need development? Do they meet my expectations? Or do they exceed my expectations? And if you check in on this stuff every couple of months, they get nuggets of really actionable feedback. They know where they stand with the year. They have something or some things to work on over the next two to three months to get better. And then when the review cycle comes up at the end of the year, well, no surprise where they stand. Promotion, firing conversations are all much easier because they know where to stand with you. And it takes some work. I would set prep calls in my calendar. Okay, a week before my interview with Alex, I would have a little half hour prep meeting where I would throw this thing out. Gather my thoughts.

• 37:50Speaker 00

  Takes a little bit of energy to provide good feedback. And then I would have an extra half an hour on top of our normal one on one where we would just talk about career shit. We wouldn't talk about day to day. And then I found that worked really well if you were consistent.

• 38:03Harry Stebbings?

  Okay, you're talking about kind of career shit and promotion in particular. What should PMs focus on if they really want to get promoted as a PM?

• 38:11Speaker 00

  A, you should have the strongest point of view. Anybody in the company about the ideal customer for your area. Like if you're the an ops PM at GitLab, you should know what those platform engineers want, how they work, what makes them tick, what their problems are.

• 38:27Harry Stebbings?

  Is that not a way to get promoted though? If we think about like, I'm in GitLab. How does having a strong opinion on the ICP help me get noticed by leadership and get a promotion?

• 38:36Speaker 00

  If you really understand them, then you're going to tee up the right work, which is going to drive results. Ultimately, you want to drive results. How do you get there? I think the building block is having a ton of perspective on the target user and what they need. See if they're the right stuff for them. And if you demonstrate that you know the target user, sales is going to trust you. Marketing is going to come for you to you to make sure that your thing is marketed the right way. Engineering is going to believe you and trust you in that fight that you want every decision you make. It just, everything flows from that. So that's the very first thing I would do. And then from there, you're trying to prove business impact. So make sure you have a KPI that leadership cares about, that maps to the work that you report on regularly and openly and honestly.

• 39:21Harry Stebbings?

  My God, it's a lot of work being a product leader.

• 39:23Speaker 00

  It really is.

• 39:24Harry Stebbings?

  I'm listening to this, I prep cool before the meeting. I'm like, oh, fuck, this sounds awful.

• 39:32Harry Stebbings?

  I think one of the reasons this show is successful is because I'm not in it. And so I bring this very natural like, oh. You do venture dude, venture is so much better.

• 39:43Speaker 00

  You gotta want it bad to be a TM, that's for sure.

• 39:46Harry Stebbings?

  I mean, God, yeah. You've spoken before also about like iterating and learning. I think one common feature of any product team or leadership role especially as well is the product review cycle. Quite hard to know how to do a product review well if you've never done one before.

• 40:03Speaker 00

  Yeah.

• 40:03Harry Stebbings?

  How do you do product reviews well? And what have been some big lessons in what works?

• 40:08Speaker 00

  If you're really, you may be able to combine all of those, but I tend to separate reviews that are about work in progress from reviews about the product or product area as a whole. As an example, this opportunity, let's say you're a product manager in the ops area of GitLab. I used to have a monthly KPI review with each area. So we had a dev monthly KPI review, a sec monthly KPI review, an ops monthly KPI review. This was more about the area as a whole. Each area had a target KPI or numerous KPIs that they were managing to. And the rhythm of that was I would ask them, okay, what happened in the numbers? What's going on? What did you do last month to drive your target metric? And what are you going to do next month?

• 40:53Speaker 00

  It was sort of this constant check in to make sure that their priorities and actions at least passed the test that they were reasonably aligned with the target metric they were trying to measure. So that was kind of high level. How's this whole thing doing? Then you also have work in progress. So you're the ops PM and you have this idea that we need a much better incident management feature set. That's where I would get into reviews on this in progress work. And that's where that opportunity came, this then comes in. Okay, Ops PM, show me the canvas for the problem we're solving with incident management. And if the concept looks good and you sort of green like that, well then they move to the next phase, which is building a prototype, usually with design.

• 41:36Speaker 00

  And then we would go back and review the prototype as a solution. Okay, PM, you had these 10 customers, they wanted this. Does this design really solve that for them?

• 41:45Harry Stebbings?

  Can I just jump in and ask, just so we know, for this product review, who's in the room and who's set the agenda? Is it just the people working on it and the leader? Is it the wider product team? Who sets the agenda?

• 41:57Speaker 00

  I think for the opportunity canvas and the solution, well, it's called the prototype review. The PM, the designer, who are doing the work, the product leader or leaders. For me, it was me, the design leader. We have a VP of design at both Cindera and GitLab. The two of us were in there, plus the PN and the designer and anybody else they wanted to invite. And that sometimes met the engineering manager. Usually around four or five, and that's sort of on this conceptual pre-engineering phase.

• 42:28Harry Stebbings?

  Can I just jump in and ask, sorry, I'm too intrigued.

• 42:30Speaker 00

  Yeah, go for it.

• 42:30Harry Stebbings?

  You obviously worked at GitLab remote. A lot of people say that in terms of product reviews, being remote enables a lot more broad participation because people can obviously be kind of spectators, so to speak, without being a presence in the room. How do you advise remote versus non-remote when it comes to product reviews and wider participation?

• 42:49Speaker 00

  I think you can open it up to more. I was always of the mind that the more the merrier. But I wanted to be careful about how many voices were in the room because they usually fairly quick meetings. You know, maybe 30, 30 minutes to an hour max. You got to be quick in getting to the point and getting to the conclusions. At GitLab, we recorded all of them and we posted them and anybody who wanted to watch them could. So we were very transparent about all these meetings. You could see them all as an internal employee.

• 43:20Harry Stebbings?

  What happens post-product review? How do you ensure that the learnings are qualified, shared and then enacted?

• 43:27Speaker 00

  I think that's where these monthly reviews came in. So let's just say we agreed that they should move on with a project and we liked the concept and we liked the kind of the design prototype. Then it gets fed into the development cycle. And then as they're talking about their KPI review, they can reflect on what we did last month, what we're going to do next month. And then you start to see how this thing's getting iterated on as it projects all towards the ultimate customer value. That's why I separated them out a little bit. Here's the first two or more about making sure this is something we want to take on as a good investment. And the other was more like. Managing the ongoing priorities is the least of building.

• 44:07Harry Stebbings?

  I'm just thinking, like, when we think about managing those priorities, how do you, as a product leader, determine between resolving left technical debt versus pursuing that new opportunities of revenue generation?

• 44:19Speaker 00

  I think this depends on stage, broadly speaking. If you're very early, you're probably heavy in new feature build out. If you're more mature, you've probably got a lot of surface area that already exists and you might want to allocate a heavier percentage to iterative improvements of things that have already shipped and tech debt. At SendGrid, we had a recommended split, and granted, this is when we were sort of 100 million-ish, of 60, 30, 10, 60 on net new, 30 on iterative improvements, 10 on tech debt. It was totally fungible. Teams could recommend a different split. But in the absence of a better idea, that was our recommendation on how to allocate efforts.

• 45:01Speaker 00

  If you short iterative improvements or you short tech debt, it will bite you. And so I think having a healthy amount of ongoing improvement to the current experience and a healthy amount of ability for engineers to burn down tech debt produces, reduces the risk of these big pileups that sometimes companies get in.

• 45:20Harry Stebbings?

  Scott, you've been a product leader for many years now at some of the best. Before we do a quick fire, just with the theme of reflection, what do you know now that you could tell yourself entering the first product leadership position you took?

• 45:34Speaker 00

  Oh, my God, it'd be a long time.

• 45:35Harry Stebbings?

  You can call yourself the night before that first job and say, you should know this.

• 45:41Speaker 00

  Yeah, I think it would be really start with the problem and start with the customer and do a ton of customer discovery. My first PN role was in a place that was kind of one of these enterprise factories that... Tended to do what large companies asked them to do. And I get it, that was where the incentives were aligned. But ultimately, I don't think that product was everything it could have been. If I'd been more focused on starting with the problem, starting with the customer. So I wish I had that epiphany earlier.

• 46:15Harry Stebbings?

  Do you worry that we're going back to that world now with the centralization of budgets back to CFOs? With CFOs being tighter on spend, with bottoms up being more constrained?

• 46:26Speaker 00

  I think that's a reasonable hypothesis. There are fewer engineers, fewer PMs, fewer designers. People are being asked to do more with less. I would guess the average PM probably has less time to get out of the building. And there's more focus on revenue. And if you're a product manager and you're trying to drive a longer term vision that might have payoff in a year or two or three, versus the sales team coming and saying, hey, we gotta make the number of this quarter. I need this for company X. It's pretty hard to push back on that. So I can imagine that a lot of PMs are facing that and might be shifting more towards doing really tactical, short term stuff.

• 47:06Harry Stebbings?

  I want to move into a quick fire, Scott. I could talk to you all day. I've loved this. The best shows for me are where it's bluntly as granular and actionable as possible. But then I would add one thing, which is really quite specific. When it's quite numbered, people don't know this, but when you number points, they instantly become, I think it's 64% more memorable because it structures it in someone's mind. And so this has been fantastic. I want to move into a quick fire. So I'm going to say a short statement. You give me your immediate thoughts. Does that sound okay?

• 47:34Speaker 00

  Sounds good.

• 47:35Harry Stebbings?

  Okay, so tell me, when do we listen to users, customers, versus when do we actually respectfully continue as planned on product roadmap?

• 47:44Speaker 00

  Broadly speaking, listen to customers about their problems and generally ignore their recommendations around solution. Because they may not, building exactly what they want may not be the most elegant way to solve the problem. That's generally a rule of thumb.

• 48:00Harry Stebbings?

  What's the biggest mistake founders make when hiring product teams?

• 48:04Speaker 00

  Relying on logo versus the actual accomplishments. And I also think there's, in a related fashion, hire people who don't actually want to be at the stage you are. If early-stage founders try to hire a, maybe a really product leader, and they hire somebody from maybe a company that's doing a hundred million or a billion, make sure they want to get back into that hands-on hustle, wear a lot of hats mode. Because I see a lot of failed entry because it sounds glamorous to be the first head of, but these people actually all work that way. Early-stage, there's a lot of discovery, there's a lot of creativity. You're more on the art mode versus science. You have to want to be there.

• 48:47Speaker 00

  That early discovery or taking product from zero or a small number to a bigger number is a much different thought exercise.

• 48:56Harry Stebbings?

  What function has the most friction with product? And how do you, as a leader, minimize it?

• 49:02Speaker 00

  It's a tie between engineering and sales, depending on the orientation of a company. Some companies are very engineering-led, some are very sales-led. When those are, when the scales really tip one way or the other, they end up tending to want to do the priority work. They tend to want PMs to just be order takers and do what they think. If you're a good PM and you want to run the process the right way, that can create a ton of friction. If they don't really understand why your direction is better than theirs.

• 49:35Harry Stebbings?

  Does AI change product teams much?

• 49:39Speaker 00

  It will. It probably has to some degree. If nothing else, in terms of the projects they're being asked to work on. So A, they're going to have to understand its domain. They're going to have to get familiar with LLMs. They're going to have to learn how to prompt. They're going to have to learn how to create a, maybe more of a chat experience versus a typical UI experience. The product, the end result for customers is going to change. Two, you'll start to be able to use tools from AI to automate some of this stuff. You know, I mentioned product usage data and customer insights. I can imagine AI will do a nice job of summarizing that for you. So you don't have to incur as much brain damage with the input and synthesis and get some help on that.

• 50:24Speaker 00

  So PMs will, will have more help synthesizing tons of inputs. And then ultimately, I think it might change the role. Like right now, there's a pretty clear division between what PMs do, what designers do, what engineering, engineers do. PMs decide what to do, designers decide how it's going to, like what the user experience is in design and engineering. Builds it, codes it. I can imagine those lines blurring. If you get into a world where code assistants can do some prototyping, where code assistants can generate 80% of the code. And then it becomes more about how do you prompt the thing to come up with the right user experience?

• 51:03Speaker 00

  How do you, again, I think, I think PM still has a role because you're still going to need to decide who we're going after, what problems we're going to solve, all of that. But the lines may have been blurred. I can imagine we'll triple role down the line where they're doing PM, design and engineering with the help of AI.

• 51:22Harry Stebbings?

  Tell me, what would you most like to change about the world of products?

• 51:25Speaker 00

  Well, it's well worn advice at this time, but it's the same advice I would have given myself way back when. Talk to customers more. You know, I feel like even though that's like the most obvious thing in product, I'm guessing 90% of PMs don't do almost of it. And it shows up in a number of failed products and failed projects. And so I wish PMs would just start there and then fill in the rest of it.

• 51:49Harry Stebbings?

  What recent company product strategy have you been most impressed by?

• 51:53Speaker 00

  Well, at big scale, Amazon's amazing. I've learned a ton from the way they do things. And if you haven't read, working backwards, I recommend it. For a more recent, smaller company example, there's a company called The Browser Company.

• 52:08Harry Stebbings?

  Yeah, Josh.

• 52:09Speaker 00

  Very charismatic dude. And they have a browser called Arc, which is a crowded space with a tough, you know, playing with giants there. But I've heard a ton of buzz about them and I've used, I use it myself on on an Arc browser right now. Lots of thoughtful UX touches.

• 52:26Speaker 00

  And they made it a little more fun to go browse the web. And I think they have an ambitious vision of how one might access the internet down the line. For example, using AI in a clever way and making it very easy to interact with ChatGPT through their browser. So they've done a really nice job of trying to create some differentiation to a great product and user experience.

• 52:48Harry Stebbings?

  I'm looking now, actually, I got introduced to him through a mutual friend. He's in Paris now. There we go. Scott, listen, I've loved doing this. This has been so much fun. Thank you so much for putting up with my meandering questions and you've been amazing.

• 53:02Speaker 00

  Well, my pleasure. It was a lot of fun. Thanks for having me here.

• 53:06Harry Stebbings?

  For me, that was just one of the best shows. I started 20 product, 20 growth and 20 sales because I wanted to bring the knowledge of the best operators to you, who maybe doesn't have that wisdom and experience and seen all the things that they've seen. He went so granular there in his lessons today. If you want to see more from us, you can find it on YouTube by searching for 20VC. But before we leave you today, your website is the front door of your business. But the way teams build and optimize it is broken. Stuck between inflexible templates and cumbersome code-dependent solutions. There is a better and faster way. Enter Webflow, a visual-first platform that empowers you to create freely. Experience a CMS where you can build dynamic content without code.

• 53:50Harry Stebbings?

  Native localization that lets you translate your site in one click to reach global audiences. And third-party apps and integrations so you can build faster. Now you can ship webpages in weeks instead of months and save millions in development costs. These are the real results for companies like Orange Theory, Dropbox and Idio. Get started today at webflow.com. That's webflow.com. More than a website builder. And once you have a great site or product, the next question is how do you get your users to do what you want them to do? Well, that's where Pendo comes in. Pendo is the only all-in-one product experience platform for any type of application. Honestly, Pendo's differentiation

• 54:36Harry Stebbings?

  Is in its platform. Every capability from analytics to in-app guidance to session replay, mobile, feedback management and roadmapping are all purpose-built to work together. But don't take my word for it. 10,000 companies, 10,000 use Pendo. And Pendo also does Mind the Product, the world's largest community of product management professionals. Check out their amazing free product today on pendo.io/20product-podcast. And speaking of amazing products like Webflow there, incredible stat. Did you know that half of product managers say bad process and tooling are their biggest challenges? Well, with Airtable, you'll make sure you're building

• 55:17Harry Stebbings?

  the right products grounded in what customers actually want. In fact, unify your entire product portfolio and workflows in one place and ship those products to customers faster than ever. One global financial services company even said it cut its time to market for new features by half using Airtable. Maximize your team's impact too with Airtable and give it a try for free at airtable.com today. That's airtable.com to get started. As always, I so appreciate all your support. Really, it means the world to me and I can't wait to bring you a fantastic episode this coming Friday.