20 Product: Robinhood CPO on The 3 Stages of Product Management, How to Structure and Execute Great Product Reviews, The Secret to Building a World-Class Hiring Funnel in Product Team Building with Aparna Chennapragada

Aparna Chennapragada is Chief Product Officer @ Robinhood, the company revolutionizing consumer finance with commission-free investing, and tools to help shape your financial future. As for Aparna, prior to Robinhood, she spent an incredible 12 years at Google, most recently as VP and GM for Consumer Shopping and also as the lead AR and Visual Search products. Aparna is also an active angel investor with a portfolio including Khatabook, Statsig and On Deck to name a few. If that was not enough, Aparna is also a board member at Capital One. In Today’s Episode with Aparna Chennapragada You Will Learn: 1.) Origins in Product: How Aparna made her way into the world of product and product management? What were Aparna's biggest takeaways from her 12 years at Google? What does product management mean to Arpana today? 2.) Customer Discovery: 101 What are the 3 different stages of product management? What does great customer discovery look like? What are the best questions to ask? How should one dig deeper? Where do so many make mistakes in customer discovery? What should product people take from the answers? What should they disregard? 3.) The Hiring Process: How should founders breakdown the process of hiring for their first in product? What does the interview process look like? How should founders structure it? What core questions should teams ask of prospective candidates? What are red flags when interviewing potential product hires? What literal tests and case studies can founders do to test the quality of candidates? 4.) The Onboarding Process: How should founders structure the onboarding process for new product hires? What can founders do to make PMs successful in their first 30 days? Where do many product hires make the biggest mistakes in the first 30 days? What can product hires do to build trust with their new team? Items Mentioned in Today's Episode with Aparna Chennapragada Aparna's Fave Resource: Shishir's Executive Onboarding

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Harry Stebbings?

  This is 20 Product with me, Harry Stebbings. Now, this is the show where we're joined by the world's best product leaders to reveal their tips, tactics and strategies when it comes to building the best products and scaling the best product teams. Today, I'm so thrilled to be joined by Aparna Chennapragada, Chief Product Officer at Robinhood, the company revolutionizing consumer finance with commission-free investing and tools to help you shape your financial future. As for Aparna, prior to Robinhood, she spent an incredible 12 years at Google, most recently as VP and GM for consumer shopping, and also as the Lead for AR and Visual Search Products. Aparna is also an active angel investor with a portfolio including the likes of Cutter Book, Statsig and OnDeck, to name a few.

• 0:39Harry Stebbings?

  And if that wasn't enough, Aparna is also a board member at Capital One. And I do also want to say a huge thank you to Vlad at Robinhood for the intro today. But before we dive into the show today, in the ever-changing landscape of work, how can you ensure your people, workflows and data stay connected? This is how. Airtable allows you to create flexible systems that help your team operate as one, break down the barriers that leave teams feeling disconnected. With Airtable, your team works from a single source of truth that is always up to date. Plus, Airtable keeps your projects tied to the ambitious goals of your organization and keeps all teams working in harmony towards them. With their plug-and-play approach, individual contributors have the flexibility to build tools that fit the unique way they work with no code required.

• 1:20Harry Stebbings?

  Airtable is bridging the information gap and bringing teams like ours together. I absolutely love it. Give it a try for free at airtable.com today. That's airtable.com to get started for free. And speaking of amazing products like Airtable there, product leaders trust Product Board to enable their teams to build products that matter. From startups to industry titans, over 5,000 companies rely on Product Board to get the right products to market faster, including Microsoft, Zoom, UiPath and Gainsight, to name a few. Product Board can help you create a scalable, transparent and standardized process, so your PMs really understand what their customers need, and then prioritize the right features to build next. Stakeholders feel the love too, with an easy-to-view roadmap that automatically updates,

• 2:03Harry Stebbings?

  so everyone knows what you're building and why. Make data-driven product decisions that result in higher revenue and user Adoption, your product teams will be empowered to create delightful customer experiences. Visit productboard.com forward slash teams to learn more. And finally, did you know only a third of product teams actually do continuous product research? A lack of time, tools and resources mean user insights are being left in the dark. With Maze, you can shine a light on what your users really want and make strategic product decisions faster. With versatile features, from prototype testing to card sorting and surveys. Maze empowers over forty thousand brands, from Samsung to Braze, to test and act fast. So whether you're a product manager or an entire team looking to scale learnings.

• 2:48Harry Stebbings?

  Maze helps you collect deeper insights while saving your team time and headaches. So if you're looking to ship products faster without losing out on valuable feedback. Check out maze.co, that's M A Z dot co and get thirty percent off for three months. That's thirty percent off for three months with the code two zero V C. But that is enough from me. So now I'm so excited to hand over to Apurnath Chenapragada, Chief Product Officer at Robinhood.

• 3:18Speaker 01

  You have now arrived at your destination.

• 3:21Harry Stebbings?

  Apurnath, listen, I'm so excited for this one. I heard so many great things from Vlad. So thank you so much for joining me today.

• 3:27Speaker 02

  Thank you for having me, Harry. Likewise, Vlad raves about you, but so do many others. Super excited to be here.

• 3:33Harry Stebbings?

  Listen, I pay Vlad a tremendous amount of money to say such nice things, but I would love to start with a little bit on you. So, you know, we look at this incredible career working with some of the best product teams with the likes of Google and obviously now with Robinhood. How did you get your start and come to lead some of the most powerful product talks with Google and Robinhood?

• 3:51Speaker 02

  I have to take you back several decades, Harry, if that's okay. I grew up in India. I was the black sheep of the family. I was literally the only, I am the non-doctor in the family. Like, it's a walking Indian cliche. Mum, dad, brother, sister, uncle, aunt, you get the idea. And then I said, oh, I love math. And then they just, they were like just shocked. So I went into computer science. I fell in love with the C S and did my undergrad. And then I joined a PhD program in computer science. And then my mom was like, oh, great. At least you'll be a fake doctor. And then I dropped out. But I got my first break, I'd say, in what the kids would call Web one, MIT company, a spin-off called Akamai. Think about this as the infrastructure layer of the internet, and how do you actually deliver content to any website anywhere in the world?

• 4:34Speaker 02

  And that kind of, I think, got me hooked into just like both building products, making them like coding, but also just like over time, actually figure out how to build products and scale products. And of course, at Google, I did a bunch of that.

• 4:47Harry Stebbings?

  I have to ask you, twelve years at Google, it's an incredible amount of time. What were some of the biggest takeaways from your time with Google, specifically, I guess, with regards to product? And how do you think it had impacted your mindset?

• 4:58Speaker 02

  Yeah, twelve years is a long time.

• 4:59Harry Stebbings?

  Yeah.

• 5:00Speaker 02

  You had me for reminding me.

• 5:02Harry Stebbings?

  Sorry.

• 5:03Speaker 02

  You know, it's actually, I feel like it's like three Googles, even during the time I was there. And I think I'd say the first Google was, this is just like mainly search, starting to make forays in a desktop world. You have to remember, it's like 2009, and like mobile wasn't even really a thing. And that era was, for me at least, it was a front row seat into one of the life-changing products, and it still remains, Google Search. And it's one of those fascinating things where you say, oh, wow, like, there's such complexity in the engine that powers it all, all the intelligence. There's just one search box. So it's just like, anybody can do this, right? So how do you kind of have that, like, work in that highly constrained environment to build products and features? That's one thing. And I think the second Google was more around, okay, how do you actually then

• 5:46Speaker 02

  start to think about the mobile era? What products make sense there? How do you grow maps? Me personally, I worked on Google Assistant, this idea of saying, how do you have like such a personal device that can be your assistant, versus a desktop or a laptop? And then the third era now, I think the companies trying to do this is like saying, what is the next era? What's that big next act? Is it augmented reality? What's that big next platform shift? I'm biased, because I'm really, really bullish on finance and intersection of tech as one of those big platform shifts with crypto, but also broadly financial services.

• 6:19Harry Stebbings?

  Now, Apuna, the questions are going to get easier, okay? So bear with me on this one. But, uh, product. Is it more an art or a science? And how do you approach that question?

• 6:30Speaker 02

  I think this is one of those, the shortest but most unsatisfactory answer is, I guess it's yes, but also it's like it depends to overcorrect. I think typically, I think the last two decades, especially on the consumer side, it's been, you know, with Google, with Facebook, it's been thought of a lot more as kind of the data-driven, technology-driven, science. Overcorrect that I would say there's a lot more craft than people think of. I think of like these three different stages of product management, right? There's a solves stage, there's scale, and there's sustain, if you will. And I think at the sustain and grow, kind of like this large-scale billion-user products, of course, you have to have a lot more disciplined, left-brain analytical approach to it. But at the solve stage, you gotta have that nonlinear thinking.

• 7:13Speaker 02

  So I think like the answer really depends on, well, what products are you building? And what stage and in what context?

• 7:18Harry Stebbings?

  I think product management kind of encapsulates all three. I mean, different roles of product management within the three. But I find it's a term that's kind of so loosely thrown around saying it means a lot to different people. What does product management mean to you? And how do you define product management?

• 7:32Speaker 02

  That's a good one. Exactly. Like one of the most overloaded terms, I think, and overused terms. The only other one is web three, I think.

• 7:41Speaker 02

  Close there. To me, I think my definition is a product manager is someone who solves problems for people. That's it. That's the main thing. I mean, as someone said, the main thing is to keep the main thing the main thing. There's a whole bunch of other things that you need as a product manager to get shit done in a company. But the main job you have is are you solving problems for people as in customers, could be consumers, enterprise, and each of these is an underlined word. So if you say solves problems, in the beginning, you are the one solving the problem. As you grow in your career, you're getting problems solved. You have a team, you have a machine, like a system that helps the problem solving, like that systematizes problem solving.

• 8:22Speaker 02

  Problems is also underlined, which is, I don't think it means things that are very well-defined in the beginning. Again, early stage of products or even like well into a product, pre-PMF, you actually don't know the problem. You kind of roughly know the space. And some things are articulated, some things are unarticulated. And you just, the art and the science of it is like really. Really teasing apart, what's the real problems? It's so cliched to say this, but it's that Jobsian thing about the faster horse versus the car. Like, people wouldn't have like said, oh, I need an iPhone. But in some cases, I think there are very, very big businesses and really meaningful products to be built with faster horses. So I think that's a discrimination. And then the last underlying word for me is people. If you're not solving for real people, so I think oftentimes you get abstracted away.

• 9:04Speaker 02

  This is a big disease in consumer product management, as I see in large companies or at large products, where you're so abstracted away from the individual. At the end of the day, there's a person using that tool. Even if you have an API business, it's a developer at the other end. Once you have like a billion user product or hundreds of millions of people, it is so easy to just look at the graphs and like, you know, growth machine and all of that and forget the real people. So I think that's one of the things I think too. So solve problems for people.

• 9:31Harry Stebbings?

  How do you determine whether to follow customer feedback versus go on gut intuition?

• 9:37Speaker 02

  To me, you see tweets, forms from people, you see kind of like books written about product management. For me, it is really context and stage and person dependent. In fact, actually, I'll give you a personal example where I built this product called Google Lens. The idea is why isn't the camera the search box for the world? Everybody knows how to use a camera, people are taking a whole bunch of pictures. Like, there's tons of things that people want to search for just using the camera. They can't describe it in words. And one of the first few things when we did the customer research, all focused on the US. And of course, we got all these like fancy use cases, right? Oh, like, wouldn't it be cool if it overlaid like a scene with all these like, you know, interesting ways to decorate the room, et cetera? And then we hit gold when the user researcher actually said, hey, actually, we should also do this in India.

• 10:20Speaker 02

  And it turns out the use case that actually really popped there and was powerful was using the camera to translate science, because the functional literacy is low enough that researcher said, it is there, that truth and that insight is right there. And people are articulating in like odd words. So it's both of these things, you have to listen to the customers, but where and who? But also the other's part of it. And this is where user research, and I see this like front and center in Robinhood, it's one of the best places for researchers, is to say, how do you process, what questions do you ask? For example, ask Questions about their current and past behavior and problems. Don't ask questions about would you use the solution? Because people can't tell, like it's hard for anyone to like predict what they'd use. Anyways, as you can tell, I'm really passionate about like this part of product management.

• 11:03Speaker 02

  Whereas that there's kind of this delicate dance of asking customers. There's a dialogue, but it's not like pass through filter for everything that they're saying.

• 11:10Harry Stebbings?

  I wanna go to a hypothetical situation, okay? So let's pretend that I'm actually a talented founder, which is a stretch, so let's get creative. And you're an advisor to my company, it's an early stage company. Upon a, when do I hire a product manager? What are the signs that I need to hire my first product manager?

• 11:27Speaker 02

  So as someone who has actually literally given this advice to someone as of two weeks ago, I'm gonna cheat a bit and share what I said to this founder.

• 11:34Harry Stebbings?

  Right.

• 11:35Speaker 02

  I said two things. One, the biggest risk here is an act of commission, not omission. What I said was, you're gonna make a bigger mistake. And like it's a bigger negative impact on the company if you hire the wrong person versus hire the right person. It's a high leverage hire, unless you absolutely, absolutely need it. At the earliest stage, you're the founder, you are the one solving problems for people. So if you outsource that job, that's a big risk for you, right? You need to live and breathe that problem enough that at some point, if it becomes obvious that there's another person or there's a co-processor that'll help and like there's arms and legs that are needed here, yes. But until then, don't do it because you can't think of the problem for people. Because that is a symptom of a bigger problem there, right? And you can't solve it by hiring a product manager.

• 12:20Speaker 02

  I think the second is, I would say, one of the questions people ask in this ponder as to is like, well, folks say about pre-product market fit, what's the role of a product manager? It is actually something similar to what we just described. I think a really effective, like hands-on, solve-stage product manager can be the boots on the ground, like actually talk to customers, actually do a whole bunch of research and connect the dots from like what the problems, I translate that to what is the minimum viable product that is being built. And so you need somebody who's just like deeply invested in the problem space. Do not hire a general athlete from PM is my advice.

• 12:55Harry Stebbings?

  Okay, so we figured out that I need this. This is the person that we're going for. Why structure a hiring process for a PM? I've never hired a PM before. Like, what does it look like? What does the first meeting look like? Second, how do I do it?

• 13:08Speaker 02

  You know, obviously, the Google stage hiring process is different from like what I've been doing, building the team in Robinhood. I think across the board, there's one invariant, which is you have to decide upfront, what stage are you hiring for, right? So in this your case, if you're saying like it's an early stage product, the most important factor I would look for is just like the hustle and drive at that stage. At a later stage, and for a different product, the domain experience may matter a lot more. But my bias and my preference is really higher for the hustle and extremely strong, high tolerance for ambiguity and uncertainty. Because like if they get queasy on the roller coaster ride, they're not the right people.

• 13:44Harry Stebbings?

  Okay, so what questions do you ask to tease out the hustle, the drive, the comfortable being uncomfortable?

• 13:51Speaker 02

  So there's two things. One is it's what they did versus what they're saying. Because some folks interview very well and that doesn't often always translate. In fact, actually, like I think studies have over and over proven for things that are kind of like, there's a lot of practitioner judgment and decision making discipline involved. The interviewing correlation is only weak. So I would look a lot at kind of the back channel on this person. For example, and this sounds like a little bit of a self critical, or I'm not criticizing Facebook or Google or any big companies, but there are different phases for big companies. Sometimes like there's all these, you know, amazing tailwinds that help companies grow. You know, if you were working on news feed in Facebook or search, of course, the web grew with it, mobile grew with it.

• 14:34Speaker 02

  So you have to differentiate between people who were in the parade versus helped the parade. And you certainly have to look for the jerks who pushed everybody aside and somehow got to the front of the parade. That's why having a strong and diverse network helps here.

• 14:48Harry Stebbings?

  Are there any literal tasks that you do to determine skill levels with product people?

• 14:52Speaker 02

  Oh, yeah. All of this is on top of kind of the actual like test, right? The best thing is to simulate what they would do. In my experience, whatever done in the past, and I'll be doing that in Robinhood too, is think of it as a case study. Have people think about what are the aspects? Here's a problem scenario. Here's a product opportunity to build. What would you do? And for a startup, that's a fantastic way, by the way, because this is like, almost like you're simulating how they would think and how they would operate. What you can't get that you can evaluate the product thinking skills. And on top of it, you do want to kind of like, you cannot evaluate the will they get it done, right? That's the hustle and like, there will be roadblocks. I mean, it's by default shit breaks. You have to be able to do that and like, go against that.

• 15:32Harry Stebbings?

  When you've made hiring mistakes, what have those mistakes been?

• 15:35Speaker 02

  Oh, do you have five hours? Okay. Look, if I look at the patterns or the common patterns across hiring mistakes, both the ones that I've made and friends. One mistake is this timing mistake, which is overhiring, hiring sooner than required. This is an interesting one because I think in product management, by definition, it's an interfacing with multiple functions, et cetera. And like, influence is important, but mistaking noise for influence and cloud. I think that's one of the things you see that, you know, don't mistake drama and activity for progress that people have made. Sometimes I've kind of been drawn to folks who have like big personalities, right? And say, hey, they've like, really have like, strong thoughts, deep insights. But if you can't bring people along, you're shit out of luck as a product leader.

• 16:19Speaker 02

  Because it is one of those things, solving problems for people doesn't just happen within an organization. I often say, sometimes this season's firefighters, last season's arsonists. So you have to be careful about, that's one of the anti-patterns, I'd say.

• 16:31Harry Stebbings?

  How do you think about bringing people along? What do you do well that allows people to bring people along with you?

• 16:37Speaker 02

  I think one is, this sounds like easy to say, but hard to do kind of thing. One thing I do almost, you know, like, I overdo it, is repeat the product vision as many times as possible. And repetition doesn't spoil the prayer, as they say. So it becomes easier when everybody can see and internalize the same thing we're going for. Like I said, repetition doesn't spoil the prayer. Ha, you see what I did there? The other thing, identify the key folks there, because the anti-pattern again is like, I have seen that at Google where it's this kumbaya and everybody is trying to collaborate and it's like, you know, they're deadlocked and jammed because no one can make progress. But everybody gets along with everybody. Yay. That's not good either. So I think identifying the key folks that do need to make progress, that set of like, core set of folks.

• 17:20Speaker 02

  Then the rest of the dominoes fall and there's so there's like key influencers and they're the super fans of the product. And then the rest of them kind of can be brought along by them. So that way you're not the one constantly kind of doing everything across the company.

• 17:31Harry Stebbings?

  You know, one thing I think that everyone does terribly, I think onboarding, especially in product teams, actually. How do you think about the optimal onboarding process? Again, like, how do you think about lessons you've learned that makes the best onboarding process for product team members joining?

• 17:46Speaker 02

  Oh, this is so timely, given that in Robin Hood, obviously, we have both product leaders and product team members joining. And I completely agree with you, like, including my own experience generally is that's like an afterthought. Or we assume, especially for senior folks that figure it out. Here I have to plug in one thing that really helped me is this Coda doc by my good friend Shishir. He runs Coda and he has this amazing executive onboarding doc that I use to the hilt. And it talks about how it's onboarding, you think about as two steps, actually. One is this like first phase of come in, listen, the listen to talk ratio needs to be like significantly high. You know, just get coffee, do what is needed to kind of actually like build that camaraderie with the team.

• 18:29Speaker 02

  Don't come in guns blazing, hot takes, right? That's like really important. And it's a learning phase and then there's a doing phase, but don't wait too long to kind of make sure you're brought in for a reason, right? You gotta kind of get that first win and for that first win, like the leader or the manager needs to identify what is that first win for that person? And set them up for success and tell them who are the key folks that need to be part of their this person's success.

• 18:53Harry Stebbings?

  Yeah, I totally get you. When we think about like the first month for a new product manager, what does that look like? How should we onboard them? What should we expect of them? How do we set them up for success?

• 19:04Speaker 02

  Yeah, if you think about a PM, absolutely the first phase, dive deep into the product itself. If there's an existing product, there's an MVP, that's a separate, but it's just like know everything there is about what the product is. And you should know the nooks and crannies of like every part of the product, PMs in Robin Hood, if you're hearing this.

• 19:21Harry Stebbings?

  Take notes, people.

• 19:22Speaker 02

  Everything about and use it across different platforms too, right? Like I see sometimes folks are like, oh, I use it on my iPhone and it's great. And then I'm an. Android user and it sucks like you didn't look at it or whatever, right? So I think like different platforms, that sounds unsexy, but that's the most important thing. Like just load up that whole context. That's one in the very first week, if you will. And then the second, really immerse in the data and actually look at who's using it. Who are the customers, right? So first is product, second is who are the customers? Jump into research sessions, join them, like watch and learn from kind of like what are the insights? What are people talking about it? Go on any forums on the app, like really look at what customers are saying about the product. And so by the end of the month, you should know at least for your domain or your feature, your product, both the products and the customers.

• 20:07Speaker 02

  And the third one, this is again a preference. I like you could do it without it probably, but I feel strongly about it, which is the how, the tech. So sit down with the engineers, actually figure out what are the key systems here? What's the journey of the customer through the systems, through the product? And so the inner wiring, inner engine, this partly comes from my like training as an engineer and like the fact that I found it extremely useful for very tech-heavy products. I mean, Google products, they're great products to use, but they also brought the technology gun to the knife fight. Unless you understand the innards of it, you don't know what change takes two weeks versus what change takes two years. And the fastest way to lose credibility with the team, with the engineering team as a product manager is suggesting something that sounds simple, but that takes three years

• 20:51Speaker 02

  and not understanding that there's like an eighty percent solve there that probably takes three weeks. And you won't get that just like that.

• 20:57Harry Stebbings?

  On the flip side, what are the signs that the new PM's not working out?

• 21:01Speaker 02

  You know, I've made mistakes here too. And the mistake I've made in here is sometimes trying to make it work for too long. Because you obviously like see the power of this person and recruiting is a long process. If someone's kind of like, it's the same as customers, like acquisition is more expensive than retention. You wanna kind of figure out and make it work. But the two patterns I've seen there that bite people, one is just like competence. If you're an infrastructure engineer, you can say, okay, these are hard skills that can be like somewhat translated to different environments. If you've worked in Uber, it should roughly kind of work in Google. But I think with product management, so much as context and stage and other people dependent, like because you're a piece, you're an important piece of big.

• 21:40Speaker 02

  Jigs our puzzle. And so one is just the contextual competence. They may have done really well in a different environment, but they're not thriving, right? It's an ecology, like they're not thriving in this organism because they just don't understand the domain or they haven't kind of like.

• 21:54Harry Stebbings?

  My question is, what is not thriving? I don't like, is it the cadence of their like feedback? Is it their involvement in team meetings?

• 22:02Speaker 02

  Yeah, there are two kinds of not thriving that I've seen. One is just they're making poor moves. In the first month at least, they haven't actually done the thing about like just getting deep into the product and deep into customers and working with, like building those relationships with other key folks, because the place existed before you came. Things were working. Yes, you were brought in to kind of like make things significantly better. If you don't kind of acknowledge that, like there is an existing ecology and you have to fit in, that shows because other people start to say, well, like this is not being useful at all. So make yourself useful is the first thing that you have to do. That is pretty clear. And that's actually fixable. If you catch that then, I think it's most competent people, they should be able to.

• 22:41Speaker 02

  The second kind of not thriving is less fixable, loosely bucketed as culture fit or kind of like they're competent, but they're not a culture fit. They're doing the right things, they're saying the smart things, helping, they're useful in some ways, but they're just so alien to the culture. Every company is an ecology, there's a culture fit and if you don't fit in there, you get organ rejected.

• 23:01Harry Stebbings?

  You know, a really interesting one, which is different according to different cultures. And I'm kind of weirdly passionate about it considering I'm not a product person, but product reviews I find fascinating. When we think about how you as a product leader like to structure them, how do you think about structuring product reviews? How often do you do them? Who's invited? Is there an agenda? Like, take me inside the world of product reviews for you.

• 23:22Speaker 02

  Oh, now we have to talk for three hours. I am big on product reviews. I'm also a huge fan of minimum viable process. I don't like process theater as I see sometimes where people are just like doing a bunch of things and producing docs and like reviews and a whole bunch of things. And then that kind of fogs up the space. But I'm a huge fan of product reviews and I've done that at Google, I do that at Robinhood and I dedicate like a meaningful amount of my time and I tell my product leaders to do the same. But there are different kinds of. Reviews. This goes back to what we were talking about earlier on the stages of the products, the solve, scale and sustain. When the product review is for an early fragile idea,

• 24:02Speaker 02

  still in the solve stage, seeking product market fit, you want a small group and you want the problem solvers in the group, regardless of the organizational hierarchy. And you need a divergent mindset. You don't want to kind of say, okay, what's the action item? Okay, okay, okay, this thing clearly doesn't work because like we asked two people and they don't want it. You need that white space. And then you still need somebody to kind of like track the action items and all of that stuff. But you need ideas of fragile and you can't deal with them the same way you do in a scales. And then at a scaled stage product review, you want to lead with the data. You want to lead with the metrics, you want to kind of see, and you want to lead with the framework. Like by that time, this product has hit product market fit, there needs to be a framework to say, okay, what are the key knobs here? What are these three or four things that

• 24:45Speaker 02

  need to happen? And who's responsible for it? And is it a decision making forum? Is it an update? I generally don't like just an FYI update reviews as because you can do it asynchronously.

• 24:54Harry Stebbings?

  So then how do you structure them? Is there an agenda? Item one, this product feature, item two, this product feature? How do you structure them if it's not an FYI thoughts?

• 25:02Speaker 02

  Yeah, FYI thoughts are the worst. I think, again, like FYI thoughts are fine if it's the earlier stage we were talking about, because it is actually dangerous to prematurely structure the first kind of reviews where they're early solve stage discussions. On the scale and sustain stage reviews, I typically have the area leader, there's one clear like product leader that owns the agenda, right? That owns what needs to be, because it's an expensive meeting. And typically, I do have key cross-functional folks. What that means is if this product has big implications on marketing, of course, we need to kind of make sure that person's there. If it has significant impact on customer service, let's make sure that like those folks are there. Definitely have research and design at the table. I'm a huge, huge proponent of that. I remember even cancelling

• 25:45Speaker 02

  some reviews back in Google when like folks would forget to bring the researcher design leads. I'm like, yeah, we're not doing this. That's such a pleasant and positive surprise for me at Robinhood, which is designers and researchers that are there from the beginning. It's only Company that won the Apple Design Award. Design and research need to be at the table too for reviews.

• 26:05Harry Stebbings?

  Being the product leader that you are, your words carry weight upon her. How do you communicate in a way that is feedback to be ingested, not a directive to be followed? That's tough.

• 26:17Speaker 02

  Yes, it is. As recently as yesterday, I was in a review where, you know, that's a lesson I've learned over time, which is when you're a senior leader, your whisper is heard as a scream, right? You just say, hey, you guys should think about it. And the next thing you know, they've spun up like gigantic effort to say, and like it's a telephone game. Having seen it a couple of times, I explicitly preface it. I call it like this is the iTunes Terms of Service disclaimer. This is super long preface and say, hey, by the way, what I'm going to say is I have very strong conviction. This is what we need to do. That's a directive one. And then some other cases, I'm like, you guys should consider this. If you don't do it, to be honest, if you don't do it before the launch, I'm not going to cry a river. But I do think that there's something here and you guys should explore. As a leader, you have to have to separate that out.

• 27:00Speaker 02

  And that's probably one of the biggest mistakes me, other product leaders, senior execs make at any company.

• 27:06Harry Stebbings?

  It's hard. These questions are not getting easier.

• 27:08Speaker 02

  Yeah.

• 27:09Harry Stebbings?

  How, liar, Harry, that this was getting easier. I do want to say, because there's a lot of thoughts, there's a lot of feedback, there's a lot of energy in product reviews often. How do you prioritize what to focus on when there is so much feedback and thoughts coming out of them?

• 27:24Speaker 02

  Yeah, and I think I often joke saying, one of the best things I learned from Google, wide-eyed pushy rail PM, taking good meeting notes. People think of it as, oh, taking notes, isn't that something, somebody else's job? Google was like maniacal about kind of making sure that the product managers, ones responsible, took the notes. And it was a very good, very specific structure. I forget who it came from, maybe from Jonathan Rosenberg or Marissa Mayer or someone. You had very clear action items, you had specific people, specific things that they had to follow, discussion summary, key decisions made. And then on top of it, the twist I put on top of it, not all action items are made equal to your point. Some are considerations, some are like high priority things, some are like, don't even think about doing anything else, like fix this. And so what I ask is typically the lead who's responsible for that discussion to own the.

• 28:10Speaker 02

  Prioritization and then like publish that back and say, hey, by the way, these are all the things that we need to do. This is a P zero. We're gonna go fix that. And that's where you do need the judgment of the product area. If you can't trust that person to kind of make that call and you're making that call constantly, you're failing.

• 28:24Harry Stebbings?

  What do you think are the biggest mistakes that people make when conducting product reviews? Why do they go wrong?

• 28:30Speaker 02

  One is certainly this stage appropriate review, as I call it, right? Like they do it both ways, right? Which is you have an early idea, there's like a really passionate engineer and a product person who worked on it. The researcher is saying, here are some really good insights. They might be product market fit here. And then immediately you're thinking, how is this going to scale? This is a classic big company thing. It's like the South Park thing, right? Step one, interesting ideas. Step two, question marks. Step three, how does it scale to billion views? Don't rush to step three. Think about actually like sit with that. The other extreme is the cacophony of a review, right? Like no one knows who owns the, like, what's the agenda? Why are you here for? Is this a decision making? No one's framing the conversation. There are obvious fixes for both of these things, which is actually like specific people in charge of it and setting the tone.

• 29:14Harry Stebbings?

  Have your product reviews changed in terms of participants with COVID and Zoom? Allowing more people to actually be in a room, but also to kind of not be in a room?

• 29:23Speaker 02

  Yeah, it's interesting. We tried a couple of experiments here. One is like have two classes of folks. So one group is kind of the fishbowl where like you have a set of people who are actually participants, but you allow observers. And so actually that's a great way for people to have peripheral vision. But it works only for like kind of some kinds of product reviews where you're really going deeper into product. And people actually can like watch that, but it's a bit unsatisfactory, obviously for people who are observing. But it's actually been good to kind of like expose a few folks for, you know, who are junior or kind of like learning the ropes. It's a great way to kind of train them, right? They can't actually be in every product review. You normalize what you expect.

• 30:03Harry Stebbings?

  I do wanna just ask two more questions before we dive into a quick fire. And the first is like, when you look back at product decisions that you've made, can you take me to one that was a mistake? And how did it impact your mindset?

• 30:16Speaker 02

  Oh, again, do you have three hours, Harry? No, I'm kidding. I have a couple of interesting ones, which I feel like I got a lot of mileage out of like post facto. At that time, it felt really shitty, but. This is around, I think it was two thousand eleven or so, again, like, mobile phones were starting to be a thing. And I had the hunch, which is the right hunch, that personalization was like, your device is yours and like, the humans are adapting to computers. Computers need to adapt to humans. The phone should do a lot more for me, damn it. But where I started, though, is that actually, like, I wrapped up excitement around personalization as an effort in search. And so, like, you type in something, depending on like, who you are, maybe like, you need different results and different content, et cetera. Long story

• 30:58Speaker 02

  short, that hypothesis didn't pan out. And it turns out that actually, the most important signal, once you in something like search, is what you specifically, or I specifically, put in. I mean, other than like, location, if you, of course, you type in pizza, or I type in pizza, you want different results. But for the most part, that wasn't actually like, making a big difference to customers' happiness.

• 31:17Harry Stebbings?

  How did that change your mindset? What mileage do you think you got out of it?

• 31:20Speaker 02

  The one big thing I got out of it was like, to actually like, have a limited test of the hypothesis. I mean, Google was also the land of abundance in terms of number of people who are ready to jump on an idea. It just grew too fast, like, there's a bunch of people working on it. And the ROI on like, what you got out of it was so low. That's one. Like, how do you kind of stage the investment? Especially if you're a startup and a younger company, that becomes really important. But I think the second one is this, that same idea, I never let go of it. I thought, oh, that's because it's a pull, but what if it's a push of content? And so that led to a product I worked on called Google Now. And the idea was saying, hey, like, if you have a flight upcoming, or if your regular commute, and there's like, traffic delays, we should tell you. Like, now, that's a very personalized

• 32:03Speaker 02

  thing, right? Like, your thing will be very different from mine. And that was really helpful for customers. So it's like, sometimes, you know, you're one zip code away from the right idea, and how do you tweak that?

• 32:13Harry Stebbings?

  When it goes wrong, and you're the leader, and you've got a PM who's hurt, confidence-wise, they're hit. How do you bring them up and make sure that their confidence doesn't get too damaged despite a failed product?

• 32:24Speaker 02

  I think part of it is you as a manager and a leader, having their back. Because I think the worst thing you can do is like, fault people for having a hunch, having a hypothesis, and like, taking a risk. I think part of it, honestly, is the performance and evaluation and like, growth culture. Of the company and how you encourage those kinds of bets. Because if you have a company where you say, hey, like, what metrics did you move? And that's the only way you can actually get recognized and promoted or evaluated. Then what you end up having is a bunch of risk covers folks who'll just all gravitate towards the sure bet, right? Which is not great company.

• 32:54Harry Stebbings?

  Final one, you've made angel investments as well. How's angel investing impacted your product mind, do you think? From the experiences that you have now working with a lot of early stage companies on the investing side?

• 33:04Speaker 02

  Yeah, I didn't think they would connect in very obvious ways. But now I do see like two ways that I both helpful as an investor, but also help bidirectional. You know, sometimes like in products, in organizations, you're caught up in this buzzword bingo, right? AI, big data, da da, web three, NFT. When you talk to builders, you kind of get down to the frontline obsession, as you call it. I think Amazon calls that. You just kind of like basically say, remove all these levels of direction and say, what problems do people have and who's solving the problems? And I think that's a very, very clear mindset to keep coming back to, regardless of what level you are at a company. That's kind of like in one direction. I think the other direction is, I think, like some of the problems that the founders describe. I have seen this movie before and there are two possible endings.

• 33:49Speaker 02

  Just like the scripts go, that's been interesting to see.

• 33:52Harry Stebbings?

  Listen, I want to move into my favorite, which is a quick fire answer. I say a short statement, you hit me with your immediate thoughts. It's incredibly tense. Are you okay with this?

• 33:59Speaker 02

  Oh, sign me up. Let's do it. I have no idea what you're gonna ask, but sure.

• 34:04Harry Stebbings?

  Okay, so what in product have you changed your mind on recently?

• 34:08Speaker 02

  I changed my mind actually about Twitter. Like I previously, obviously still like the product like, but I felt like, oh my God, if only this could change and that could change. And I just feel like they're on a roll. I love the product momentum. Part of it, I think, is like having strong engineering systems. So I love seeing that.

• 34:23Harry Stebbings?

  I actually just had Kaivon on from Twitter earlier today.

• 34:26Speaker 02

  Kaivon, yeah.

• 34:27Harry Stebbings?

  It was fantastic. But no, totally agree with you there. Tell me, what one piece of advice would you give to a product leader starting a new role today?

• 34:34Speaker 02

  Step one, make coffee for the team. You can also do it literally, but just like be helpful. Step two, solve a problem for, focus on specific problems that the team has. Step three, be patient. Don't rush in.

• 34:46Harry Stebbings?

  What's the biggest challenge? Of your role with Robin Hood today.

• 34:49Speaker 02

  I think it's a multi-dimensional problem, right? So previously, I was just responsible for building products. Now I feel like as an executive at the company, it's company building and product building at the same time in a very, very fast-moving environment. It's a lot of fun.

• 35:02Harry Stebbings?

  A lot of fun, but I get you on the challenge. What would you most like to change about the world of product today in terms of like product thinking?

• 35:09Speaker 02

  Yes, one myth I really want to change. There is implicitly or explicitly, it's all these tech bros with hoodies talking about like, I mean, we have this myth that those are the product guys, the product people. Awesome product people come in all colors, all shapes, all forms, all ages. I think that's one myth, like that somehow there's one type, prototype. I think if I can get a bonus thing to change, I think it is really, there's a set of people who talk about products. There's a bunch of other people building products and often they're not the same. I think like we should get more practitioners to go to the balcony. I mean, many of us are so busy on the dance floor, like so busy doing the things that we're, we don't often get the time to kind of reflect back.

• 35:52Speaker 02

  And if we can do, get more folks, that's why I love what you're doing here. If you're gonna get more folks to kind of like actually share their battle scars and share that it's not this glossy shiny thing that like just shows up on cute tweets.

• 36:03Harry Stebbings?

  You set me up, bring it to that, okay. Who's the unsung hero in product for you that maybe is more behind the scenes?

• 36:09Speaker 02

  This is a longer answer, but I think I feel strongly about this, which is you should look at people who have worked through the hard parts of upper companies' evolution. It's the temptation would be to say, hey, here's a hard company, let's get the folks who are right now like riding the wave to talk about them or to talk to them and highlight. The unsung heroes are the ones that actually stuck with kind of the ups and downs and actually make the company emerge. So like, look at companies that did go through the up and down. I think Snap comes to mind, by the way, like even though maybe Evan is the PM there, but I think like this idea of the roller coaster ride, but folks have kind of steadfastly like let the companies and products through that.

• 36:46Harry Stebbings?

  Final one for you. What recent company product strategy have you been most impressed by, other than Robin Hood?

• 36:52Speaker 02

  Actually, I have a very like non-product example for this. Just maybe it's recency. I was just watching the This new like thirty minute bit from Aziz Ansari on, I think on Netflix. And I really like the fact that, you know, short of controversy and like kind of like this big rise, there's like turbulence and the way he consistently kind of came back and quietly started to like work on the craft in an authentic way, I think is a great example. By the way, that's the other peeve I have about product world, which is all the metaphors are just like sports or war. Like I love theater and film, so like often my metaphors are like from that world.

• 37:26Harry Stebbings?

  Apana, I have said this has been so much fun. Thank you so much for joining me. Thank you for having such great character, for emotion. Honestly, it's not often that I meet someone with such passion that's so evident. So really, thank you so much for doing this for me.

• 37:39Speaker 02

  You can tell. I love my craft. I love my field. I feel very lucky. No, thank you for having me and for all those builders out there, keep at it.

• 37:49Harry Stebbings?

  On this one, it's pretty simple for me. Apana is just an incredibly special person. I absolutely loved doing that show. She was incredible. If you'd like to see more from us behind the scenes, of course you can on twenty vc dot com. I always love to see you there. But before we leave you today, in the ever changing landscape of work, how can you ensure your people, workflows and data stay connected? This is how Airtable allows you to create flexible systems that help your team operate as one. Break down the barriers that leave teams feeling disconnected. With Airtable, your team works from a single source of truth that is always up to date. Plus, Airtable keeps your projects tied to the ambitious goals of your organization and keeps all teams working in harmony towards them. With their plug and play approach, individual contributors have the flexibility to build tools that fit the unique way they work with no code required.

• 38:33Harry Stebbings?

  Airtable is bridging the information gap and bringing teams like ours together. I absolutely love it. Give it a try for free at airtable dot com today. That's airtable dot com to get started for free. And speaking of amazing products like Airtable there, product leaders trust Product Board to enable their teams to build products that matter. From startups to industry titans, over five thousand companies rely on Product Board to get the right products to market faster, including Microsoft, Zoom, UiPath and Gamesite, to name a few. Product Board can help you create a scalable, transparent and standardized process. So your PM Really understand what their customers need and then prioritize the right features to build next. Stakeholders feel the love too, with an easy to view roadmap that automatically updates

• 39:17Harry Stebbings?

  so everyone knows what you're building and why. Make data-driven product decisions that result in higher revenue and user adoption. Your product teams will be empowered to create delightful customer experiences. Visit productboard.com/teams to learn more. And finally, did you know, only a third of product teams actually do continuous product research? A lack of time, tools, and resources mean user insights are being left in the dark. With Maze, you can shine a light on what your users really want and make strategic product decisions faster. With versatile features, from prototype testing to card sorting and surveys, Maze empowers over 40,000 brands, from Samsung to Braze, to test and act fast. So whether you're a product manager or an entire team looking to scale learnings, Maze

• 40:01Harry Stebbings?

  helps you collect deeper insights while saving your team time and headaches. So if you're looking to ship products faster without losing out on valuable feedback, check out maze.co, that's M-A-Z dot co, and get 30% off for three months, that's 30% off for three months with the code 20VC. As always, I so appreciate all your support and I can't wait to bring you a fantastic episode this coming Friday with Enrique at Brex.