Career frameworks, A/B testing, onboarding tips, selling to engineers | Laura Schaffer (Amplitude)

Laura Schaffer is the brand-new VP of Growth at Amplitude. Prior to this role, she spent over 10 years leading product management and growth teams at Twilio, Bandwidth, and Rapid. In today’s episode, we talk about the role of experimentation and data in growth, and Laura shares stories of big wins from her time leading growth teams. She explains how customer insights helped her uplevel her career and how she (surprisingly) thinks about qualitative versus quantitative data. We wrap up our conversation by discussing where the best ideas come from and what you need to know if you’re selling to developers. — Brought to you by Public—Invest in stocks, treasuries, crypto, and more | Eppo—Run reliable, impactful experiments | Writer—Generative AI for the enterprise Find the full transcript here: https://www.lennysnewsletter.com/p/career-frameworks-ab-testing-counterintuitive Where to find Laura Schaffer: • LinkedIn: https://www.linkedin.com/in/lauraschaffer/ Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • Twitter: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ Referenced: • Elena Verna on Lenny’s Podcast: https://www.lennyspodcast.com/elena-verna-on-how-b2b-growth-is-changing-product-led-growth-product-led-sales-why-you-should-go-freemium-not-trial-what-features-to-make-free-and-much-more/ • Bandwidth: https://www.bandwidth.com/ • Twilio: https://ahoy.twilio.com/ • Jeff Lawson on LinkedIn: https://www.linkedin.com/in/jeffiel/ • The Surprising Power of Online Experiments: https://hbr.org/2017/09/the-surprising-power-of-online-experiments • Reforge: https://www.reforge.com/ • Online Experimentation at Microsoft: https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf • The Simple Path to Wealth: Your Road Map to Financial Independence and a Rich, Free Life: https://www.amazon.com/Simple-Path-Wealth-financial-independence/dp/1533667926 • Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones: https://www.amazon.com/Atomic-Habits-Proven-Build-Break/dp/0735211299/ • James Clear on The Tim Ferriss Show: https://podcasts.apple.com/us/podcast/648-james-clear-atomic-habits-simple-strategies-for/id863897795?i=1000592431628 • The Great British Baking Show on PBS: https://www.pbs.org/food/shows/great-british-baking-show/ • Hotjar: https://www.hotjar.com/ • Amplitude: https://amplitude.com/ • Builder: https://www.builder.io/ • ChatGPT: https://chat.openai.com/ • Lenny Bot: https://www.lennybot.com/ • Segment: https://segment.com/ • Senior Growth PM, Monetization, at Amplitude: https://boards.greenhouse.io/amplitude/jobs/6636704002 • Lead Growth PM at Builder: https://boards.greenhouse.io/builder/jobs/4814755004?gh_src=30cfda2d4us • Growth PM at Rapid: https://jobs.lever.co/rapidapi/8d2611d1-6463-4919-9817-31f61e730831 In this episode, we cover: (00:00) Laura’s background (04:15) How to carve your own career path, and an example from Bandwidth (05:50) Laura’s career growth framework (10:18) The value of customer insights (12:25) The “voice of the customer” report (16:14) Leaning into your strengths (18:16) The experiment that shifted the way Laura thinks about friction (20:20) Questions that improved Twilio’s onboarding and conversion rate (28:53) Thinking about the psyche of your users (31:26) The hot dog analogy for burying “scary stuff” (33:58) Why it’s better to be iterative and why experiments fail (36:21) Saving money by validating fast (41:58) Where the best ideas come from (49:51) Experimentation lessons (52:54) The amount of time a growth team needs to be successful (54:43) The big change at Twilio that led to tens of millions of dollars (58:41) The need for both PLG and enterprise, and how Amplitude plans to tap into PLG (1:05:42) What it’s like to serve developers (1:11:16) Lightning round Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  And like the, you know, dead of the night, and by that I mean like, you know, 7:00 PM or something on, I think it was a, pretty sure it was a Friday. We just asked for forgiveness and kind of put these questions into the signup flow and ran as an A B test with a small group. And, you know, I'm fully expecting, okay, this is going to like hurt our numbers, but maybe it won't be so bad, you know? And I'm prepared to advocate the power of this data that we're getting. And I was totally, I'm thinking of like written, start to write like the, like the framework for how I wanted to, to surface this. And we start to get the data for this thing. I'm not kidding, an improved conversion. Like we, there's no personalization, nothing past it, just the questions. It improved conversion by like 5%. Like just improved signups. And it's one of those like, what?

• 0:45Speaker 00

  Like, okay, this, like, what is going on here?

• 0:50Lenny Rachitsky?

  Welcome to Lenny's podcast where I interview world-class product leaders and growth experts to learn from their hard-won experiences building and growing today's most successful products. Today, my guest is Laura Schaffer. The week we recorded this chat turned out to be Laura's first week in a new gig as Head of Growth for Amplitude, taking over for a previous legendary guest, Elena Verna. Prior to Amplitude, Laura was VP of Product and Growth at a company called Rapid. Before that, she spent over seven years at Twilio as Head of Growth and PM Lead of the Growth Platform and Experimentation Platform at Twilio. In our conversation, we dig into Laura's career growth framework and the importance of carving your own path versus waiting for one to be carved for you.

• 1:33Lenny Rachitsky?

  We also get into a bunch of tactical and surprising advice around running experiments, making decisions on gut versus data, developing your growth strategy, and how to sell your product to developers. Laura has a wealth of wisdom and I learned a lot from our conversation. With that, I bring you Laura Schaefer after a short word from our wonderful sponsors.

• 1:55Lenny Rachitsky?

  This episode is brought to you by Public.com, who want to tell you about their new treasury accounts, which earn a 4.8% yield on your cash. That is higher than a high-yield savings account, while still being backed by the full faith and credit of the US government. Treasury yields are at a 15-year high, but buying US treasuries is super complicated if you go to a bank or navigate an ancient government website. Or at least, that was the case. Now you can move your cash into US treasuries with the flexibility of a bank account. You can access your cash whenever you want, even before your treasury bills hit maturity. There are no hold periods, no settlement days, just a safe place to park your cash and earn a reliable yield. Public will automatically reinvest your treasury bills at maturity,

• 2:40Lenny Rachitsky?

  so you don't have to do anything to continue growing your yield. And you can manage your treasuries alongside stocks, ETFs, crypto, and any alternative assets. Do all your investing in one place and earn 4.8%, a higher yield than a high-yield savings account, only with a treasury account at public.com/leni.

• 3:01Lenny Rachitsky?

  This episode is brought to you by EPO. EPO is a next-generation A/B testing platform, built by Airbnb alums for modern growth teams. Companies like Netlify, Contentful, and Cameo rely on EPO to power their experiments. Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern growth team stack. This leads to wasted time building internal tools or trying to run your experiments through a clunky marketing tool. When I was at Airbnb, one of the things that I loved about our experimentation platform was being able to easily slice results by device, by country, and by user stage. EPO does all that and more: delivering results quickly, avoiding annoying prolonged analytics cycles,

• 3:45Lenny Rachitsky?

  and helping you easily get to the root cause of any issue you discover. EPO lets you go beyond basic click-through metrics and instead use your North Star metrics, like activation, retention, subscriptions, and payments. And EPO supports tests on the front end, the back end, email marketing, and even machine learning clients. Check out EPO at getepo.com, get E P P O dot com, and 10x your experiment velocity.

• 4:15Lenny Rachitsky?

  Laura, welcome to the podcast.

• 4:18Speaker 00

  So thanks, Lenny. It is so great to be here. Thanks for having me.

• 4:21Lenny Rachitsky?

  It's great to have you. So I asked Elena, Elena, Elena, I'm not even sure how to pronounce your name. Maybe you know.

• 4:26Speaker 00

  Elena.

• 4:27Lenny Rachitsky?

  Elena, okay, okay. I think I've said it wrong all the time, all this time. Okay, Elena. So I asked Elena Verna, who's a popular guest on this podcast, who I should have on this podcast. And you were the first person that immediately came to mind. And so I'm really excited that we're doing this and that you agreed to be on.

• 4:42Speaker 00

  Well, she's the best and I'm really happy that she referred me because I'm just stoked to be here. So thanks for listening to her guidance.

• 4:49Lenny Rachitsky?

  Absolutely. And it's kind of a cool time to be chatting. You're kind of the newly minted head of growth at Amplitude. And so congrats, first of all.

• 4:57Speaker 00

  Thank you. Appreciate that. Yeah, this is my day two and a half here.

• 5:03Lenny Rachitsky?

  Wow.

• 5:03Speaker 00

  Very.

• 5:03Lenny Rachitsky?

  You're a veteran.

• 5:04Speaker 00

  Yeah, right.

• 5:06Lenny Rachitsky?

  I love it. Some companies, there's like a little percentage that shows you how many people have joined before you. And I wonder what that percentage already is.

• 5:12Speaker 00

  We have that at Twilio and I got pretty, pretty high up there. After a while, we had like a stack rank in a spreadsheet and yeah, but it's, it is funny. So wherever, wherever that thing exists in Amplitude, I am like right fresh there at the very bottom.

• 5:25Lenny Rachitsky?

  What was the number you got to Twilio? Any, do you remember?

• 5:28Speaker 00

  Yeah, no, I was, I was very proud to crack like the top 50. That was like my claim the same because as people like left, right, you kind of like move up, right?

• 5:36Lenny Rachitsky?

  Right. Yeah.

• 5:38Speaker 00

  So it, well, yeah, right. It's kind of like on one hand, it's like, oh, very cool. I'm like one of the OGs. On the other hand, it's like, oh my gosh, like this person's color. Like that's a bummer. It's a shift, but I'm excited about it for sure.

• 5:50Lenny Rachitsky?

  So you have this new exciting role and I thought it'd be fun to start to chat about career growth and just how you think about career growth. I know you have a framework of how you think about your own career growth. Clearly it's worked out. So I'm curious to hear about it and see, and see how it could be helpful to folks that are, that are listening. So yeah, can you just tell us about how you think about career growth?

• 6:11Speaker 00

  Career growth is often right. Like it's definitely not a straight lineup, but there's definitely some kind of frameworks and methods that have worked really well for me. And I think to dive into it, it's first good to just talk about the one that I most typically see people use to try to grow their career and why that can be a little problematic. Which is that I see most people are, you know, try to work really hard at kind of the job that they have, right? Within the role that they're, that they have at a company. Do whatever you can to grow there, show your manager all these things. I see people keep spreadsheets and sort of wins. So you can cover performance reviews, you know, maybe you try to get better advocating for yourself, maybe try to get peers to notice or your manager's peers.

• 6:50Speaker 00

  And that's all good, like, that's all good stuff. But the problem with it is that you're limited to what your manager's ability is to advocate for you, to promote you. And you're also limited by the explicit trajectory of your role at that company and where the, there's room for that or not at the company. And then often that perception can sometimes be a little bit in contrast to what your perception is, right? And also other things that happen, like you're, you know, your manager leaves and then you kind of have to restart with someone else, right? So the method that I use kind of tries to take that power back a little bit. And something that I learned really early on in my career, I was very lucky to learn by accident, was at a company called Bandwidth, which is like my first kind of quote, quote, real job.

• 7:33Speaker 00

  And, you know, Bandwidth is now a public company and they've done all kinds of crazy, amazing things. But I joined when it was just 50 people and I actually joined in sales. And I was, you know, just hungry to kind of make it succeed and grow and, you know, bright eyes, everything, you know, first kind of real job. But I realized after a few months of being in sales that I was often like repeating the same thing over again, like using the same thing to sell over and over again. And it's like, gosh, like this isn't ideal for the customer because they're going to call me and ask me these questions and wait and get these answers and all this stuff. And it's not ideal for the company because they're like paying commission on this every time. Like that's not going to be efficient for our growth.

• 8:13Speaker 00

  And because we were small, I was able to catch our GM and I was just like, hey, um, You know, I've noticed this pattern where like I'm repeating things kind of over and over again. And like they're asking the same thing, like I think we should put that online. I think we should make that available so they can just like see it and then buy it. Cause we had an online checkout process and I was kind of expecting him to be like, oh, well, I know it's important, but you know, for this or this or another, like we need to like, you know, we need to do it this way. And obviously you've thought all about it. Kind of thinking like, oh, I'm going to come in like this new person. He's just going to help me understand what I'm like missing here. There's like a little bit of that that I was expecting. And he goes, wait, wait, like, tell me more about that. What do you mean?

• 8:54Speaker 00

  And by the end of the conversation, he was like, hey, why don't you like go do that? Why don't you go build that like experience? Why don't you put that stuff in like in a, on a self serve flow? And we called it e-commerce manager and it was like kind of growth before this, but this is like 2010. And that, that moved me into a totally new, a new position. And the main learning that I had from that was, which really kind of took life at Twilio. And I absolutely worked for me there and I'm happy to talk about that too. But the, the core that learning was your executive team and executive teams at companies are often very sharp. But the nature of their day to day just does not link them with customers. Right? And that means that over time, especially as a company grows,

• 9:37Speaker 00

  they often lose access to some of the best insights. And you know, in the heartbeat of the people who they're providing value to, in contrast to folks that are closer to the problem. And so that means that your superpower is in really pulling those insights in and bringing them to life. Seeing close to the customer, there's not a single leader or executive that isn't going to be stoked to hear about valuable customer insights that highlight problems they might not be seeing. And there's a lot of those. So especially, you know, when they align to North Star metrics, those, those ones are, are sort of the powerful ones. That was the way that I grew my career at FLY and I'm happy to share kind of that journey too.

• 10:18Lenny Rachitsky?

  Yeah, it'd actually be cool to hear maybe another example of that. But I think, I think an interesting thing that comes up for me here is sometimes you may have an awesome idea and it may not immediately happen. It may not be like, yes, Laura, let's move on this right immediately. And I think it's important to just recognize like they're not going to follow all your ideas. Uh, but they're always looking for better ideas and. They like to your point, they may not have the information that will lead to an idea that you will have because you're like on the ground, dealing with real problems day to day. So I think it's important to recognize you're not going to always get your way and that's normal.

• 10:51Speaker 00

  Yeah, totally. And right. And it's, it's kind of about like, almost like building up your, your individual like brand a little bit. And I think one of the most powerful and accessible ways to do that is learning about your customers. There's always those people at, at companies who's like, oh, well, you know, I mean, she just knows our customers or he just knew our customers. They just know our customers, like they just know. And it's like, well, hi, they just know, like, let's ask that person, let's get their feedback. And those people often have a good amount of kind of, you know, brand recognition and powers in the company. And they're often thought of when a company needs to do something new or different, or, you know, someone is, you know, hiring, maybe they're thinking about that person for like a cross team thing. So it's, it's one of the ways

• 11:32Speaker 00

  that you can kind of build, build that, that brand. And again, it's, it's, I think it's a sweet spot because it's something that is very valuable to everyone, all the way up to the most senior leaders, which we can talk about here in a minute. And so it's going to be valuable for you and a valuable tool, kind of no matter where you're at in your career. And yeah, and that's not always an immediate payoff, but it often does give you a trajectory outside of just your role and just your manager. It gives you something a little bit broader.

• 12:05Lenny Rachitsky?

  So maybe a simple way of describing to kind of mirror back what you're saying is kind of carve your own path. Don't necessarily assume your managers will give you the path that makes most sense for you, or even give you the biggest opportunity, just like propose, hey, I think this might be a better opportunity and I'd love to pursue it. I'd love to hear the Twilio example, if that's generally.

• 12:24Speaker 00

  Yeah. So I actually joined, when I joined Twilio, there was no growth team at all, like not even a breath of it. Uh, I joined in product marketing and I was leading our product marketing for our messaging lines. And, you know, but I followed the same guy that I just mentioned, I made it a, you know, my own kind of personal policy to like, hey, I'm going to do my job and I'm like, do well and keep notes of things I'm doing well and all that kind of stuff. Cause it's good. But I'm also going to get to know our customers and I'm going to get to know our customers really well. And I pay attention when I'm connecting with them, not just about the space I'm in, but broadly, what are some of the pain points and things they're articulating that, that are relevant to the business and what we're trying to get done. And one of the things that came up. Was that users were struggling and folks were struggling to get started and use Twilio.

• 13:09Speaker 00

  And that contrasted so deeply to some of the things that our executive team was saying and had high conviction in our company had high conviction, which is that Twilio was so easy to use. In fact, it was like top three things about Twilio that we were really trying to get out of their brand. We're so easy. Developers love us. They say we're so easy. And there's always like, there were tweets coming all the time, like, you know, developers saying like, oh my gosh, like, they got started in a couple of minutes. So there's all these things that kind of made that compound and made that conviction stick. But as I was talking to customers, I was hearing a very different story and it made sense as we were penetrating new markets, adding more products. We were adding complexity and we were pulling in folks who were a little bit less motivated. And those things contributed to people saying, this is, this is difficult.

• 13:52Speaker 00

  And so, you know, at the time, you know, this, this wasn't a, you know, 50 person where I could just kind of go to the floor and like go to someone and be like, hey, like, there's this thing I heard about, like, I think we should do something about it. But there was another tactic that I could take and this is, I just started sharing a voice of the customer report. I started sharing my insights, started writing down and just sharing them and it became this kind of digest. And eventually people were like, hey, can I, can you share with me? Can you share with me? Can you get on your list? Can you share with me? And this was in like a few months of me joining, I was doing this. And then that turned into, hey, you should like host a quarterly voice of the customer session for, for all of product, right? And, and, and this was a request that was coming from some of the senior leaders of the company. And when our Jeff, Jeff Lawson, who's our CEO at the time, heard about it, he started attending too.

• 14:38Speaker 00

  So now in the session, I started pulling in other people's insights too, right? Cause this was, these were, you know, now they had a forum for this, I could kind of do that. And, and, and, you know, have people send that to me and I could compile it and all of these things. And so then, you know, this kind of established me as that person who knows about the customer, even after short tenure. And then, you know, when, you know, came time to do annual planning that, that year, and I joined in 2014 at the end, so this is 2015. I pitched this idea, hey, We think that it's easy. It is not. Here's data that I have, the information that I have. And I think that we need to start a growth team here and that needs to be a core focus.

• 15:19Speaker 00

  And I was able to bring in really critical, um, you know, kind of partner to that and other folks who could support that because I had built up some of that trust. So by the time I was making that pitch, I had someone like, you know, on, uh, Andre Crowe, who was like the seventh hire at Twilio and got to like number three on that spreadsheet or whatever. He was really close their CEO being like, yeah, like we desperately need this. I'm seeing this. He, he led a, you know, website. He basically created the Twilio brand. And he like led all the website stuff and he's like, yeah, we definitely need this. So not only did I have that kind of little bit of trust, right? From the executive team, but I also had folks who were just trusted on their own advocating and supporting this that I was doing. And so it was approved with like, it's almost very easily. I mean, I put stuff together for it, but it was, um,

• 16:04Speaker 00

  you know, kind of the meeting before the meeting had already been done by us all the things. So it helped me create the growth, engineering growth product team at Twilio.

• 16:14Lenny Rachitsky?

  I love just how proactive your advice is here. There's a lot of people that don't do well and then just like, oh, I never had the opportunity or I kind of get kept got looked over all this time. And I love that there's all this, just like, here's things you can be doing to get in front of people to provide value, to just create opportunity for yourself. Any other advice along the lines of just like, here's the kind of things you could do for yourself versus waiting for someone to come and give you opportunity?

• 16:41Speaker 00

  Yeah. I mean, I think that's the most easily actionable because to do all of our jobs, we need to know customers, like we need to know about customer insights, product we need to know, but then also customer facing teams, especially those who want to crack into product. Your, your insights are extremely valuable. You're talking to customers every day. You know more about their problems and their pain than a lot of other people do. And so, you know, that's, that is by far and away to me that the most powerful and accessible one by, by anyone in any role, in any space. But, you know, I'll also say that, you know, that broader concept of just, hey, like there's things that you know, and things of value that you know, that others can benefit from at your company. And, and building your brand as someone that is supportive, smart, creative, able to solve problems.

• 17:25Speaker 00

  You know, make sure that you're sharing that, right? And so, you know, Maybe you're really freaking good at communicating with brevity. I suck at that, by the way. So like more power to anyone that can do that. I'm actively working on it. So, so share that, like go, go to your general Slack channel or whatever. And just be like, hey, like just run up like some tips for how to do it, you know, some ways that I am good at this. And those kinds of things can really go a long way towards people starting to view you as, you know, an SME and not just the space that you're in, but in sort of broader areas. And that can always present and open doors for you when other people are sort of looking up to you and seeing you as someone who's, who's strong in ways outside of just the role that you're in.

• 18:07Lenny Rachitsky?

  SME is a subject matter expert, is that right?

• 18:10Speaker 00

  Yes. Good. Thank you for unpacking my acronyms. That's another thing that I am actively working on.

• 18:16Lenny Rachitsky?

  I got you. I'll be on the lookout. Maybe one last question along these lines is, do you have any advice for framing the proposal? Framing an opportunity to, you know, your manager, higher ups that you see has worked best broadly?

• 18:31Speaker 00

  Yeah. So, yeah. And one thing I want to say too is like with this stuff, I don't think that necessarily does go like counter to what your manager is doing. It's more like supporting them like, right? Like I've done this kind of stuff and then it's helped my manager promote me. Right? So it's, it's not necessarily, oh, we'll do this if your manager is like failing you or they, you know, are not supporting you or they can't support you. It's more like do this because this is going to be an accelerator for yourself irrespective of your manager. But then also it'll be an accelerator for your manager and supporting you. Cause one of the things that, you know, it comes into play a lot when managers are figuring out promotions and doing those things is like they'll sit in a room, often calibrations and, you know, with a bunch of people, and it makes it a lot easier when those people have had some kind of

• 19:13Speaker 00

  access or exposure or whatever to you in a positive light. So these things can all run, you know, with, you know, your manager and not against. But it's just another way of you kind of taking, taking back the ability to build that momentum instead of relying on all of that going through one single other person.

• 19:34Lenny Rachitsky?

  What I like about your second example is you just did it. You just started doing that kind of report for the company. It wasn't like, hey, I have a proposal. Here's what I think you should do. Should we do it? Just like, yeah, just do it.

• 19:44Speaker 00

  Yeah, like, uh, un-gate your knowledge, I think, is the buzzword that I'm hearing, right? Um, and you could do that within your, I think that's an Elena-ism, like, see if we can, how many times we can, we can bring her up. But right, it's just, you know, like, you can do that within your own, you know, company. Like, everybody, everybody is skilled at things that they aren't, you know, explicit to their role or their space. And I think that ungating that opens opportunities. And if you're not sure, then go to my favorite go-to, which is talk customers, get insights. Those are incredibly valuable. So rarely do people share those when they find them. So be the person that does that.

• 20:20Lenny Rachitsky?

  Another area I want to chat about is experimentation and growth and data, which makes sense. You have strong perspectives on being the new head of growth amplitude. So maybe we start with experimentation.

• 20:36Lenny Rachitsky?

  You mentioned that you had, there's like a really interesting, surprising results in an experiment you ran at Twilio that kind of maybe changed your perspective on experimentation and what you think might work and not work.

• 20:46Speaker 00

  Hundred percent. Yeah, I'm sort of fortunate to have two mind-blowing experiments that really shifted the way that I think about growth. So one of them, one of my favorite ones, happened very early on at Twilio. So after I created this growth team, like, one of the things that I saw as, as to me, an issue was that at our signup flow, we, we just asked people for a username and an email, like a password. And that was it. And that's actually relatively common at the time. This is kind of a while ago now, like, everybody kind of is segmenting users. But we didn't. And we actually, there was a lot of existing conviction around that. It was like, hey, we, retarded developers, developers, they, you know, they just want to do, they just want to get their hands on things. Don't put anything in their way. It's going to be disastrous. We

• 21:28Speaker 00

  don't want any shenanigans here with these, these folks. Let's just let them in the gates. But to me, this was a really big assumption to make, and, and very costly one. It's like, okay, if that's the case, we don't know, we're not going to know anything about anyone. And we didn't know who was signing up. We didn't know what they wanted to do. And that hurt our ability to understand that people were performing from a quantitative perspective. You know, we were a little bit lost with prioritization. There's a number of implications here. But it's obviously a very contentious kind of space. So I, you know, This is the very first thing that I that I did, the first experiment that I ran. I I did, you know, some research and I said, okay, what are the most important questions to ask? Like, what would I do really, really need to know? And it was stuff like, you know, what what language are, you know, you coding in? And what's your use case?

• 22:12Speaker 00

  What product do you want to use? And then, you know, there's, you know, one around like, are you a developer at all? Or are you something else? Because there is sort of rumors that we're having not just developers sign up, which is a whole other interesting kind of story. And I, you know, seeing these questions that would also, you know, potentially be things that that are developers signing up would sort of understand why we're asking, that it would feel natural. And but anyway, you know, again, like adding anything to the signup was very contentious. But I really just wanted to get a little bit of data on it. So I wanted to run a test. I didn't have a team, didn't have an engineering team yet. And none of that stuff had bailed out. It was just me, myself and I. But like I said, I had started to to build a little bit of trust and pulled in, go to Andre, who I mentioned earlier, who, because he was early employee and just he kind of had access to everything, like one of those those people.

• 22:57Speaker 00

  And he had also was, you know, supportive of of this and kind of had similar hunches. And so, like the, you know, dead of the night, and by that I mean, like, you know, 7:00 PM or something on, I think it was a, pretty sure it was a Friday. We just asked for forgiveness and kind of put these questions into the signup flow and ran as an A B test with a small group. And, you know, I'm fully expecting, okay, this is going to like hurt our numbers, but maybe it won't be so bad, you know? And I'm prepared to advocate the power of this data that we're getting. And I was totally, you know, I'm thinking of like written, started to write like the like the framework for how I wanted to to surface this. And we start to get the data for this thing. I'm not kidding, an improved conversion.

• 23:39Speaker 00

  Like, we, there's no personalization, nothing past it, just the questions. It improved conversion by like 5%. Like, just improved signups. And it's one of those like, what? Like, okay, this, like, what is going on here? And I actually dug into it. And what I found from just talking to a few customers, went through the flow, I'm just like, learning about how they felt about it. It was actually for folks, it was like comforting. You know, when you think about it, when, when users are signing up for your product for the very first time, like it's new, right? This is new. That means it's scary. They're, they're expecting it to be difficult. They're anticipating that there's going to be friction and challenges and then that they're not going to figure it out. That was like looking for the bogeyman, right?

• 24:24Speaker 00

  And that's the headspace. It's often the headspace that any of us are in when we're doing something new for the first time. Like, oh, this is very challenging. And so by putting in these questions, it's like, what's your language? It's like, oh, like I do, I, I code in JavaScript and, and I'm, I can select that. Well, that's something uncomfortable and that would make my, my journey easier. Like, yeah, bingo. Oh, and you're, you know, that's, that's my use case. Okay. Like I'm in the right place here. Like, you know, it was, it was actually giving folks something comforting and, and challenging the notion that this was going to be difficult, just the questions. Because it was aligning to some of the things that they were organically thinking about, which is, what if they don't support my language? Or like, can I even do this use case I want to do?

• 25:09Speaker 00

  And so it was just a really interesting, you know, the takeaway for me for this, like the really interesting takeaway was the psyche of the user is so, so critical, right? Like that's just as important as understanding your product and the broader like market you're applying to and all those things. Like just the psyche of, of users, new people doing things for the first time with your, in your user flow. Like, understanding that is powerful. You know, and the simple, like kitschy thing I say is that, you know, ultimately the learning here was bad friction is bad and good friction is good, right? There's no such thing as being simple as just all friction is bad, which is sort of kind of what I had assumed going into this.

• 25:49Lenny Rachitsky?

  I love that you, uh, you were new to Twilio and you just Yolo'd an experiment to production.

• 25:55Speaker 00

  Yolo'd.

• 25:55Lenny Rachitsky?

  That's a. It's a big move.

• 25:58Speaker 00

  It ended up being like very, you know, very helpful for everyone. I, you know, shared the insights from it and like all these things that shared the data.

• 26:04Lenny Rachitsky?

  And it drove conversion.

• 26:06Speaker 00

  But for sure, like, you know, use such, you know, processes with, with caution, uh, for sure. Yeah.

• 26:13Lenny Rachitsky?

  I love it.

• 26:13Speaker 00

  The right way to do is not like advocating for, for the engineers here is the right way to make any changes in production is, is, you know, through or with the approval of engineering. But it was, um, it was, it was the right move overall and definitely helped business, right? So, uh,

• 26:27Lenny Rachitsky?

  Yeah, I love it. That's great. I like that move. I think we need more of that probably. I want to dig into which actually, so what is it you changed? You added, how many questions? And then what were the questions?

• 26:36Speaker 00

  There was a question around like, what language are you coding in? And then as an option to that, it was like, oh, like I'm not, I'm actually not coding, like I'm not a developer. Like, so that was, uh, it, you know, for us, it actually gave us two really, really interesting data points. One was like, you know, how many developers versus people who are not coding or in our flow? And then what language are the coding, which was massively helpful, not just for like growth and onboarding, but for documentation team, docs team, you're gonna like, what we should, you know, that ended up being a critical way for us to gauge trends over time and catch things before like, you know, whatever, you know, reports would come out in the year, what people are doing, sort of see it. And then also product, what product are you interested in using?

• 27:16Speaker 00

  That was very critical for knowing kind of the basics of how to organize someone's onboarding, right? Are you doing SMS? Are you doing voice? You know, two, eight, one, whatever. And then use case, and use cases like you're doing a point reminders or are you doing like a, you know, auto responder or are you doing anonymous communications like for a dating app or something, right? So those were, those were the very first questions.

• 27:39Lenny Rachitsky?

  Wow. Okay. So it was like four, four dropdown questions. And that increased conversion. I love these examples where friction and increased conversion, like there's so few of them, like you hear about, this could work and it's rare. And so what did you take away? Like, like what's the pattern you took from this? Like, there's the idea it's good friction, but is there something that you're like, here's what is a sign of this is going to be good friction?

• 28:02Speaker 00

  This still alleviated a problem, alleviated the problem they had where they're coming in and worried that it was going to be difficult. Or that they weren't going to be able to figure it out. They weren't going to be able to get their footing. And I'd say that that's not unique to Twilio. That's, that's something that I think users experience at any front door, at any company, any sign up, beginning of sign up flows, like. Here we go, right? Like buck, like buckling up, especially when you know it's in a work context and you know, there, there might be extra pressure on you to succeed, or you know, for you to make an, an accurate assessment, right? So you're just kind of, you know, I think that psyche of like, okay, am I like in the right place? Is this gonna do what I needed to do? Like, can, can I figure it out? Am I capable?

• 28:43Speaker 00

  Like that's, these are extremely common things for people to feel when, when they're signing up. And so, you know, certainly, you know, that I think can, can carry out to any, any place. So I'd encourage absolutely everybody to be putting those, those kinds of experiences within their early onboarding. Not just, you know, for you selfishly, so you can learn and segment them appropriately, but also so the user can feel more confident as they're getting going like, hey, I'm in the right place. This is gonna do what I needed to do. But I think that the, the carryover there is just that, the psyche of the user, right? And just being so aware that it's not so cookie cutter as what is the problem my market experiences? And what can my product do to help them? There's also this other thing in the room, which is so important to people's success, their ability to succeed with your products

• 29:29Speaker 00

  and your self-serve experiences, which is what is the mentality and the psyche of the person at the various stages in your journey? And if, you know, if you're, if you're not incorporating that or addressing that, you will absolutely miss things or things will fail and you'll be very confused as to why. We had a great, um, experiment that I'm happy to talk about where like same, same concept, a totally different situation, which is later, you know, in, in onboarding, one of the things that, that we tried to do over time to make Tullio less complex was to offer steps like, you know, seasonal onboarding, like welcome, step one, like here's what you want to build. Great. We all know that now. Okay.

• 30:13Speaker 00

  Step one, go to this thing. Step two, go to this thing, three, this thing, four, this thing, five, bam, you're live. Congratulations. Aha, all these things. So, you know, we shipped that, you know, and got that out there. And I was like, yeah, it was like improved conversion, but it wasn't like that great. It's like, man, like we went from like there being like apps to nothing, like choose your own adventure, figure it out, go figure it out. Good luck to this kind of prescriptive thing. And it like wasn't converting as well. And, you know, sort of talked to some users and There wasn't anything particularly obvious that was coming out as to like what the issue was. It was like, oh yeah, like I do step one and we did like mocks with the people like, okay, now I do step two. But there was one thing that like I was hearing that was coming out that's like feels like something.

• 30:57Speaker 00

  And that was that the telephone number, the telecom part, developers, when they were coming into Twilio, it was like things that were familiar to them, APIs, the language they're coding in, code samples, documentation, things that like the bogeyman, right? The things that would like psychologically trip them up. Telecom, phone numbers, things that like, you know, the, these things that just were completely out of the zone of anything that they'd ever worked with before. Especially, you know, early, earlier on in Twilio's journey, but even now, right? Telecom's very different beast for most developers. And guess what was step one? Get a phone number because that's step one. Anytime that anyone's trying to teach them how to use Twilio like one-on-one, they're always going to sit down next to me like,

• 31:41Speaker 00

  okay, like here, we're going to go get a phone number and configure it. And that's what anyone every time will do. However, in a self-serve experience, when you don't have that safe person sitting next to you being like, don't worry, it's gonna be okay. I'm gonna take you through this crazy telecom journey. They're on their own, but that psyche is like, oh my God, telecom, well, I can't do that. That sounds scary. Like getting a phone number and configure it, like whoa, I'm out of my depth, right? And so what did we do to test this out, like test out whether that was the issue? Actually, and it's first one, the MVP, they kicked them out of the, the portal entirely and put them into a docs page where we could kind of manufacture an experience where the first thing they saw was code and they're in the docs, safe place, the language that they're coding in, and then like

• 32:26Speaker 00

  knock in there, I was like, oh, get a phone number, like, let's go configure it, right? Not as like step one, not as like the leading thing, but sort of embedded. And the analogy I have for this is like killing the hot dog. So like if anyone ever has, if you've, if anyone's got a dog or an animal that you have to feed a pill to, it's like, you can't just feed the pill to the animal. It's never going to happen. But if you shove it inside of a hot dog, which like, looks good and that's exciting, then you can get them to consume it more easily. And so this

• 32:53Lenny Rachitsky?

  We do peanut butter. That's our move.

• 32:55Speaker 00

  That, yeah, exactly. Right? Yeah. Hot dog peanut butter, all, all that, right? You, you, you kind of bury it, right? You embed the like scary unpleasant thing. And so that's what you said with the phone number stuff, the telecom stuff. And guess what? Even they were, we couldn't amount of the, the, you know, console and they're going off and, and we had no easy return button. It converted better because we were addressing the big problem that was there at the time, which is their psyche. They were not ready to come in and immediately thrown into a phone number experience that was, you know, letting the bogeyman out to party. And that's not what was going to work. We needed to, you know, put that bogeyman pill in the hot dog. And so, you know, then, you know, once that validated, then we can actually go through that, the business of putting that into the, the onboarding

• 33:36Speaker 00

  flow correctly. And then that can convert even better. But yeah, so again, the psyche of your user is such a critical thing to be thinking about. And if something like very logical isn't converting well, sometimes it means that you're battling against the psyche of the user and you want to take a step back and, you know, think about and learn about where someone is psychologically in your space.

• 33:58Lenny Rachitsky?

  It feels like you had this experiment that was like a complete redesign of the onboarding flow and that didn't work. And then your second attempt was like a different approach. That's like a full onboarding flow. And I'm curious, do you have a take on just when you run experiments? And there's something we dealt with a lot at Airbnb and other places, like you just redesign the whole thing? Or is it better to iteratively work from where you are today and just experiment piece by piece towards some future, much better experience?

• 34:26Speaker 00

  Here's what I would say to this is that from a high level, it's always going to be better to be iterative. And the reason that it's better is that roughly 80% of the times, 80% of the time, our hypotheses and the things that we believe will be true are And this is like amazing. I mean, the there's an amazing article out there. I'm happy to share with you. You can put in the the show notes.

• 34:48Lenny Rachitsky?

  Yeah, absolutely.

• 34:49Speaker 00

  That really takes a scientific approach to proving that out. Companies like Netflix and Microsoft is over and over again. Eighty plus percent. Some companies say ninety percent of things fail. And so when you the the closer you get to something that is, you know, you kind of go bury your your, you know, head in the sand or go into an attic and belt something for six months and ship it. The more likely it is that you are going to ship the eighty percent wrong stuff, right? Whereas the more iterative you are, the more likely it is you're going to catch it sooner. And, you know, failure doesn't have to be a wall. It can be a compass, right? It can be the thing that leads you to the right thing. And so, you know, you always want to, as best you can, get stuff in front of customers

• 35:34Speaker 00

  so that you can get that compass and, you know, get get that compass activated, know where to go. And yeah, and so that means doing, you know, ugly things. I tell my teams all the time. If it's not embarrassing, you've gone too far, right? Gotta be embarrassing. The first thing, that was embarrassing. Kicking people out, you know, onboarding. Spend all this money and like whatever to get them into your sign off loan. And the first thing we do, get out of here, right? I mean, that's that's nuts. But if it hadn't validated, that would have been a very cheap, but very valuable learning. Instead, it was a very powerful, cheap learning in the other direction where, okay, now we know we can invest in it. We know that's the right thing to do.

• 36:12Speaker 00

  So always better to be iterative so that you are letting failure work for you instead of having it be a trap that you fall into.

• 36:21Lenny Rachitsky?

  I know that stat you just shared is like per experiment, you're probably wrong eighty percent of the time. In my experience, launching a whole redesign is is negative a hundred percent of the time. I've been I've grown weary to avoid that as much as possible, which is like, you kind of know that you're taught that as you go into growth and product that you're just like, no, come on, let's make it awesome. Just redesign this whole thing. Especially, you know, your designer is always like, no, let's start again. It's make it amazing, but it always ends up being negative and you're like, okay, well, it's too late now. We gotta launch this thing. We don't have time to start again.

• 36:52Speaker 00

  Well, it's funny. I mean, in the articles and you'll see like it was, um, it was written by somebody from Microsoft who kind of built the presentation platform and did all these cool things. Like as he went into actually trying to apply a scientific method of figuring out like how often people are wrong about their hypotheses and, and you know what they're planning to do. He's like, I wonder if like that applies to us here at like Microsoft. There's even for him, that kind of question of like, and it in and or out, right? Like it's just, you know, and I think it's challenging to, you know, when there's a lot of smart people in the world in this space doing things and it's very difficult to think, gosh, am I really wrong? Like eight plus 90% of the time. But when you think about it, it makes total sense, right? Cause what has to happen for something to be successful?

• 37:34Speaker 00

  You have to understand the problem perfectly. You have to then understand who's having the problem perfectly, the customer. At what time they're having the problem? Then you've got to put the right solution in front of them to solve that problem. Maybe you've got the problem right and all that stuff right, but your solution something off, right? Or maybe your solution is right, but maybe it's just not presented and kind of communicated in the right way. You could have any one of those things off and it's not going to succeed, right? It's not going to have, you know, the, the metric impact you're expecting to have. So in that context, it's almost like incredible. We do succeed 20 to 10% of the time, given everything that has to line up. And so I think it's one of those things where, you know, you really want to go into it, embracing that.

• 38:19Speaker 00

  Okay. This isn't about how smart I am or how good my team is or any of that stuff. It's just, hey, the logic of this is a, you know, challenging to get it right and let's embrace that and let's, um, lean into, let's lean into, to that, that knowledge and make it a part of our strategy instead of fighting against it.

• 38:38Lenny Rachitsky?

  Have you found anything that helps you increase those odds? Or is it just this is the way of the world and you probably can't significantly increase the chances your experiment works out?

• 38:51Speaker 00

  So here's the thing, I mean, I think, I think there's, there's very little that we can do to make that base easier, right? All those things have to be figured out. And so I, I definitely think that you, uh, that, that everybody is going to be in a space where their original, you know, ideas, untested ideas are going to be around that hit rate. However, the way that you go about validating those can be totally different. And you can be very fast about validating those ideas and that's the key, right? And, you know, uh, A B testing is one of the most expensive kinds of ways to validate an experiment, right? It, uh, you know, often requires design and engineering.

• 39:33Speaker 00

  And the PM or growth person or marketing person who's crafting it, right? All these things are investments that take a lot of time, even for simple things. And then you have the time factor, how long does the thing have to run to have, and have an impact? So all of that is extremely expensive. And so I think the key is to just think through, okay, what are the things I can do to quickly validate what, what these ideas are that we and you can do that with painted doors, right? Which is where you test rate the, the concept and the idea before it exists versus the actual experience. You can do mocks, like create, have, you know, create, you know, if you've got a designer, create those mocks for that experience. Put it in front of people, see how they engage with it. That can be so powerful. You could invalidate

• 40:18Speaker 00

  tons of hypotheses at that state. You know, you really want to only things, the only things you want to get to that kind of that deep A B testing environment or ones that have been kind of vetted along the way. And that way you reduce your fail rate, right? Because you're, you're failing faster by using other methods. So, you know, I think that's, I more advocate for that side, like let's fail fast by using those tools rather than figuring out a way that you can rise above where everyone else is operating and figure out ways to solve all that complex stuff better because that's going to be challenging, but you can always get better at experimenting and validating things faster.

• 40:58Lenny Rachitsky?

  This episode is brought to you by Writer. How much hype have you been hearing about generative AI? So much. But how do you take it from a shiny toy to an actual business tool? It helps you do your actual job. Writer is an enterprise-grade generative AI platform built specifically for the needs of businesses and already widely deployed at world-class brands like Uber, Spotify, HubSpot, and Uipath. With Writer, you can break through content bottlenecks across your organization, from marketing webpages to sales emails and product messages, to creating high-quality on-brand content at scale. And unlike other AI applications, Writer's training happens securely on your data and your style and brand guidelines that you provide specific to your organization.

• 41:43Lenny Rachitsky?

  The result is that you get consistent content in your brand voice at scale. Get AI that your people will love. For a limited time, listeners to Lenny's podcast can get 20% off if they go to writer.com/Lenny. That's writer.com/Lenny. Where do you find the best ideas come from for driving meaningful lift? Like, is it gut instinct type and experience bucket? Or is it data telling you like, hey, here's a huge opportunity in your experience?

• 42:14Speaker 00

  I'm a very like data-driven person. Like, I self-describe and think of myself that way. Because, you know, in large part because of that, feel right? You have to be constantly checking yourself and data is a really great way to do that. But I definitely think that I would be described as someone who's like going more by their gut when looking at data and results. Just because of the way that I approach it, which is, I am very comfortable and very common in using qualitative responses and things like that and supplement to quantitative data to make a decision. And that puts less of a burden on the quantitative to really make an assessment of whether something was working or not. You know, one of the things I see.

• 42:53Speaker 00

  Uh, like I, I think sometimes goes against whether folks deal, though I'm seeing kind of things shift a little is that 95% confidence rate. Right? I like, I, I came from, uh, like my background, like in college, I was in a lab running experiments or really publishing to a journal and stuff. And we had to have that 95% confidence rate, had to, because the things that were coming out of the lab and being published were influencing things like how we do education and how we understand how bias works and when it shows up and therefore we can combat it. Things were, were wrong and sending a bunch of baloney, like that can cause some significantly bad things, like false positives, false negatives in that context can be very, um, dangerous, for lack of a word.

• 43:34Speaker 00

  You, you think of other, you know, pharmaceuticals, like there's the 95% confidence rate belongs in some companies and some industries. Because the risk of, of, you know, the impact of a false success is very high. But those of us converting users and trying to upsell folks, like we're very fortunate to not have that level of, of burden on us and we can take advantage of that. Right? And so there are definitely times where I will advocate for, and I will push for, and I will myself use lower, can, lower confidence intervals in 95%. Especially if that like doubles the amount of experiments that you can run in a year, right? And the day, these are all methods that we use to try to validate the hypotheses that we have.

• 44:19Speaker 00

  And if, if you have, like, if you're doing a 95% confidence in a role, you're still accepting a 5%, you know, some amount of, of false success. Do that a little bit more, challenge you to do a little bit more, and then run way more experience. If you look at the, the net of what your team is doing over the course of your, what you're doing over the course of a year, you will be positive.

• 44:42Lenny Rachitsky?

  Wow, that is a big idea, idea of releasing the key value confidence interval for experimentation in data teams. You like any, everyone would be happy to like be excited about this, probably maybe not some data scientists on teams. Do you do that? And like, how do you, is that how you operate on your teams? Just like, we don't need 95% confidence?

• 45:02Speaker 00

  So I'll say this is actually very critically important. You must have this game plan set before you run something. A failure mode that I've seen so many teams fall into is they'll run the experiment or whatever it is, and then they'll, you like. Make the data fit.

• 45:19Lenny Rachitsky?

  Yeah.

• 45:19Speaker 00

  The hypothesis, right?

• 45:21Lenny Rachitsky?

  Yeah.

• 45:21Speaker 00

  Or sometimes alone without a hypothesis, just like this is gonna do better things for our metrics, but not like a core reason as to why or what, you know, what exactly are we testing here? And so this, this is another area where we could absolutely fall into that trap. Let's hovering like an idiot. I think it's good. That, that Laura person said it was cool. So I, I think that that's fine. Like, that will always be a trap, right? So it needs to be very deliberately thought of in advance as a way of like, hey, here's, we're gonna validate this. And always, always, always, if you're going to, you know, accept more risk of a false success or false positive, false negative, you want to then be really thinking about how you're gonna harden your, your validation of hypothesis.

• 46:03Speaker 00

  For example, let's take that one we talked about with, with uh, Twilio where we're kicking people out and we're sending them to that, you know, that the pill and hot dog experiment, right? And we're sending people to that experience to kind of hide the phone number. You know, and, and that case, right? Let's say that we were going to accept a lower confidence interval. I would very much wanna see qualitative feedback to confirm that that hypothesis was true. I wanna be looking at the qualitative data from the ones where people were thrown into the existing flow and once put into the docs, that one of them felt more confident and more like this was really easy to get through and like they, they felt in their territory and things like that. And I'd be wanting to hear from the ones who were in the other one, things like,

• 46:46Speaker 00

  I kinda got stuck on that phone number center, like, ah, like, kinda figure this out, but it just, it feels like it's, I'm out of my depth. Like, I would want to be looking for other things to corroborate um, the hard data that I'm seeing, right? And yes, it opens the door to, you know, whenever, whenever you open the door to more risk acceptance, you are going to have some false successes there. But all of these things together can overall make it more likely that you're shipping more things that are going to positively influence the customer. And again, I can't say it enough. It is a huge risk in of itself to not ship as much as you possibly could in a year. That is a huge risk given that very high fail rate.

• 47:31Speaker 00

  So, you know, I wouldn't, you know, to those data scientists and I've, you know, chatted with a few of my time. What I try to explain is that, that article, that data, that the 80%, like that's hard data, right? About what a detriment it can be if you are, um, you know, if, if you don't run enough experiments, you could literally, if you just run 10 in a year, odds are maybe two around impact two of a course of an entire year. If you kind of take that approach. So, you know, data scientist can understand, hey, if we do this, you know, if we, we do this down, we can like double or triple, whatever it is, are the number of experiments we can run. And overall net, that's going to result in, you know, few, you know, more successes

• 48:12Speaker 00

  that will overall net us to a positive place. Like you can still tell a data story to the data scientist about why you're doing this. Again, this is why, you know, when you ask that question, identify it's a very data driven person. But I think, you know, some of the methods that I use can, uh, you know, sound at the surface level as more like, oh, I'm going by my gut. But again, very data driven is just embracing the reality of some of the hard data that I don't think we all embrace or are even aware of sometimes about that, that fail rate.

• 48:42Lenny Rachitsky?

  This is awesome. This is a big idea. Have you written about this anywhere for folks that maybe want to try this approach at their company? And if not, you should.

• 48:50Speaker 00

  I appreciate that. I, it's funny. I, it's been, it's like all my general life to do is to start writing some of this down. I have three children, one of whom is five months old, the two and four. And so, um, sometimes I'll start to write and then like one of them will like crawl across the keyboard and it's, you know, like this. By, you know, one of them, I mean, all of them, most times, but eventually, yeah, I'd be very happy to, to do that if, if folks would be interested. I, um, I'm always, I'm always happy to do whatever I can to help folks, help empower folks with knowledge to do better. Cause this, none of this is like secret sauce, really. It's just sort of. Learn from experience and it's always better to learn from others' experience than your own is faster.

• 49:33Speaker 00

  So yeah, I would definitely aspire to Lenny is that I think that's the best that I can say, but eventually my kids will get older. I hear this and and maybe I I can do so.

• 49:42Lenny Rachitsky?

  Hopefully. Uh, cool. So maybe if you're watching this YouTube, leave a comment if you want Laura to write in depth about this idea and spread it to your company. Okay, I want to talk about growth, but I have one last question just along the lines of experimentation. Is there any other, just like, I don't know, big lessons or takeaways of just of running experiments that would be interesting to share?

• 50:03Speaker 00

  I think we kind of got into this one like a little bit, but I just really want to exclamation point underline it, which is that notion of kind of making the data rap to fit like a concept, you know? I think a lot of teams are, you know, feel and are under a lot of pressure to show progress, right? And what did you do this month? Where do the metrics move and like, you know, and and it can cause folks to feel like they they have to do that where it's like, oh, gosh, it's like everyone's got the experience where you run an experiment and you're like looking at the data, refresh, refresh, refresh. Oh, my gosh, and actually perform worse, you know, or it's like not the same. And gosh, we got everyone really excited about this thing that we all like worked on really hard, like, oh, my God, what are we, what are we going to say?

• 50:48Speaker 00

  In the QBR, you know, the the monthly report, whatever it is that that's sort of these things, you know, the results come to light. And and to this, I'd say this, that it's incredibly important for growth teams to educate out and for folks outside of growth and leading growth, especially to to understand that the the best way for a growth team to succeed, the the only way really for them to succeed is to embrace the fact that they're there to validate, to to understand what the biggest opportunities are and to go after them. And that is not something that can be done on a weekly timeline, sometimes even a monthly, depending on the space you're in and what's known and unknown.

• 51:33Speaker 00

  And so any growth team that's that's beholden to short timeline wins and improvement is always going to be dangerous. That's a that's an environment that's conducive to vanity metric usage and fabric, you know, massaging the data to fit. And ones that are more successful are ones that are reporting over longer periods of time. Because any growth team given enough time to kind of fail enough times to learn the right thing to do is absolutely going to have to show success, real success, right? Like, not that, okay, we're going to make this data fit, but like, real moving the metrics to success. And so definitely, you know, educating out, if you find yourself in a position where you're you are beholden to that, you know, share that 80% fail rate, like, just math, statistics, data, like, that is not, you know,

• 52:16Speaker 00

  you cannot be successful in an environment, but over time, you can be. And so that's, that's one thing I definitely would would draw on that they spent, you end up spending a decent amount of my pie chart at Twilio and then also at Rapid where I was after that. And I'm sure I'll spend some time at Amplitude as well, just helping folks kind of understand what is the the healthiest ecosystem, most powerful ecosystem for a growth team to operate in. And time and expectations over time is a big part of that.

• 52:46Lenny Rachitsky?

  We say pie chart, it's like the pie chart of your time, like, a big chunk of your time goes to this. That's awesome. I like that. I use pie charts a lot to describe that same idea. Just to be a little more concrete there, what is the timeframe you think is the minimum for a growth team to be thinking across?

• 53:01Speaker 00

  I think it's good, especially for for newer teams, but even teams in general, commit, commit to something that you can kind of do over the course of a year. And, you know, low, medium, high, right? Is always helpful in that space. I mean, you know, a lot of times

• 53:15Lenny Rachitsky?

  What do you mean by that?

• 53:16Speaker 00

  By low, medium, high? Low, medium, high, more like, you know, hey, we've got a few bets that we have or few core hypotheses. And if they take off, that's going to be our high bucket. Like, wow, like, we think these things could be kind of lightning on a bottle here. But they could also be, you know, a bunch of vapor missed. So, but it's totally running, we're not going to know. And if those bear out, though, then, you know, that's, that's our high. And hey, we've got a few things that we think are safer, you know, maybe it was sort of validated a bit in the previous year, what have you. And, you know, these looking at the metrics this amount. So it's helpful to give people, though, that, that, that construct, right? It deviates from it, very hard deviates from this notion of like, here's the single number that we're going to hit, right? Just things that help people kind of understand that space a little bit better and what to expect.

• 53:59Speaker 00

  And because of that, it can be a little bit lumpy. Like, you know, there were some things that you released, I mean, it's truly for the most number of years, it's can be easy to talk about this construct here, but like, you know, there's one thing that we did that generated like tens of millions of dollars of pipeline is really, really powerful and took, you know, some time to kind of navigate and, and, and validate. A lot of times we did like that onboarding stuff that's talking about like, kind of like catching those things like that was, you know, could happen on a little bit of a faster clip, but still, you know, took some time to, to validate and understand. But yeah, over the course of a year, you should generally kind of be able to commit to, to movement, but help people understand the methods there so that they're not coming at you on a weekly basis being like, and what did you do this, these past couple of days?

• 54:43Lenny Rachitsky?

  Okay, I gotta, I gotta follow up on a couple of these things. Uh, what is it that, what was that big change at Twilio that led to tens of millions of dollars?

• 54:50Speaker 00

  This is part of, um, the, the, the course that I teach at Reforg, so, um,

• 54:54Lenny Rachitsky?

  Amazing. Good luck for Reforg. What's the course called?

• 54:57Speaker 00

  Just so Activision Retention. I think it's, um, live is, is my part. I think it's, um, I don't know. We can, I'll get, I'll give you a link so we can

• 55:04Lenny Rachitsky?

  Yeah.

• 55:05Speaker 00

  In the show notes, but, um, yeah, the high, high level version is we, you know, this was, this was kind of deeper into my, my journey at Twilio. This is, uh, you know, fast forward a few years, you know, kind of built up this, this team and some cool things going on. But I was really looking for kind of what's like the next big thing for us to do, like what, what could that be? And I noticed, you know, remember as, you know, that, that question very back in the day when I asked about like the kind of developer versus not developer folks in our

• 55:34Lenny Rachitsky?

  Yeah.

• 55:34Speaker 00

  We saw that that little non-developer little dude ad was kind of growing, right? We were actually the number of people in the ecosystem who were identifying themselves as not a developer, we're, we're in the space. But very interestingly, they were, as we kind of got, got more refined in our understanding of those folks, a lot of them wanted to build with Twilio. Like, you know, there was a hypothesis of like, oh, well, maybe they're lost. I mean, they just want pricing. Like, maybe they set up by mistake. And I was like, no, they're here to build. They want to build, you know, and then they, they kind of struggled through the developer onboarding and some of them would succeed and someone. But anyway, um, the, it was all about identifying like, what did they need to succeed? Like, were they, if we were, made them successful, could it?

• 56:17Speaker 00

  Contribute to dollars. You know, one of the core learnings I'd heard from sales at the time was, hey, it's very challenging for us to get the kind of folks where when a developer's not involved yet to to go from zero to one to get something off the ground. But man, if we can get them to do that, if I can get them to $1 and spend, I can get them to five, and then five and get them 50, like 10,000. Then I can get them 100,000. This whole long journey. Like, hey, Laura, if your team can just get them like off the ground, man, we can do so much. So yeah, the journey is all about, okay, what what were the things that were missing in the experience we were offering? And, you know, ultimately, was they couldn't write code from scratch. That was really difficult. And also, we're going to stand up a server. That was difficult. But we ended up iteratively experimenting a way to validate those hypotheses and what's

• 57:02Speaker 00

  the right way to do this. And yeah, that was great. It's called a quick deploy on Code Exchange. Anyone can go there and deploy an app without having to write code and kind of get in a ha moment there with Twilio.

• 57:16Lenny Rachitsky?

  That is awesome. So basically, it's like a low code Twilio app.

• 57:20Speaker 00

  Yeah, I ended up kind of being like, we had a lot of like, like pet names, like nicknames for, I think, probably the one that that most succinctly describes it as just it ended up being kind of a create your own demo experience, which, you know, made made you talk about the psyche of people, you know, you know, we talked about how developers telecom until they can be intimidating. We'll talk about like this, you know, the the non-developer, like sometimes the buyers or like the people who were, you know, instantly buying buying decisions. Like, for them, it was like, not only was it like telco, but it was like, the developer stuff was inaccessible. But they still wanted to, you know, jump in and they wanted to have that experience. And so this was a way for us to give them momentum, give them calm. Well, geez, if I can get this running, my development team, definitely do it. Right? And so it was

• 58:01Speaker 00

  a very powerful kind of, you know, moment where we could really address the psyche of those users, get them excited about Twilio, and then give sales kind of the ability to give something powerful to those non-engineering buyers and folks they're talking to.

• 58:17Lenny Rachitsky?

  So genius, looking back, seems like an obvious win. I imagine it was not. One of my readers suggested that I start a series of the story of a feature and kind of walk through the discovery, ideation, development, iteration. And this feels like a really interesting example of that. But anyway, I got just a couple more questions. I know we've been going for an hour now. So but I have questions. I don't want to let you go just yet. And they're they're on growth. So so one question is just, you worked at Twilio, which is very product-led growth. You're gonna be, you're now at amplitude, which is more sales driven. And I know it's, you're trying to go more product led. I know Elena talks a lot about this, how every company needs to have product led motions, otherwise they're gonna be disrupted by someone that comes product led.

• 59:01Lenny Rachitsky?

  And I don't know what hires, which, which, what they fall into.

• 59:05Speaker 00

  Yeah, between the, like, SLG and PLG. I mean, yeah, for, for me, it's, um, they're, they're kind of two sides of the same coin, like, you know, growth, product growth and sales. It's all, it's all to me, very thematically, the same stuff. The difference is that with growth, you are selling with your product. And with sales, you're selling with person, like one to one. And, you know, so, so companies need to be, you know, employing both of those forces to optimally convert their audience. You know, we're in a world where people are expecting both. They're expecting to be sold by your product and sold at the enterprise level in large companies by, by a human being that's gonna listen to their specific needs and really break it off for them.

• 59:46Speaker 00

  And, you know, if you only have one, you're gonna, you're gonna miss stuff. So absolutely, I think you wanna, those, those two forces together are working well. And obviously, there's sort of, you know, different stages, you know, things work differently in different spaces. But, you know, I think when it comes to, to amplitude, I think there's a huge, you know, opportunity here. I think the key is, and the, the challenge, you know, for companies that have done like the sales thing and are trying to crack into the PLG thing is, you know, really comes down to how you fundamentally are, are approaching that space. And again, your users and, you know, the, the, where they're at and the psyche of where they're at. I think a lot of companies will say, well, okay, like, hey, we're gonna do this PLG stuff.

• 1:00:30Speaker 00

  Let's take that sales, like enterprise, whatever offering that we have, and let's, you know, chop it up a bit and like, cut access here and like, cut out this feature here and we're gonna like slap this plan out and we're gonna put a price on it and we'll maybe have like hours of debates over whether it's like 10.99 or like 104 dollars or like 75. And eventually we'll like, someone will win the battle and like slap it on and then kind of see what happens. And anyway, the discussion and the focus is a lot around the product. What are we gonna do with this product? How are we gonna crack it open and shift it? And, and then give it to these, you know, people, these users, these, these visitors. And what it's missing, I think, is, and a lot of times it's easy to miss, is that

• 1:01:12Speaker 00

  When we're doing PLG and we're shifting from sales to PLG, we need to reset. We need to recognize that, you know, again, this is, this is sales, sales via the product. What do you do when you're, what does a good sales rep do when they're engaging? They, they understand what the problem is of the person in the space they're talking to. So we need the same thing here. What are the unique problems of people who are coming into our self serve space? And I think, you know, when it comes to a company like Amplitude, you know, you're, uh, you know, a lot of the, the folks that will be kind of looking to address via the, the PLG motion. You know, there's a number of things we want to achieve there, but one of the primary things is to kind of tap into the SMB market and really give them a really, you know, startups and give them a space to land and to grow.

• 1:01:55Speaker 00

  And, you know, you have to think like, what are the challenges and unique problems that they have? Cause we're going to be using our product to kind of settle them. We need to meet them where they're at with the problems that they've got. And I think one of the things that is, that I've, I've observed from being in all these startups and, you know, advising some startups is I, I very rarely, I don't think I've ever come across a startup where they have like the right number of analysts for their needs. In fact, a lot of them like don't have any. And so what that means is that the CEO is, you know, being an analyst to create their dashboards for the board. And you know, the product manager is like being an analyst to figure out like what the heck's going on and creating reports for their product. And it's happening all over the places that people are in their roles and they have to be analysts too.

• 1:02:39Speaker 00

  And I think that that's a problem that especially, you know, younger companies and early stage companies have. And so when they're, you know, correct their psyche, what are they caring about? What are they thinking about when they're signing up for product, analytic product or something? They're looking for something that's going to help them feel reassured that they're going to be able to actually get to the bottom of the right metrics, create the reports that show things the right way. I, you know, what's the best way to show churn? There's gotta be a best way, like so many people are doing it. Guess what? There are some really good ways to do it. And there are some really successful, you know, ways to, to set up dashboards for the board. People have done that too. Like there's a lot of that knowledge that exists and all those templates and frameworks that exist. Benchmarking, are these numbers even good, right?

• 1:03:23Speaker 00

  And so one of the hypotheses that I have. Is that, you know, if we, we take that perspective and we understand, you know, that that is the problem, that there's a number of things that we can do to really change the way that self-serve experience works to help convert people and show them how Amplitude can make them kind of that powerful. But, but the thing that I think sticks across all companies, not just Amplitude, making the shift is just that, that when you're doing this, do not think this is a copy paste, but like chop it for parts thing. Don't start with your product when you're building out your strategy. Start with your customers, your users, your prospects, your people who are going to be coming into your self-serve flow. Make sure you're understanding how their problems differ because they do from the people that you're addressing at the

• 1:04:07Speaker 00

  sales-led side. And then make sure that you're orienting your experience of product around those people.

• 1:04:12Lenny Rachitsky?

  It's interesting that you almost have to kind of start again as a, as a product company, as a product. Yeah. Because you may need to solve completely different problems that eventually lead to the same place. But it's interesting what you're saying that you may end up targeting like analysts or PMs, or I know Amplitude has always focused on PMs, but.

• 1:04:32Speaker 00

  Yeah. And it's right. And there, there's always the nice thing about it is it's in some ways, it does feel like you're starting fresh because you, you do need to kind of start with the customer again and like, what's their problem. But in a lot of ways, you know, you, you can carry over a lot of the same knowledge. I mean, at that point, you kind of know what's working well. Like, Amplitude, for example, does have a ton of knowledge around what some of the best ways artists set up reports. Like, there's a lot of things that they have the momentum going, sort of like, where do you choose that momentum? And how do you put that and curate that in front of users? And make sure that they're getting the right things. There was a ton of momentum already there. It's just a little bit about harnessing it and understanding that, yeah, where are the gaps? Because there are going to be gaps. But, you know, anchoring in a customer problem is, I think, the way that you start any new product, any new thing that you're releasing should always think

• 1:05:16Speaker 00

  about like the customer and the pain point. So no different than when you're doing, you know, PLG for the first time or cracking into it, you need to be thinking again, starting again with the problem, the problems they have, the psyche that they have coming to your, your space so that you can build something that is going to effectively make them feel like, oh, you can solve my problem. You get me. And, and show them how your product's going to do that.

• 1:05:42Lenny Rachitsky?

  Final question, and this is around developers. You worked at Twilio. Obviously, Twilio sold developers. I think Rapid, where you worked right before Amplitud, also sold developers. Selling to developers feels like such a hot space right now. There's so many startups that are just building developer tools, such a huge market. It used to be not. It used to be like, there is not a market in developers. They're not gonna spend money. There's not enough of them. And now it is a big, popular spot. And so I'm curious, what have you learned about building a startup and a product that sells to developers? I imagine a lot of founders building search tools would be really curious.

• 1:06:16Speaker 00

  The first is that developers are just a very different audience from any others. I've seen so many people who have come in strong, done growth really well, or product really well with other audiences, and like, oh, like, I'm gonna take all those learnings off, pivot into serving developers. And as it being a very steep climb, because developers are so different. And let me give you like a couple just fun facts that make them really different. And some of these have kind of some interesting stories. One is developers, like, almost to one, do not look at your marketing website at all. They go straight to your signup flow. So what that means is all that beautiful context that you're setting, and you know, the product and pricing and all that stuff.

• 1:06:59Speaker 00

  Like, very, you know, very often, they're skipping all of it, context-free, and going straight to your signup. And so anything that is not, that, you know, anytime you make an assumption like, oh, well, I probably know this coming into signup, or like, well, we don't need to, that's on the marketing website. Like, none of that's gonna apply to this group of people. They are like, they're there. They're, they're looking to do a, um, I described them, like, the analogy I have for, for this group is, like, they're the Ikea buyers, who, you know, when Ikea baggage comes, they're not opening up the instruction manual and reading it, and then starting to go through. They're in there tearing open the bags and starting to like pull the pieces together and trying to build it, right? They'll come up for context and, and steps and such when, when they get stuck, if they're motivated. So, uh, that's, that's one thing. And then,

• 1:07:44Speaker 00

  you know, another one is just the aversion to talking to sales. And I think everybody can, you know, hearing that, I'm like, oh, yeah, well, I hate sales too. I mean, and I, when I sign up and get a bar of my sales, that's the worst. I totally get that. But I have just, developers are on this whole other level. There was a, there was a Fang company who signed up for Twilio. Built a P O C, launched to production, all this and operating that space for months without engaging once with sales. So I was trying to reach them, right? And I ended up being the one that talked to them like first because they like reached out to support because there was something about their delivery that was was was off there, like missing a feature. And they did not want to talk to sales. They ended up talking to me and this is when I was in product

• 1:08:28Speaker 00

  marketing and I was like my first exposure of like, these people don't want to talk to sales, you know? And then there's another one where like a giant retail company where like the engineering team signed up with their personal email addresses so they wouldn't get bombarded by by sales. It was only like later that we like found out. Anyway, it's so, but but the thing that's the most important, these are fun facts, but but the thing that is, I would say is the most important to leave, thing to leave with with listeners here is what like what makes them so different? Like, like, why? Like, why? What's the deal here? And it stems from their charter and their responsibility. So, you know, if you put ourselves in developer's shoes for a minute, a developer, if a developer is required to use your product, especially if they're

• 1:09:12Speaker 00

  like the primary user, the primary builder, it's really important to recognize that they're responsible for that. If your service goes down, that's their responsibility, not just for themselves, but their team. If the pager wakes up someone because the service they bought from you goes down, that's on them. If, oh, turns out that it doesn't work with with the systems that they said it was, well, that's on them. Like, doesn't integrate with the data the way that, you know, everyone wanted it to? That's on them. Everyone looks to developer when it's not working right and it cannot work right in so many ways. That's their failure. That can, it can cost them their job. It could cost them the trust of their team, cost them their reputation, you know?

• 1:09:53Speaker 00

  And that means that the stakes are very high for them every time that they're adopting something new. So they can't afford to take someone's word for it, especially a sales rep who might be, have some other motivations, right? From their perspective, right? They think they can't afford to trust your content or someone's word. They must do it. They must prove it to themselves. And so that's why they, for developers to be bought in, they need to do something, build something, a proof of concept at the very least, if not kind of like moving further than that. And so that means they're going to be pretty darn deep in their self-serve experience with you. Before they're ready to commit. And so if if you're a company that is providing

• 1:10:36Speaker 00

  that that requires developers to build, right? You must invest in self-serve experiences in order to effectively convert your audience. And you should be thinking of them, something akin, your your self-serve function in growth folks, someone akin to Salesforce. Because your sales developers are not going to accept sales coming in and trying to convert them at that stage.

• 1:11:00Lenny Rachitsky?

  I love that you always come back to the psyche of the user and how, you know, in this case, developers like, here's why they're responsible for this thing. Salespeople are going to convince them this is going to work and it's not. And that's a really interesting tool and that's a really cool takeaway. Is there anything else that we didn't cover before we get to a very exciting lightning round?

• 1:11:20Speaker 00

  Plenty. I think we covered it all, man.

• 1:11:22Lenny Rachitsky?

  Got all my questions and more. So with that, welcome to the very exciting lightning round. I've got six questions for you. Are you ready?

• 1:11:30Speaker 00

  I am so ready.

• 1:11:31Lenny Rachitsky?

  What are two or three books that you recommend most to other people?

• 1:11:35Speaker 00

  I'm a big believer in happiness, not just like, you know, because, you know, being crunchy or, you know, we should all be happy, but also because it helps us do our best work and we're more creative and all of this. So one is Simple Path to Wealth by Dale Collins. I don't ignore the data that money is something that often gets in the way of our happiness. You know, so many smart people that just have not figured out the whole, like, managing their finances thing. And this book will cover all of your basics. It's very easy to read. It's got an audiobook that he does, he marries himself. Simple Past Wealth, Dale Collins. He's fantastic.

• 1:12:12Lenny Rachitsky?

  What's a recent movie? Oh, wait, wait, there's more?

• 1:12:15Speaker 00

  Oh, there's, there's one more.

• 1:12:17Lenny Rachitsky?

  Oh, let's do it.

• 1:12:17Speaker 00

  All with happiness, which is Atomic Habits by James Clear. It is, if you ever want to change something yourself or something's like not quite working for you, this guy will give you a framework to change it. Guaranteed.

• 1:12:29Lenny Rachitsky?

  I really enjoyed that book. That guy's killing it. He was on Tim Ferriss. He had a great interview. Folks that don't want to read it, they can listen to that. There's a lot of cool tips there. Favorite recent movie or TV show?

• 1:12:39Speaker 00

  Unabashedly, the great British baking show. I love that show. I love that show for like all the reasons everyone loves that show. Like heartwarming and like makes, you know, makes you feel good, not lifts you, but also because it is a competitive show. Like they're trying to be the best baker and they're out there helping each other. They're like a big family. Like I've most reality competitive TV shows that I see, all of them are like cutthroat, they're sabotaging. So I'm just endlessly fascinated also by like the psychology of what's happening here. I want somebody to do like a research paper on it. Like we get to the bottom of why they're all like helping each other. It's wonderful though. Wonderful. Wow.

• 1:13:18Lenny Rachitsky?

  Interesting. I always come back to psychology with you.

• 1:13:20Speaker 00

  I know, I know. I feel like I'm just, I'm like really, really sinking deep in there, but it's true though. It's like very interesting to me. So I, and I love that show.

• 1:13:30Lenny Rachitsky?

  What's a favorite interview question that you like to ask in interviews?

• 1:13:34Speaker 00

  I love asking about a ship or release that is not cherry picked by the person you're talking to. You can get it a lot of different ways. The thing is everyone has like a big success story. Everyone does. It really doesn't actually tell you very much to ask someone like, what's a great thing you released? Cause everyone can tell that. Instead, take, take that away. What's most recent ship is a really easy one. Cause like recency time. But you know, there's other things you can do to like, kind of take that out, like just give them specific parameters for, for ship that they've shared or whatever. And, and that will allow you to listen more and learn more about their frameworks versus the, you know, outcomes. Because if you're picking a random ship, odds are it probably wasn't like fantastic. So they're going to want to talk more about

• 1:14:18Speaker 00

  how they approached getting there. And that's what you want to know about to know if they're going to succeed, what their frameworks are.

• 1:14:24Lenny Rachitsky?

  That is cool. I've never heard that one. That is a really clever idea. What are five SaaS products that you use in your day to day work? Can't say amplitude.

• 1:14:33Speaker 00

  I know, right? Um, still learning what which ones we have here. Um, but yeah, I'll share. I'll just share sort of like the ones that I like, like a lot that I've used elsewhere. So one is Hotjar. Um, Hotjar Qualaroo also works. Just anything that allows you to put some quick little thing in front of customers, get that qualitative feedback we talked about. It's a critical, critical supplement to quantitative data to understand what's really causing the change or not causing the change, do you see? So that's important. I will say, uh, Amplitude is a fantastic tool that I have used. I, I would have said that if I weren't, hadn't just joined Amplitude. So, um, I got to use it. I know. I got to use it for the first time at, at, uh, Rapid because I use Amplitude and it was, uh,

• 1:15:14Speaker 00

  was awesome. So again, like asterisk, cause I'm like working there now, but it is, I do actually like it. And you know, Slack, they're boring. Everyone says Slack, but I just have to, have to hand it to them. It makes life so much easier and just not, not their way. And then Builder, which I'll also put an asterisk on that one, but I really want to serve states. A lot of people don't know about it and it's really helpful. I do advise them. So I'm like, you know, in their corner. But this is another one I also say would, would be a powerful one. I think a lot of teams get stuck. They're not able to, um, they're, they're relying too much on their engineers to make changes. Again, we talk about rapid experimentation, getting these out, out, out. And Builder makes it really easy for folks to do that. Also like a headless, you know, CMS, you can like

• 1:16:00Speaker 00

  drag and drop headless CMS. So they do make it easy for non-engineers to make changes. So especially if you're trying to figure out how to, figure out how to get around that 80% Fogu Man that I mentioned, um, this Builder would be a good way. And then, yeah, if you want one more, I'll give you ChatGPT, which is really boring. And you know, everyone's saying that, but I think I'll just say, I don't have any like crazy things to say about it, except that I do think we all need to figure out how we pull that in to our sex. I think people who don't do that are probably gonna lose out, you know, or, or like, you know, smart AI, whatever bots. But that, that would be it for you, Lenny. Bot if you ask me in a few months after I've actually been in Amplitude for a bit, I'm sure I'd give you a different answer.

• 1:16:42Lenny Rachitsky?

  Uh, this is a good time to plug lennybot.com. Someone made a, and I wrote a newsletter or Dan Shipper who created that Bot wrote a newsletter post about how you built this thing. And so you could go ask me questions using the content of my newsletter as answers. And it's very cool. lennybot.com or lennybot.com.

• 1:17:00Speaker 00

  Amazing.

• 1:17:01Lenny Rachitsky?

  There we go.

• 1:17:02Speaker 00

  Well, there you go. I've changed my answer.

• 1:17:05Lenny Rachitsky?

  Yeah, that's all you need. Two more questions. What is something relatively minor you've changed in your product development process? That has had a lot of impact on your team's ability to execute.

• 1:17:17Speaker 00

  Yeah, the be embarrassed thing, like I mentioned earlier, like be embarrassed by the first iteration. If you were not embarrassed, you've gone too far. That really speeds up ships and helps people celebrate the unpolished as opposed to feel embarrassed about it. So embracing that.

• 1:17:35Lenny Rachitsky?

  Awesome. And final question, I know you just started Amplitude, but do you have a favorite pro tip for how to use Amplitude or maybe hidden feature people may not know about?

• 1:17:44Speaker 00

  You tell me day to nap, but I'll say one thing that was super cool, actually, that someone put together on my team at Rapid. Literally before I left, he put together a video of how powerful Amplitude could be when linked up and integrated with other things like in this case, Hotjar and Segment. There was like an Amplitude report that that someone had kind of created. And there was something that was kind of an anomaly happening there, like users were kind of using something in a way we didn't expect. And Amplitude, one of the reports surfaced it, but of course then we want to know like, why is that happening? And so we could take, find out what the event is. And then using Segment, find out what that name was and look at Hotjar and actually go in and get screencasts of

• 1:18:28Speaker 00

  people doing that exact event. And from that, we were able to kind of form some really concrete hypotheses about what actually was causing it. And so, you know, obviously, you know, talking to customers is very powerful. But in this case, just that simple use of kind of connecting and threading those technologies together could really get a good picture of that without needing to engage customers. So kind of the power, I'd say of the tip would be how you can really amplify, where you get of Amplitude when you use it with.

• 1:18:59Lenny Rachitsky?

  Amazing. Laura, we covered a lot of ground, career, experimentation, growth, embarrassment, psychology. Thank you so much for being here. Two final questions. Where can folks find you online if they want to reach out, learn more, maybe send you an appreciation or two? And two, how can listeners be useful to you?

• 1:19:17Speaker 00

  Yeah, find me on LinkedIn. I don't post a lot. Again, I will blame my three children. Eventually, I promise that I will, but I'm pretty good about responding to messages. So, so definitely link, link with me there. And then, yeah, what listeners can do, I think, yeah, I mean, I'm always happy to hear feedback, suggestions, all that. But I'll just say, I also know that it's a little bit crazy out there right now, especially folks working in tech. So I'm also cognizant of what I might be able to do to help all of you. I know there's, there's a few places I advise and, you know, Rapid Hiring, I know of a few folks that are hiring growth, strong growth people and, and product folks. So if you are interested in

• 1:19:59Speaker 00

  learning more about that, don't hesitate to hit me up. I want to make sure that, that I help as many people as I can in that respect because it's, you know, trying times and it's, I'm sure you've heard it and read it, but if you're laid off, this is not about you. It's not your fault. This is crazy world we're in. You, things will get better and I would be very, feel very lucky if I could help even one person land. So feel free to hit me up about that too.

• 1:20:21Lenny Rachitsky?

  Awesome. And maybe if you share some links, we could include links to open roles in the show notes.

• 1:20:26Speaker 00

  Yes, I know there's, there's a few that don't have JDs open yet. They're that hot off the press, but I'm happy to, yeah, service a few things there for sure. Cause I know that makes it easier for people to know.

• 1:20:37Lenny Rachitsky?

  Awesome. We will do our best with the show notes then. Laura, thank you again for being here.

• 1:20:43Speaker 00

  Yeah, thanks so much for having me. This was awesome and so much fun.

• 1:20:47Lenny Rachitsky?

  Bye, everyone.

• 1:20:49Lenny Rachitsky?

  Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.