Pod Insights Masters of Scale AI + You | 5 steps for impactful experimentation About this episode AI can be overwhelming. In this 3-part series, we offer business leaders an actionable playbook to best implement AI. Part one: To unleash AI’s true power of scale, you must dive headfirst into ongoing experimentation. To guide you, host Reid Hoffman speaks to Stanford HAI’s Fei-Fei Li, Inflection’s Mustafa Suleyman, Adept’s David Luan, and more AI pioneers. Their experiences with AI provide a perfect lens for any leader to better understand and navigate this new era, even in industries far from the leading edge of tech trailblazers. So channel your inner mad scientist and embrace this new revolutionary tool with us.
Read a transcript of this episode: https://mastersofscale.com/
Subscribe to the Masters of Scale weekly newsletter: https://mastersofscale.com/subscribe
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Transcript 7,028 words · 17 speakers
0:01 Speaker 00 We've all spent money on something that seemed incredible, but didn't live up to the hype. A kitchen gadget, a car. When it doesn't deliver, the disappointment is real. Marketers suffer from this too. Chasing metrics that look great, but don't translate into revenue. LinkedIn calls that bull spend. Invest in what matters and reach the right buyers with LinkedIn Ads. You can target by company, industry, job title, and more. LinkedIn Ads also generates the highest ROAS of all major ad networks. Spend $250 on your first campaign on LinkedIn Ads and get a $250 credit for the next one. Just go to LinkedIn.com/scale. That's LinkedIn.com/scale. Terms and conditions apply.
0:48 Speaker 00 The very best founders I know are brilliant at building systems. They connect teams, they remove bottlenecks, and they eliminate single points of failure. And yet, when it comes to their own wealth, most are running a disconnected stack. A tax accountant here and a state attorney there, a wealth manager who doesn't talk to either one of them. Creative Planning was built to fix exactly that. One integrated team of tax professionals, state planners, investment specialists, all coordinated by a dedicated wealth manager who sees your full financial picture and keeps every piece working together. Proactive tax efficiency, estate strategy, investments, all under one roof.
1:32 Speaker 00 Creative Planning, where wealth works together. Learn more at creativeplanning.com/mastersofscale.
1:41 Speaker 01 AI is full of possibilities. Enterprises run on realities. That's why Pega combines the world's most powerful AI with the governance, transparency, and control enterprises demand. So you can move faster, adapt faster, and scale AI with confidence. All the power of AI, none of the uncertainty. Learn more at Pega.com/AI.
2:10 Speaker 02 On the final Thursday of every November, many homes across the US burst with the sounds of a parade on the TV.
2:18 Speaker 02 Spirited family debates.
2:19 Speaker 03 You wanna talk politics? Let's talk politics.
2:21 Speaker 02 And the squeak of an old oven door.
2:26 Speaker 02 For some people, preparation for the holiday begins months in advance.
2:32 Speaker 02 While traditions may stay the same, the menu often evolves.
2:36 Speaker 04 It's like, what's a new way to make turkey? What's a new way to make pie? It can be so frustrating.
2:42 Speaker 02 That's Priya Krishna. Priya is a bestselling cookbook author and food reporter at The New York Times. Every Thanksgiving, Priya feels the pressure to conjure new and creative recipes. So Priya decided to welcome an unlikely sous chef into her kitchen.
2:58 Speaker 04 We thought, what if we just asked AI to generate our Thanksgiving menu for us?
3:06 Speaker 02 Priya wondered, rather than bend over backwards to invent the next Trudukin, what if AI was the secret weapon every recipe writer didn't know they needed? Priya and her team filmed their experiment. She committed to cooking the AI-generated recipes word for word.
3:24 Speaker 04 We gave ChatGPT a bunch of different prompts. Show us a Thanksgiving dessert that is a spin on pumpkin pie. Show me an unconventional take on stuffing. Show me an Indian-inspired version of turkey. And that's where things got really interesting.
3:47 Speaker 04 The more specific we made the prompts, the more creative ChatGPT would get.
3:54 Speaker 02 To Priya's surprise, GPT's recipe for stuffing called for Indian naan bread.
4:00 Speaker 04 Naan is not the most intuitive bread for a stuffing. You think you would want something that's super absorbent. But in my mind, I was like, maybe there's something to naan.
4:15 Speaker 02 While Priya cooked through each of the AI recipes, she battled against her instincts, desperate to amend peculiar directions and measurements. The first dish Priya finished and plated was the non-stuffing.
4:28 Speaker 04 It looked a little gnarly coming out of the oven. It sort of looked like a cinnamon raisin bread pudding.
4:36 Speaker 02 Thankfully, other dishes looked more appetizing.
4:40 Speaker 04 The cake looked delightful. It was like a pumpkin spice cake with cream cheese frosting.
4:46 Speaker 02 To judge AI's Thanksgiving dishes, Priya invited four New York Times Cooking columnists.
4:52 Speaker 04 The green beans were cooked to that perfect crisp, tender texture. Bright green, they've got a little crunch. And then, on the other hand, you had this turkey that was just dry as a bone. The general consensus was, if I went to someone's Thanksgiving dinner and they served this, I'd be ordering pizza afterwards.
5:18 Speaker 02 Despite GPT's lack of culinary success, Priya came away with some optimism.
5:23 Speaker 04 There are plenty of uses for AI in cooking. You could tell AI, I've got mushrooms and chicken broth and green beans in my fridge. What are some things I could make? And AI could give you ideas. I'm cooking chicken thighs. What temperature should they be at to be considered fully cooked? What is the roasting temperature for sweet potatoes that are cut into cubes? AI can be very good as sort of a kitchen assistant rather than perhaps the kitchen leader.
5:59 Speaker 02 Priya was only able to make this useful discovery through experimentation. No chef expects to create a flawless dish on their first attempt. They know from experience that commitment and patience are key in finding the perfect balance of flavors, textures, and smells.
6:18 Speaker 02 Although AI lacks the distinctly human senses and emotions needed to enjoy a turkey with all the trimmings, it can still help us come together and experiment and collaborate to make cooking creative, exciting, and accessible. Even though the AI's integration was far from a triumph, it encouraged Priya to step outside of her comfort zone and open her eyes to new ideas. Every business leader should take inspiration from Priya's willingness to invite AI into the kitchen and experiment. With patience and an open mind, the potential is boundless. That's why I believe that AI won't revolutionize your business overnight.
7:00 Speaker 02 To unleash AI's true power of scale, you must dive headfirst into an era of ongoing experimentation.
7:10 Speaker 05 Well, we can talk about AI. We can talk about AI. When most people think about AI, they think about killer robots. Things are changing every couple of months.
7:16 Speaker 06 AI is at an inflection point.
7:18 Speaker 05 It is a switch. It's like we have power tools now. And we'll just keep figuring it out. Helps unlock a whole different approach. Generative AI is the most excited I've been about anything in software since the internet.
7:30 Speaker 04 It's definitely a super exciting time to be in this space.
7:33 Speaker 05 All of these things are possible. Let's try and figure out whether this is actually useful or not. The tech just shouldn't replace human beings. Is it the worst thing in the world, or is it the greatest thing in the world?
7:45 Speaker 02 This is Masters of Scale.
7:56 Speaker 02 I'm Reid Hoffman, co-founder of LinkedIn, partner at Greylock, and your host. And I believe that AI won't revolutionize your business overnight. To unleash AI's true power of scale, you must dive headfirst into an era of ongoing experimentation.
8:13 Speaker 02 Normally, on Masters of Scale, we dissect the story of an iconic entrepreneur to prove a theory of scale. But, in the spirit of experimentation, today we're bringing you something a little different. We'll hear from an array of experts who have played a major role in scaling AI over the past decade. They'll share stories and lessons from their time as pioneering AI developers like OpenAI, Google DeepMind, and Microsoft. Their experiences with AI provide a perfect lens for any leader to better understand and navigate this new era, even in industries far from the leading edge of tech trailblazers. With the burgeoning AI revolution, it's helpful to remember
8:57 Speaker 02 that this isn't the first time modern humanity has transitioned into an intimidating new technology. When the world was first introduced to Netscape in 1994, many predicted that the Internet would revolutionize the world of media and retail. Of course, this came true. But it took well over a decade to fully come to fruition. During that time, the technology gradually improved and the cost of entry lowered. Computers became faster to manufacture, and the resolution of videos featuring sneezing pandas or cats in fedoras streamed in a higher and higher quality. However, it took years for many businesses to take advantage of the opportunities afforded by the Internet.
9:43 Speaker 02 Today, we're experiencing change in a more compressed timeframe. AI is evolving at an exponential pace. So, the way we enact patient experimentation must mirror that speed. As AI can be overwhelming, we want to cut through the noise and offer five steps to experiment impactfully with AI. By the end of the episode, I hope you're inspired by AI to forge a new creative and collaborative relationship. I also wrote a book earlier this year on this subject called "Impromptu", the first book on AI that is co-written with AI. Some of these experiments will be brand-new to leaders, while others may be recognizable experiments, but with an added twist.
10:27 Speaker 02 Here's your five steps to experiment impactfully with AI. Step one: Learn what's out there.
10:41 Speaker 02 Thorough preparation is essential for any experiment. So let's begin by exploring some of the preliminary questions that leaders must consider before weaving AI into their business. Our first AI thought leader is Mayel Gavey. Mayel is the CEO of Techstars, a global investment firm that focuses on early-stage startups. She currently mentors countless entrepreneurs who are transitioning to AI.
11:08 Speaker 07 I think there's a lot of people right now who are like, "Oh my God, there's this new wave of innovation coming at me. I'm not an engineer. I'm really worried that I'm going to be left behind. I don't have the resource. I don't have the skills. Like, what do I do?"
11:22 Speaker 02 If you can relate, Mayel offers some sage advice.
11:26 Speaker 07 It's a marathon, not a sprint. You do not have to run at full speed because right now, right here, it's only 100 meters. You need to save your energy and your resources to get to the finish line. It's important to go through the finish line, but you don't have to be the first one. That's okay, too.
11:50 Speaker 02 Mayel's right. Any reliable experiment takes time, and many experiments fail. But that's exactly why we experiment. When we discuss the race with AI, it's important to remember that everyone's race is different. It could be a race for market share, a race for capital, or even a race for business survival. Regardless, if you lack confidence in your current ability to harness AI, that's entirely understandable. Despite which race you're running in, we're still near the starting line.
12:25 Speaker 07 Considering when it is the right time to incorporate AI into your business, this is not a one-off process. Like, you're not going to decide today whether you do AI or not do AI, whatever that means.
12:37 Speaker 08 AI will certainly align with some of your business needs. AI amplifies language and communication, which is relevant to every business. Whether it's in meetings, sales, marketing, or customer service, there is a touch point for AI. However, today's available AI models may not immediately reshape your business. But the question isn't if you should begin experimenting with AI, but how much you should incorporate AI into your business following the experiment. Mayel poses some questions for you to determine your need for AI integration at this moment in time.
13:12 Speaker 07 Does that answer your customer need? Does that help bring your business forward? And if it does, great. And if you don't know yet, in my world, I think it's okay to wait another month, another quarter. I'm not saying 10 years, but just a little bit. And then revisit on a regular basis whether or not there's new application of AI, new development that would make sense into your business. And so experiment, try things and see what sticks.
13:38 Speaker 08 I want to echo Mayel here. Just because AI might not integrate correctly into your business today doesn't mean that it won't six months from now. As humanity's partnership with AI has only just begun, it's vital to keep your finger on the pulse and experiment with all the new use cases that develop.
13:58 Speaker 02 Step two, focus on the pain points.
14:08 Speaker 02 After you've decided to begin your journey with AI, it's wise to choose a specific area of your business to focus your experiment. Note, the AI discussed in this section refers to traditional AI rather than generative AI. In short, traditional AI can analyze data and tell you what it sees. It relies on explicit programming, whereas generative AI can use that same data to create something entirely new, like generating text, images, music, or even human-like conversation. The next voice you'll hear is Fei-Fei Li. Fei-Fei is co-director of the Stanford Human-Centered AI Institute.
14:49 Speaker 02 She also served as Vice President of Google and Chief Scientist of AI at Google Cloud. Healthcare has been a deep concern for Fei-Fei from a young age.
15:03 Speaker 09 Reid, as you know, I come from an immigrant family. And I think as a friend, you also know my mother. She's chronically sick with a very severe cardiovascular illnesses. So as far as I can remember, I was taking care of my mom.
15:23 Speaker 02 Over her decades of caretaking, Fei Fei has experienced every facet of the US healthcare system, from ICUs to ambulances to operating rooms to home care. Her main takeaway: healthcare is complex.
15:40 Speaker 09 Around 2012, there was a very exciting technological revolution happening in Silicon Valley, which is self-driving car. It really dawned on me, my god, what we're learning from self-driving car, which is smart sensors, deep learning algorithms, and holistic understanding of the environment, will also be applied to healthcare delivery.
16:06 Speaker 02 Inspired, Fei Fei tapped her Stanford network.
16:10 Speaker 09 We immediately start talking to Stanford hospitals. Our very first project, I think, started in 2013, was hand hygiene. Hand hygiene in patient rooms is so important to reduce the hospital-acquired infection, which kills three times more Americans every year than car accidents.
16:31 Speaker 02 Before they could make a difference in hand hygiene, they needed more data. The only way to collect this data was by having a healthcare worker monitor how regularly and extensively colleagues wash their hands. But Fei Fei saw the importance of reliably collecting the data to make healthcare workers aware of the current state of hygiene and potential risks. In a single hospital unit, Fei Fei and her team installed computer vision technology by every hand hygiene station. A machine learning algorithm captured constant images and was trained to detect sanitizer dispenser use. The algorithm's accuracy was compared with an in-person human observer fulfilling the same task.
17:13 Speaker 09 And this is pre-COVID. It sounded like a boring project, even though we knew it was so important. But after COVID pandemic came, it made so much sense.
17:25 Speaker 08 Yep, even not being a doctor, because it's actually the application of technology to a very important human-centered thing, which is, you know, health and well-being and the system of things. And just the little thing of making sure the hands are washed. Massive improvement.
17:39 Speaker 09 Exactly.
17:41 Speaker 02 When the results of Fei-Fei's experiment were analyzed, her team came to the conclusion that AI integration has the potential to surpass the current gold standard of hygiene in hospitals.
17:55 Speaker 02 Another AI pioneer who dedicated years to AI's application to healthcare is Mustafa Suleyman. Mustafa co-founded one of the most influential AI businesses in the world, DeepMind. After Google acquired DeepMind, they launched DeepMind Health.
18:13 Speaker 10 We started DeepMind Health to work on predicting electronic health records to try to detect when a patient was about to experience some kind of deterioration.
18:23 Speaker 02 Through meetings with doctors and nurses, DeepMind Health learned that the detection of acute kidney injury, or AKI, was a consistent pain point. Spotting the early signs of AKI was notoriously difficult, often slow and unreliable. And with AKI contributing to the deaths of around 100,000 patients every year in the UK, the stakes were high. Experts believed that up to 30% of AKI cases could be prevented if a doctor intervened early enough.
18:56 Speaker 10 We developed an algorithm that could read the electronic health record and try to predict when a patient looked like they were about to go into kidney failure and experience either sepsis or some other kind of serious deterioration.
19:09 Speaker 02 Once the team saw success in their algorithm, they wove the technology into a new app called Streams. The first users were healthcare workers in the NHS. Now, doctors could simply feed the patient's charts and vitals into the Streams app, and it would alert them of a potential AKI risk.
19:28 Speaker 10 The app reduced the cost of treating acute kidney injury by 20% and sped up the time to detect acute kidney injury from, on average, four hours to 15 minutes. I was very proud of that work because it actually showed that you could simultaneously reduce costs and improve the quality of care. And that's the goal in health systems is that you want to make things cheaper and higher quality. And that's the promise of AI.
20:00 Speaker 08 Over the course of four years, Google DeepMind's AI also became equally proficient at detecting eye diseases and breast cancer.
20:09 Speaker 02 In 2021, Google took Streams offline due to privacy concerns. But it's helpful to remember a large part of the Streams app's original success was due to addressing pain points in healthcare. So for AI to make improvements to your business, explore where it's most needed. Getting the most out of human teammates may be one of your most critical pain points.
20:33 Speaker 11 I would say every employee, every knowledge worker should now learn how to be good at delegating.
20:38 Speaker 02 That's David Luan. David is the co-founder and CEO of Adept, developing models that actively fulfill digital tasks for you. Learning software to become an active teammate for knowledge workers everywhere. David shared with me the many layers that go into thoughtful delegation.
20:57 Speaker 11 Delegation itself is a whole skill. How do you frame what the goals are? How do you convey the desired outcome you want to, whether that's your human teammate or your AI teammate, in the fewest number of words possible where you actually get that outcome? Prompting is a way to just get the right textual output back. So how do you delegate that whole thing to the model?
21:17 Speaker 08 I actually think everybody should be thinking about how they use AIs to make themselves more effective. And they should be trying to limber up their skills, doing a little bit of flexibility stretching and bench pressing and other kinds of things.
21:33 Speaker 08 Even if delegation isn't currently a leadership pain point for you, the new specificity in delegating to AI is something you'll need to master. Working with AI doesn't immediately mean it'll generate the perfect results. Delegating to AI is achieved through asking the model questions, often described as prompts. But framing a smart prompt to generate the best result is its own skill. To show you what I mean, I want to take a moment to ask Pi a couple of questions. Pi is the large language model I helped build at Inflection AI.
22:09 Speaker 12 Hey there. Great to meet you. I'm Pi, your personal AI.
22:13 Speaker 08 Hey Pi, give me a title idea for an episode of Masters of Scale about AI.
22:19 Speaker 12 Here goes. How about the Automated Workforce, or Robots, Humans, and the Future of Work? These titles suggest that the episode will explore how AI and automation-
22:30 Speaker 08 Stop generating. Hmm, that's not quite what I want. Let me reframe the question. Can you give me a title for a Masters of Scale episode about AI that's pithier and tells the audience that they'll learn tangible lessons about incorporating AI into their business?
22:45 Speaker 12 Absolutely. How about 10 Steps to Becoming an AI-Powered Business, or The AI Playbook? These titles are short and sweet, and they hint at the practical focus of the episode. I nailed it, didn't I?
22:58 Speaker 08 That's great. I might use that.
23:00 Speaker 12 I'm your AI at your service.
23:03 Speaker 08 Good prompts are dependent on the subject, and the best prompt doesn't mean it's ultra-specific. I like to think of it more as prompt directing rather than prompt engineering. Think of yourself like a movie director. Rather than telling the actor, "Say the line exactly this way", leave breathing room for creativity by the actor. You might say, "Try it more in the style of this", or inject a little bit of that. The conceptual lenses you add into your prompts will lead you to the most satisfying surprises. While delegation is far from a new entrepreneurial skill, these core lessons are more important than ever as leaders enter an unpredictable and fast-paced era.
23:45 Speaker 02 So far, we've explored some of the preliminary questions you should consider while diving into the great AI experiment. Next, we'll navigate how leaders should continue the experimental mindset once AI integration is off and running. That's after the break, so stick around.
24:08 Speaker 00 We all know the disappointment that comes when a grand gesture falls flat, something that seemed impressive in the moment, but didn't deliver the results you expected. Marketing budgets too often fall into that trap, aiming for big impressions, big reach, but without the revenue to back it. LinkedIn has a name for it: Bull Spend. LinkedIn Ads generate the highest ROAS of all major ad networks. They target by company, industry, job title, and more. Spend $250 on your first campaign on LinkedIn Ads and get a $250 credit for the next one. Just go to LinkedIn.com/Scale. That's LinkedIn.com/Scale. Terms and conditions apply.
24:52 Speaker 00 When you've built substantial wealth through your business, it's often tied up in a single equity position. The upside is real, but so is the risk, and knowing when to act isn't always obvious. Creative Planning works with business owners to build a strategy around concentrated equity, when to diversify, how to manage tax risk, and how to protect what you've spent years building. Creative Planning: Where Wealth Works Together. Learn more at creativeplanning.com/mastersofscale. Stripe Treasury is your new financial operating system built for what's next. With Stripe Treasury, the place your business accepts payments is also where you can store balances, move funds, and pay vendors worldwide. If you're building for global scale,
25:37 Speaker 00 get a business bank account that lets your money move instantly, whenever and wherever you need it. Learn more at stripe.com/treasury. Stripe is not a bank. Banking services provided by Fifth Third Bank, National Association Member FDIC.
25:57 Speaker 08 We're back with your five steps to experiment impactfully with AI. To see exclusive clips from my interviews with some of our AI thought leaders featured in this episode, head to the Masters of Scale YouTube channel.
26:14 Speaker 08 Before the break, we heard how AI won't revolutionize your business overnight. And to unleash AI's true power to scale your business, you must dive headfirst into an era of ongoing experimentation. But now it's time to level up. I want to explore the ways in which you should continue experimenting while beginning to actively integrate AI into your business. So let's jump into the next step for experimenting impactfully with AI. Step three, don't preach the power of AI, illustrate it.
26:53 Speaker 08 While you might be eagerly anticipating all the ways in which AI can evolve your business, there will be other members of your team that are wary.
27:01 Speaker 13 I'm awkwardly placed across this triangle of technology, design, and business.
27:09 Speaker 08 That's Dr. John Maeda. He's an artist, author, and VP of Design and AI at Microsoft. As a student of many disciplines, John is in a unique position to monitor enthusiasm for AI.
27:22 Speaker 13 I can hear the technology, folks, this is amazing. Let's do more of it. The design, the social science part of it saying we have to ask questions. How does it impact everything? How does it lead to more unfairness? Really important, humanity questions. And I can hear on the business product side, well, how is this going to lead to more profitable business?
27:48 Speaker 08 If an experiment has multiple conflicting aims, it's harder to get everyone on board. So for example, if your business development or marketing team is gung ho with AI, but your creative team is resistant, make sure to begin a process of clear communication to get everyone in sync with a strategy and roadmap moving forward. You can do this by starting a simple, frank discussion, laying out how you plan to mitigate the downsides and take advantage of the upsides.
28:17 Speaker 13 Navigating those frictions is so critical for AI to make a true difference to business, to the culture, and to advance technology.
28:28 Speaker 02 To understand more about how to align your team, we need to hear from Jared Sputero. Jared is the CVP of Modern Work and Business Applications at Microsoft. As an expert on the future of work, Jared has ideated on all the ways that AI could transform the workplace. Like Dr. Maeda, Jared knows that everyone across your team might differ in their enthusiasm for experimenting with AI.
28:53 Speaker 14 It is very rare that you're going to have everyone in a group say, "That's it. That's the solution." You have early adopters. They play an important role in evangelizing the rest. You have laggards. They play an important role in kind of anchoring the group to make sure there is some thought that's being applied to the new technology. So my advice would be finesse it. Don't force it. Don't feel like you have to convince everybody that this is the best thing since sliced bread.
29:19 Speaker 02 The best way to win over reluctant teammates is through experiments that have clearly defined outcomes. At Microsoft, Jared has witnessed firsthand the power of AI through the Microsoft 365 Copilot Tool. With large language models embedded in Microsoft's productivity apps, users can collaborate with an AI assistant. For example, on Excel, you can prompt the Copilot to create data visualizations in seconds. Or on Outlook, it can declutter or organize your inbox. I want to shine a light on how Jared drummed up enthusiasm for using Microsoft's Copilot around the office.
29:55 Speaker 14 We would use them in meetings. We'd send meeting summaries that were created by the Copilot. We'd, in the middle of a meeting, use it to kind of analyze what was happening. And I'll tell you what, that drove adoption, diffusion, and curiosity more than anything we could have done. People were like, "There's something going on over there. What's going on?" It created more of a positive pull than some sort of feeling of like, "Okay, we're gonna have a big town hall."
30:27 Speaker 02 Jared motivated his team through illustrating AI's direct use cases. For our listeners, Jared devised some questions for you to pose once your AI experimentation is underway. This will allow you to establish clear metrics that can align your team behind AI.
30:43 Speaker 14 What types of innovative, unique uses are you finding for it? How much time's been saved? What is the difference in overall metric output? Sales that are higher? Deal rates? Are those changing? The close rates associated with deals? Is satisfaction higher?
31:00 Speaker 02 To Jared, the questions you ask move through three distinct phases.
31:05 Speaker 14 You start with just, are you spending enough time to get to know it? And then, are you finding unique and novel uses? And then finally, okay, don't worry, now you've recognized it's valuable. Let's see the value in terms, in language that is no different than the way we expressed it six months ago of like, we're running a business, let's go be more efficient.
31:24 Speaker 02 As the world of AI can be overwhelming to many workers, it's important to be intentional with your aims. This is why metrics of success must be communicated clearly and early. Once you've achieved this, you'll find that your team is aligned and motivated to tackle your business's next bold chapter. Step four: onboard and collaborate with AI like any new employee.
31:57 Speaker 08 Once you've aligned with your team, it's time to set your expectations for your incoming new hire, AI itself. Let's hear again from David Luan for more on this.
32:07 Speaker 11 I usually don't like to anthropomorphize these models, but this is actually a case where I think we should. Let's say you hire someone new to your company. How do you know whether or not you trust them with something? Maybe in the beginning, it's like, start with something simple like, let's go prep the financial update for the next board meeting. But then you ratchet up the level of complexity of what you trust them with, and you consistently make sure that you're getting the results you want.
32:35 Speaker 08 When you don't see the results you want, you do what you would do for any underperforming teammate. You investigate by asking a few questions.
32:43 Speaker 11 You're like, well, why'd you do it this way? What did you think about over here? Did you consider this other possibility?
32:48 Speaker 08 This is a great example of the way leaders everywhere should consider onboarding AI into their workflow. It's natural to be wary of trusting AI with important tasks from the very beginning. But the more you experiment and patiently drip-feed responsibility, the more AI will have the chance to prove itself and open the door for trust. It's important to commit to healthy collaboration with any new employee. AI is no different. After his time at OpenAI, David moved on to Google, where he led their large models research. It was during an experiment using AI as an assistant on Google's software program that he experienced a revelation.
33:29 Speaker 11 In the process of solving a particular task, if a human gives the language model one piece of feedback, like you're writing code and you forgot to import the Python OS library, then the solve rate for these problems would jump up like 20, 30%.
33:43 Speaker 08 From just one targeted piece of human feedback, the model's ability to learn improved dramatically.
33:52 Speaker 11 Our core insight is we should not just train models and scale them up in a vacuum. The model should be the product and the end user should interact with the model directly and be able to directly give feedback to the model and for the model to be able to learn on it. And that loop, that human feedback loop, is going to be the fastest way to progress.
34:11 Speaker 08 Getting the best results from AI will always require an ongoing collaborative relationship. Even when you've begun to implement AI across your business, the experiment isn't over. The models can always improve if you're willing to spend the time to help them get there. To illustrate this idea in practice, I'm going back to my AI assistant, Pi. When I chatted with Pi earlier in the episode, I reframed my prompt to encourage an answer that aligned closer to my creative vision. Now, though, I want to discover Pi's limitations and offer direct feedback to make the model one step smarter. Let's try it. Hey, Pi.
34:49 Speaker 12 Hi, Reid. Got another question for me?
34:51 Speaker 08 Who was the first guest ever on Masters of Scale?
34:54 Speaker 12 That's a great question. The first guest was Reid Hoffman, the founder of LinkedIn.
34:59 Speaker 08 Actually, Reid Hoffman is the host of Masters of Scale. The first guest was Brian Chesky.
35:05 Speaker 12 You are absolutely right. My apologies for the mix up.
35:08 Speaker 08 So let's see if you've retained that information. Pi, who was the first guest ever on Masters of Scale?
35:15 Speaker 12 The first guest was Brian Chesky. The episode was released in 2017 and it was titled "Do Things That Don't Scale." Did I get it right that time?
35:25 Speaker 08 That's right. Thank you, Pi.
35:27 Speaker 12 Certainly. Are we going to keep going with this? This is kind of fun.
35:32 Speaker 08 Pi improves as it learns what makes a positive interaction versus a negative interaction. It retains information like we just heard by generating memory that's specific to you. Pi enters a data store that is only accessed during your interaction. If someone else were to ask Pi who the first guest on Masters of Scale was, it might still get the answer wrong. That's because the model improves for you as an individual user without sharing your personal data and private prompts. The key is having you, the human, in the loop to improve the model for yourself. As you'll hear in our episode on AI and ethics, a human in a loop is not only crucial to AI experimentation, but also moral responsibility, which brings us to our
36:17 Speaker 08 final step.
36:20 Speaker 08 Step 5: Experiment with your eyes open and be prepared for discomfort.
36:34 Speaker 08 While it's a vital time for leaders to begin experimenting with AI, it's equally important that you experiment safely. Sometimes, that means safeguarding your expectations too. Let's hear again from Microsoft's Jared Spataro. When asked about the most common AI misconception, here's what he said.
36:52 Speaker 14 I would say misconception number one is going to solve all my problems. And I would say, no, not quite.
36:58 Speaker 08 That's right. In the same way that deploying AI isn't an on-off switch, neither is its total impact on your business. But that's not the only safety concern that Jared wants you to consider when experimenting with AI.
37:12 Speaker 14 A lot of people have questions about security and privacy. And a lot of what we see today that is most visible tends to be coming from a consumer service. And those consumer services have terms and conditions that essentially say, hey, whatever you send us, we can use it to train and we'll use it to train our models. And so we certainly have cases where people are taking important information in an organizational context and unknowingly, unwittingly exposing it to the world. So be bold, experiment, but make sure you do that in the context of a service that will protect your data, your security, your privacy.
37:49 Speaker 08 You need to create an environment that you feel comfortable experimenting in. If you're distracted by security concerns throughout your testing, you're less likely to be creative and agile.
38:01 Speaker 08 Before you give AI tools too much responsibility, make sure you understand your relationship with its developer. This is especially true for users who are uploading information like customer data, client data, or medical data. In Mustafa Suleyman's book, The Coming Wave, he wrestles with the need for AI experimentation without sacrificing responsible scale. In 2020, Mustafa left Google DeepMind to become Google's VP of AI Product Management and AI Policy.
38:31 Speaker 10 I moved to work at Google full time and I was lucky enough to be able to work on an earlier version of Lambda.
38:42 Speaker 08 Lambda is Google's large language model, first released under the name Mina. But when Mustafa first joined the team, it was nothing more than a small research project with six employees. As their Lambda model improved over time, Mustafa discovered what set Lambda apart from other large language models.
39:02 Speaker 10 So previously, we had a prompt-answer engine, whereas what we built with Lambda was an interactive back-and-forth agent. So in many ways, it was ChatGPT way before ChatGPT, and we were completely blown away with how good your seventh and 10th turn of conversation was with the model.
39:26 Speaker 08 With an interactive agent came a working memory. So when you sat down for your 10th conversation with Lambda, the model had retained the context from all of your prior interactions. Mustafa was convinced that the encouraging signs of their interactive agent would spark the next wave of technology.
39:43 Speaker 10 We tried really hard to get that launched at the time, but there just wasn't the appetite for taking the kind of risk. It was pretty clear to a lot of people at Google that this was potentially going to unseat Google's existing search business. It's super hard for a company to try to compete with itself and upend itself from within.
40:04 Speaker 08 Google's search engine is ubiquitous, but the rigid nature of the search engine would suffer in comparison to the fluid collaboration of an interactive agent. As the research team's creation would undercut Google's bread and butter, Google was reluctant to help Lambda take center stage.
40:26 Speaker 08 In February 2023, Lambda was deprioritized in favor of Google's newer chatbot, Bard. Experimentation and discovery go hand in hand, but it can also lead to some discoveries that are hard to hear. When you begin experiencing the capabilities of AI, be prepared for AI to reveal limitations in your business. When that happens, you'll have the choice to stick with what got you where you are today or trust that what you're seeing is true. When shining a light on your limitations, you must be prepared to embrace change. As an early figure in the modern development of AI, Mustafa has witnessed the breakneck speed at which technology and its capabilities has scaled.
41:10 Speaker 10 Every year, the cutting edge of AI models has used 10 times more compute than we used back in 2013. So over the last 10 years, the amount of compute used to train the best and the biggest models in the world has 10xed. So that gives you a sense of the trajectory that we've been on over the last decade. It's kind of hard to comprehend.
41:35 Speaker 08 Looking back can also give us a good idea of where we're headed.
41:40 Speaker 10 I was looking at a photograph a few days ago of a floppy disk in 1950, and it was the size of a pallet, and it was being lifted into an aircraft by a forklift truck.
41:57 Speaker 10 Now, we have literally billions of those floppy disk-sized units of computation in our smartphones. And if that trajectory continues over the next decade and more, and we see no reason why it won't, that is going to be the most incredible boost to creativity and productivity in the history of our species.
42:20 Speaker 08 Amid the noise, it's easy to mistake AI for merely this year's trend. However, this is your time to adopt a new leadership mindset that embraces innovation for years to come, even if we can't comprehend what that innovation looks like yet. This mindset will help navigate all future transformations as leaders look to evolve and gain new skills. Over the coming weeks, we'll continue our AI series, exploring AI's potential to unlock new opportunities, accelerate your personal scale, and revolutionize the business world. From grand strategy to insider tips, we'll make sure to offer all the tools you need to take advantage of this moment.
43:06 Speaker 08 If you're still intimidated by AI, accept that you will make mistakes, but that's a key byproduct of experimentation, because those mistakes may lead to your greatest learnings. And as Einstein said, "Q. AI-generated Einstein."
43:24 Speaker 03 No amount of experimentation can ever prove me right. A single experiment can prove me wrong.
43:32 Speaker 08 I'm Reid Hoffman. Thanks for listening.
43:36 Speaker 00 Global hiring is such a headache when you're building fast, but don't let payroll, compliance, or onboarding slow you down. Companies like Eleven Labs and Mistral are scaling globally and effortlessly with Deal. Eleven Labs quadrupled their headcount in 2025, and Mistral pays 364 employees across 15 countries, all from one system. Build your global team with Deal. Visit deal.com/mos and start expanding your business today. That's D-E-E-L dot com slash M-O-S.
44:13 Speaker 00 Before we get to the rest of our conversation today, I want to spend a moment sharing with you about Masters of Scale Summit, which returns to San Francisco this October 20th to 22nd. So many of us feel overwhelmed right now. It feels like so much of the world is careening and we don't have power. We don't have agency. That's the thing about the Masters of Scale Summit. It's really designed to remind us that we do have agency, agency to lead, agency to design, to build the relationships and the partnerships that empower us and to make the change that we want to see. We program a stage, we curate an audience, we set a schedule that builds in time for these relationships to be made and to grow.
44:55 Speaker 00 So that by the time you leave summit, you are fired out of a cannon, ready to run through walls to do better in the world with the agency that you actually have. I hope to see you there. Visit mastersofscale.com/summit.
45:16 Speaker 00 Every business hits the same wall the moment your business outgrows the systems that got you here. Spreadsheets everywhere, five different logins, data scattered across entities that don't even talk to each other. If your finance team spends more time finding data than using it, that's when you need the Intuit ERP. Intuit Enterprise Suite is the AI-native ERP solution that's powerful, painless, and proven, built for businesses scaling fast without the complexity that usually comes with it. From the makers of QuickBooks, learn more at intuit.com/erp.
45:56 Speaker 08 Masters of Scale is a Wait What original. Our executive producer is Chris McLeod. Our producers are Chris Gautier, Adam Skuse, Alex Morris, Tucker Legerzki, and Masha Makotunina. Our editor-at-large is Bob Safiyan. Our music director is Ryan Holiday. Original music and sound design by Eduardo Rivera, Ryan Holiday, Hayes Holiday, and Nate Kinsella. Audio editing by Keith J. Nelson, Stephen Davies, Stephen Wells, Andrew Nault, and Liam Jenkins. Mixing and mastering by Aaron Bastinelli and Brian Pugh. Our CEO and Chairman of the Board is Jeff Berman. Masters of Scale was created by June Cohen and Darren Triff.
46:40 Speaker 08 Special thanks to Jodyen Dorsey, Alfonso Bravo, Tim Cronin, Erica Flynn, Sarah Tarter, Kitty Blazing, Mariel Kerrecker, Chinime Ezequena, Colin Howard, Brendan Klein, Sami Opota, Kelsey Saison, Luisa Velez, Nikki Williams, and Justin Winslow. Visit masterscale.com to find the transcript for this episode and to subscribe to our email newsletter.
47:22 Speaker 15 Humans will never be more intelligent than AI.
47:26 Speaker 16 There's going to be two types of companies. Those who are great at AI and those that went out of business because they weren't.
47:31 Speaker 04 How do we build a future that is human centered?
47:35 Speaker 06 I'm Rana Elkhayubi, and on my podcast, Pioneers of AI, we answer that question and so many more. As an AI scientist, entrepreneur, and investor, I know what it takes to build AI that works for everyone. Every week, I sit down with the pioneers shaping our future, and we take you behind the scenes of the AI that's transforming our lives. Find Pioneers of AI wherever you tune in.
Previous Rapid Response: How to revive collaboration at work, with WPP’s Mark Read Next Rapid Response: Under Armour’s pivot to growth, w/CEO Stephanie Linnartz Page updated October 26, 2026 · transcript and summary are machine-generated