20VC: Are Foundation Models Becoming Commoditised? Do OpenAI and Anthropic of the World Have a Sustaining Moat? Why Smaller Models May Work Better? Why Incumbents with Data Power Win the AI War with Christian Kleinerman, SVP Product @ Snowflake

Christian Kleinerman is the SVP of Product @ Snowflake. Before Snowflake, Christian spent close to 5 years at Google as a Senior Director of Product Management @ YouTube working on their infrastructure and data systems. Before YouTube, Christian spent over 13 years at Microsoft serving as General Manager of the Data Warehousing product unit where he was responsible for a broad portfolio of products. In Today's Episode with Christian Kleinerman We Discuss: 1. Lessons from the Greats: How did Christian first make his way into the world of product? What are 1-2 of his biggest lessons from working with Satya Nadella and Frank Slootman? What are 1-2 of hs biggest product lessons from Google and Microsoft? 2. Generative AI: Real vs Fake: How does Christian analyze the current generative AI landscape? Which segments will be the fastest to adopt? Which will be the slowest? What aspects of the ecosystems are overblown? Which are under-appreciated? How does Christian respond to many VCs who suggest that many startups are simply wrappers on GPT? 3. Models 101: Why Size is Not Everything! What matters more, the size of the data or the size of the model? Will any of the models used today be used in a year? Does Christian believe Alex @ Nabla is right in saying that "the most successful companies will be those that are able to transition between models the easiest"? How are we seeing the evolution of model size impact the accuracy of result snad size of data required? 4. Incumbent vs startup & Open vs Closed: Who is best positioned to win; startups or incumbents? What are the nuances; which spaces are best served for startups to win vs incumbents? Will open or closed source be the dominant mode? What are the single biggest challenges preventing open from being successful?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We've seen companies that with seven employees have creating models that are comparable for some use cases to what OpenAI or Anthropic do. At the end of the day, it's a data problem and model, they're getting commoditized until the next big innovation comes and you allocate some more value to the model. If anything, model size will influence things like cost and latency. So smaller may be better.

• 0:23Harry Stebbings?

  Welcome back. This is 20 VC with me, Harry Stebbings. And joining us in the hot seat for this deep dive on generative AI is Christian Kleinerman, SVP of Product at Snowflake. Before Snowflake, Christian spent close to five years at Google as a Senior Director of Product Management at YouTube, working on their infrastructure and data systems. Before YouTube, Christian spent over 13 years at Microsoft, serving as General Manager of the Data Warehousing Product Unit, where he was responsible for a broad portfolio of products. But before we dive into the show today, you know all those mind-numbing tedious tasks that seemingly take up half your day. Well, Coda is here with their new AI-powered work assistant that helps you and your team not just finish tasks, but make progress.

• 1:03Harry Stebbings?

  So your product team can bring a feature to market faster by using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions and more. Your sales team can engage more customers by bringing in data from sources like Salesforce and then use Coda AI to suggest action items, meeting agendas or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes and writing mock press releases to visualize taglines. With Coda AI, you can reimagine your to-do list and how you collaborate. So you're not only finishing tasks, but really making progress. If you want a work assistant that lets you get back to work, you can get started with Coda AI today for free.

• 1:46Harry Stebbings?

  Head over to coda.io/20vc. That's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings. But now you can reduce costs up to 30% and actually reward your employees. How do you do this? Well, Nirvan rewards your employees with personal travel credit every time they save their company. Money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game changing all in one travel corporate card and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc.

• 2:31Harry Stebbings?

  And last but by no means least, we need to talk cash. As of the 29th of June, you can get 5.5% yield on your cash with 26 week treasury bills. But buying treasury bills, it's not that easy and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their treasury accounts make it simple to earn a high yield on your cash and it takes 20 seconds. Here's how it works. Sign up at public.com, easily purchase 26 week treasury bills that automatically roll over at maturity for a compounding yield. Plus, there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account. Of course, to receive the full guaranteed yield. Of course, to receive the full guaranteed yield, you do have to hold to maturity.

• 3:16Harry Stebbings?

  But here's the thing, these are T bills, which means your investment has the complete backing of the US government, making it one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4% yield on your cash.

• 3:32Harry Stebbings?

  You have now arrived at your destination.

• 3:36Harry Stebbings?

  Christian, I am so excited for this. I've heard so many good things, so I would love to start with your entrined product. How did you come to be SVP of Product at Snowflake? Let's start there, Christian.

• 3:47Speaker 00

  Thank you for having me here. I'm born and raised in Colombia and South America. I did a startup there. I learned what not to do. I did another startup in the US. I learned what else not to do. So at some point, like I need to learn from, from the guys that really know how to build software. This is 1999. I joined Microsoft, did a long stint in data all the time. SQL Server, appliances, when appliances were the thing to build, and then cloud services. And from then I went over to YouTube at Google, where I was responsible for the infrastructure, including. Data Systems and I think all of that set me up for understanding data, being a data junkie. When the opportunity opened up for Snowflake, I'm like, I could appreciate the technology and that type of company. So I'm like, I'm ready to be there.

• 4:32Harry Stebbings?

  It was the charm and charisma of Frank. I don't blame you. I had the same feeling when he looked into my eyes. I do have to ask, you mentioned that you learned what not to do. If there were one or two things that you really learned what not to do, what would they be, Christian?

• 4:45Speaker 00

  I would say talent being the driver of truly great outcomes. I would say don't ever compromise on talent. Don't ever say, yeah, this person doesn't have the background, but maybe has the right intention, just take a bet. No, I think that nothing substitutes talent. That's a very clear lesson learned. And the other thing that has been very clear is building a scalable business is difficult. One of those startups, we built some scheduling software for airlines. The thesis was you build it once, works, then you just resell it and you can be the next Microsoft. It was not the case. There was a lot of customization. It would turn out into more of a services business. So scalability and building platforms was another lesson learned.

• 5:29Harry Stebbings?

  If that's the lessons learned from respectfully the startups that maybe didn't go to plan, we could say. When you think about Google and Microsoft, they're such symbolic institutions in our environment. What are once two of your biggest takeaways from 13 years of Microsoft? I mean, shit, it's a long time. And then, you know, the four to five years that you had at Google, what are one or two of those big takeaways?

• 5:49Speaker 00

  For Microsoft, I think I got the ease of use and value of simplicity in products. At the time, SQL Server was coming from way behind, competing with market leaders that were Oracle and IBM. And the way in was not to have every single feature and capability that those technologies had. It was just simplify things. If you turn something that is a subset of the capability, but dramatically easier to use, that gets a following. And that was a very, very clear lesson learned. And I've seen it over and over. And by the way, the Snowflake story follows a big part of that journey, which is if you simplify things to a point that it is delightful to use, people adopt. That's from the. Microsoft time, let me think from the YouTube time, maybe the biggest lesson learned.

• 6:34Speaker 00

  Consumer products have many more elements beyond just technical difficulty. There's a lot of timing, what are the trends with consumer behavior. Sure, you need to have a good business model, you need to have good technology, but there are things that you can't control.

• 6:50Harry Stebbings?

  Is simple always better in product?

• 6:52Speaker 00

  I would say yes. You'll say, all things being equal, there are points where you will oversimplify. But I do think that make things as simple as possible and no more.

• 7:02Harry Stebbings?

  If you could call yourself up the night before your first role in product, before Microsoft, before Google, what would you call yourself up and advise yourself, knowing all that you know now?

• 7:12Speaker 00

  The fundamentals of making something work as advertised make a very big difference. So simplicity is part of it, even things like latency make a big difference. So there's nothing like the magical experience of you try to use a product. And then whatever, a technology or a device or an appliance at home. And if it just simply works and does what it's expected, I think that is a magical thing. It's very hard to make happen in all situations, but that would be the big word of advice.

• 7:41Harry Stebbings?

  I do want to focus the show on a joint passion for both of us. It's funny, we discussed a little bit in terms of topics to discuss today. And I was very excited when I got your suggestions. And I want to start from the top, which is kind of ecosystem level, and then we'll move down. But it's generative AI is kind of the biggest buzzword of today. So when we think about gen AI as a buzzword, like top down, how do you analyze the ecosystem today? Is the hype overblown as a starting point?

• 8:07Speaker 00

  I would say that for sure, there is hype, for sure, there's a turmoil on what are you doing on gen AI. And people are rushing to somehow stitch some gen AI to their products. But if you look past that, there is fundamental innovation there, there is fundamental disruption. There is the opportunity to change pretty much every single interaction between humans and computers into something that is friendlier and simpler. So yes, there's fog and there's noise around it, but it will clear up. And at the end of the day, it's going to be a different world.

• 8:42Harry Stebbings?

  Do you think mobile is a good analogy to AI, or do you think it's actually more significant?

• 8:48Speaker 00

  I think it's comparable. I think it's of the scale of the internet. I think it's of the scale of mobile. One of those things where anything and everything will get better.

• 8:56Harry Stebbings?

  Can I ask, what do you think is most exciting? When you look at the different verticals, use cases, opportunities? What do you think is most exciting?

• 9:04Speaker 00

  I would say that this is a real shot in the arm to the creative businesses. That's where we saw the initial use cases and stable diffusion or mid journey and those things. In many ways, because what in other industries, we would call a bug or a hallucination. In the creative world, there are features. There's a, it's a goodness to come up with something that has not been done before. Or it's a mix and match of existing things. So creative industries are probably the sweetest spot. I love what Adobe has been doing with their products and how they've integrated GNI. Massive kudos to them. Then there's the opportunity in every other industry, in every other vertical.

• 9:44Speaker 00

  It's only that requires a little bit of understanding, correctness, understanding, data maturity, things like that. But every customer that I talk to these days, they're looking into doing something. They're all trying to figure out how to get started and how to get there. But I think it applies to all sorts of businesses.

• 9:59Harry Stebbings?

  It's interesting you said there about kind of they're all looking to do something. You know what I'm finding though, because I speak to these enterprises too? They've got no freaking clue how to do it, Christian. They're all like, yeah, so it's exciting. How do I do that, Harry? And I think the biggest businesses in AI will be built in the implementation services businesses. Helping large enterprises implement AI in a meaningful way into their enterprise over the next decade. Do you agree with me in terms of this lack of enterprise education on implementation? And the potential opportunity for implementation services given that?

• 10:30Speaker 00

  I agree, part of it is services, implementation, education. But I also think that the stack and the way to think about this is evolving. I don't think that we have a very clean, this is the three or four components that you use. And this is the type of use case that you apply. Companies are playing with. Hey, the LLM is magical and you send it to questions and magic answers come back. Some others have combined vector retrieval with LLMs. But within that, there's a lot of variability. Which model, which database or which vector database? How much do you prompt versus fine tune? So I would say a lot of that is still being figured out. I think the stack continues to evolve, will continue to mature.

• 11:14Speaker 00

  And in parallel, okay, then company need to get educated on when to use what for what use case.

• 11:19Harry Stebbings?

  I get you. In terms of kind of verticals that are fastest to adopt, I think this is the other thing that we all get a little bit over excited about, which is like, speed of adoption always takes longer than people think. Many, if you can believe it, Christian enterprises in Europe still have no idea what Slack is. And so my question to you is, when we think about the different verticals, who do you think is first to adopt and the fast movers? Who do you think is slow to adopt?

• 11:41Speaker 00

  I think it completely correlates with data maturity. You may have heard some of us at Snowflake talk about there is no AI or gen AI strategy without a data strategy. It's not just a line, it's a truth that we strongly believe in. From that perspective, I would say financial services are at the forefront. Most of the financials have figured out for a long time how to organize data and leverage data for competitive advantages. Go look at the sophistication of hedge funds as an example. Retail and CPG companies have also been very wise at using data. Who would be on the slowest to adopt? Maybe I would point at public sector.

• 12:20Speaker 00

  Not that they're not data savvy, but they have a lot of regulation and constraints that may make it harder for them to adopt.

• 12:27Harry Stebbings?

  How do you think about incentives being a problem? Point here is like, if you think about healthcare, healthcare is largely dictated by, you know, government budgets, by the NHS in the UK or alternative in the US, which is run by politicians. You don't want to replace nurses with AI-driven robots procedures, because then you'd be firing nurses. And then the front page of the newspaper is Biden or Rishi Sunak fires nurses. So the incentive mechanism is misaligned to increasing efficiency in healthcare or in a lot of public services, because it likely leads to job loss. How do you think about the incentive function being the problem for adoption versus implementation speed?

• 13:07Speaker 00

  I think it may be true that once we're ready to wholly replace nurses and other functions, probably there's an incentive problem, but I don't think that we're there yet. I think of most of the use cases right now are around productivity boosts or assistance, co-pilots as opposed to replacement. So think of it as I want to help people be more productive, not wholesale replace. So at this point, I would say the incentive should be working. Maybe a year, a couple of years from now, what you're saying becomes true.

• 13:37Harry Stebbings?

  You said there about there's no gen AI without being incredible data strategy. You said to me before generative AI is democratizing data access. You left me with that as a cliffhanger, Christian. How so? And why do you believe this?

• 13:51Speaker 00

  Yeah, if you think of the role of traditional business intelligence technology, was to sort of bridge the impedance mismatch between the business and business terminology and business users and a technology that frankly is just for a few people. Like writing SQL statements is not something that most people in a company do. And that was a role of that technology to do that translation. But it still requires some mapping and curation and effectively, how do you inform that impedance mismatch? I think gen AI has the opportunity to turbocharge this type of translation where the language is natural language and the answers come in natural language. But along the way, there's traditional database lookups, traditional retrieval.

• 14:36Speaker 00

  And it has the opportunity to democratize data dramatically more than who we are today. Not that BI has not done a really good job, but I think it's going to be now data for everyone.

• 14:45Harry Stebbings?

  When we think about kind of data as a competitive moat, I've had a lot of people on the show say before, Bunny, that data is so freely accessible today. It's no longer this prized possession that incumbents can hail and use to their advantage, given how freely accessible it is. To what extent do you still place a premium on data ownership to leverage versus the freedom to access data?

• 15:06Speaker 00

  Yeah, I would separate there's both public data and private data privately owned by enterprises. But the premise of your question is based on public data. And I would say if things did not change, probably the language models or the models in general would all converge towards. They're all training on the same data and at some point this what mix of data you use, but you'll trend towards the same answer. The interesting trend towards there is the notion of many companies realizing that their data is being used and monetized by these models. So there are companies rethinking and changing their data policies. Are you allowed to crawl me? Are you allowed to train models with this? I think all of this will shift in the next six, 12 months.

• 15:50Speaker 00

  This is all starting already because in the same way that search changed the rules of engagement with public data. Gen AI is doing the same thing and the companies that are behind the data are doing so.

• 16:01Harry Stebbings?

  You mentioned that companies realizing that their data is being used bluntly and they're not being able to monetize it. And they're losing eyeballs because of it on their sites. How do you think about the business model of the future to ensure that they don't lose this revenue despite their loss of data control?

• 16:16Speaker 00

  I don't know if they're going to be able to keep all the existing revenue, but for sure they should be able to capture some amount of revenue. But clearly their data is valuable. Right now it's not being paid for. That says that there's a value gap there. And you see it in some of the license terms that are being floated around. I think it will shift, so then it changes the economics of all this.

• 16:36Harry Stebbings?

  If you were to place a value on data versus model, what would you place? A hundred is your total pie size. Is it eighty, twenty? Is it fifty, fifty? How do you weigh the importance in terms of data and model?

• 16:50Speaker 00

  The vast majority goes to data, ninety plus percent. Certainly, there is a lot of IP and technology that has gone into how do you build these models? But that is becoming less a differentiating aspect and what's becoming bigger is data. If anything, you hear folks doing the math on are we going to run out of public data to improve these models significantly?

• 17:11Harry Stebbings?

  Why are models as little as ten? And why is such value being placed on the likes of OpenAI, Bard, Anthropic? If actually models are ten and not a significant chunk more?

• 17:23Speaker 00

  If you look at it, where things are today or where they were six months ago, maybe models would have taken a bigger edge because they paved the way on how do you model the data? You could make the case that it has been there all along. But I would stick to the 10% or the smaller number because now you see how many foundation models are being created. We've seen companies that with seven employees have creating models that are comparable for some use cases to what OpenAI or Anthropic do. At the end of the day, it's a data problem and model, these are strong words, but I think they're getting commoditized. Until the next big innovation comes and you allocate some more value to the model.

• 18:02Harry Stebbings?

  Cass, what's the next big innovation, do you think?

• 18:04Speaker 00

  What has happened for for language models is coming for computer vision, for images. Democratization, how do you simplify it? Then there's the intersection of those true multimodal languages, which there are, there are many of them out there. But how do you turn it into, it's completely seamless to go and intersect images, speech, text, all of it into a richer human computer interface.

• 18:28Harry Stebbings?

  Do you think it is the existing model incumbents, your OpenAIs, your Anthropics of the world, who chase down those innovations or do you think it's net new platforms?

• 18:38Speaker 00

  I'm pretty sure all of them are chasing that. I don't know for a fact, but I'm willing to venture that they are definitely chasing those innovations. What we're seeing right now has generated such a big spur of creativity. Most new startups getting created, somehow they all want to chase some aspect of Virginia Revolution. So maybe it's a hybrid of both.

• 18:57Harry Stebbings?

  You mentioned that startups chasing it. So many VCs say they're just a wrapper on top of a GPT model. And it's kind of brushed off in that way as just a GPT wrapper. Do you think that's fair? And actually, these models and the providers will create and kill off all the startups with their verticalized use cases? Or do you actually think that VCs are being short sighted with the kind of handoff of it's just a wrapper?

• 19:21Speaker 00

  I think there's many categories. There are some very shallow wrappers on top of GPT-4. I don't place much value on them. I usually ask, hey, how long did it take you to build this? Oftentimes, it's a week or two. I don't think there's a company there. I do think that there are some very deep wrappers on top of GPT-4 that apply domain-specific, legal or other domain, that I think you end up with a true way to bring GPT-4 to a given market or industry. I think those are value. But my comment on startups was there are actually many startups chasing the different aspects of innovation on the core technology. I was at a dinner a few weeks ago and someone was saying, we're trying to blend fine-tuning with prompting.

• 20:06Speaker 00

  Someone else was saying, we're trying to address the limitations of the transformer model. Someone else was trying to look at how to do computer vision better. So, once you start looking at the core tech, not just application of existing models, I think there's innovation everywhere. There will be a number of new startups creating new things and probably the OpenAI and Anthropic are continuing their innovation, which those are effectively research companies.

• 20:29Harry Stebbings?

  They absolutely are in many ways research companies. The thing I want to ask is, often the size of the model is quite hailed. How important do you think model size is today, Christian?

• 20:39Speaker 00

  I think it depends on the use case. For a generic consumer product like ChatGPT, where anything is fair game, the model is supposed to know about every possible topic, it speaks every language, etc. I would say that for those use cases, a model that is large and has lots of cumulative knowledge is very valuable. For specialized use cases, which is what I see more in the enterprise, model matters less, model size. If anything, model size will influence things like cost and latency. So, smaller may be better. And now there's plenty of examples that have been run where a smaller model, fine-tuned for a specific purpose or a specific dataset, produces results better than a generic model.

• 21:24Speaker 00

  So, it's use case dependent.

• 21:25Harry Stebbings?

  Do you think we actually do that? We downsize model size to increase efficiency, to reduce latency, to reduce cost? Or do you think we actually just increase efficiency of compute to be able to ingest and work with larger model sizes more efficiently?

• 21:40Speaker 00

  Both and as I was mentioning startups, one of the companies that I talked to was working entirely on model compression, only to improve latency and cost. So I think there's research and development on both. Sure, the computer is getting faster and cheaper, but smaller models are better. Like if you want to have one or more model calls in the serving path of a consumer product, you have a budget of a few milliseconds to go make things happen and size will matter.

• 22:07Harry Stebbings?

  It's interesting there, kind of thinking about the efficiency, the cost, the latency. I had the Norm from Character on the show and he said, "The biggest problem that we have is the cost of training. It was like $2 million to train one single model." How do we think about the cost of training changing over time? Will we see a democratization there which will allow startups to fine tune and train themselves? Would it continue to be high? Help me understand that.

• 22:30Speaker 00

  For sure, like all of this, the computers is trending down. But the other piece is there's a lot of reinvention of the core training. If you think about how many of the data sets and data sources are common between all the foundation models out there and they're going through the exact same processes or very similar processes. Are there ways to take a common subset and then just fine tune on top of it and avoid the cost? I think we've seen reasonably good results under that path. I do think that both approach wise and compute cost wise, cost of training is going to get lower.

• 23:04Harry Stebbings?

  Cost of training gets lower. There's also another thing which is like the longevity of models. I had one guest email at Stability and he said that no models we use today will be used in a year. Do you think that's true?

• 23:15Speaker 00

  Depends on how you define a model. If you say Llama and Llama 2 are the same model, then yes, all of them will continue to be used for a while. If you pin a specific version, yeah, for sure, there will be new versions of any of those models, Claude, Claude 2, et cetera. But the reality is there will be new models and new refinements on an ongoing basis.

• 23:37Harry Stebbings?

  When we think about kind of those refinements, companies that are able to transition between models obviously have the most flexibility. I think it was Alex at Nabla, another incredible founder we had on the show, he said, "The best companies will be able to transition between models at ease and those that can will win." Do you think that's right in terms of the importance of flexibility's transition between models being a core determinant of success?

• 24:01Speaker 00

  A hundred percent agree. There is so much innovation in the landscape of models that anyone that builds too tightly coupled to a given model is sort of giving up optionality for the future. I just came back from Japan two days ago and a big recommendation I was doing in a large forum was make sure that you build optionality into which model you're leveraging. Even though it's easier to learn the ins and outs of a single model, I think the optionality matters especially. There's way too much change in innovation going on.

• 24:30Harry Stebbings?

  What does it mean to make sure you have optionality? What do you do differently if you're thinking with optionality in mind?

• 24:37Speaker 00

  Oh, you'd wanna be able to plug in different models, but it's not just replace the model and use everything else the same. Certain models respond different to a specific type of prompt. In the same way that you have a hardware abstraction layer or a cloud abstraction layer, you should have a model abstraction layer that knows how to translate a specific request that your application needs to do to a model with its intricacies or specific characteristics.

• 25:01Harry Stebbings?

  In terms of the transition between models, implementation is not that easy. You're not just handing over data with ease. What are the biggest challenges to adoption, do you think, for startups and for companies moving forwards when they think about working with new models and data migration to them?

• 25:18Speaker 00

  There are a lot of issues. Probably the most obvious one is around the correctness and dependability of answers. If you ask folks, one of the key concerns is like, oh, these models make up stuff. So that is the obvious one. But then there are second order issues, maybe less obvious for people, which is security and privacy of data, the data that is used for the question. And then there are even more complex questions on the rights to the answers. A Wall Street company asked me if we feed a number of portfolio trading strategies into a model and it makes a recommendation and the recommendation makes money, could anyone have claims on that answer? And it gets very complicated very quickly.

• 26:00Harry Stebbings?

  How do we solve the security side? I totally get you. You've got customer data, you've got transaction data, but you do want access to the L L Ms. What do enterprises do to get the benefits of access to them without the security issues that come with just sending thousands of customers' data?

• 26:18Speaker 00

  Evolve their platforms where it's easy to bring L L Ms to the data as opposed to send large data volumes to where the L L Ms are. That's where you see Amazon has new services to do this. Microsoft obviously has it with Azure OpenAI. The trend is create private and secure endpoints that can run close to your data. And by implication, not only don't have to move and copy a lot of data, but more important, there are some assurances on what is done with your data.

• 26:48Harry Stebbings?

  So we bring them closer in terms of the endpoints. You mentioned the hedge funds there and like the right to the answer. And whether it's proprietary or whether it's spread across 10 hedge funds that also wanted that information, how do you think that plays out? Does anyone have rights? Do you buy rights to answers?

• 27:04Speaker 00

  I do think that some of these statements about public web data owners changing licenses or potentially licensing the data gives one path forward. There's been interesting developments in the last week or so. Microsoft saying that they'll stand by customers from a copyright perspective. That was meaningful.

• 27:22Harry Stebbings?

  For people listening, why is that meaningful? Microsoft will stand by customers for copyright data.

• 27:27Speaker 00

  Because enterprises are worried about if they incorporate GenAI into any of their products or services in a way that they truly depend on it. And at some point lawsuits start to fly everywhere. They're exposed and because of those concerns, it has held back enterprises. So Microsoft's statement is material in alleviating those concerns that are very real. I talk to people every week and those are real concerns.

• 27:51Harry Stebbings?

  Do you think they will set a precedent now, which for the next year or two, while there is ambiguity in confidence from enterprises, that actually these large providers must provide a backstop to their enterprise customers to allow them to onboard with confidence?

• 28:07Speaker 00

  I think it alleviates a faction of customers in a category of concerns. I don't think it takes care of all of it. There are opinions out there on the, does copyright law even apply to GenAI? And it's a fascinating debate, but I think the standing by folks from a copyright perspective is a step forward.

• 28:25Harry Stebbings?

  Are there any other big regulatory intricacies or challenges that you think not enough people are paying attention to?

• 28:32Speaker 00

  The use case conversation is difficult. Much has been said, oh, don't worry about it. Most of the bad things that you can do with Gen AI are already regulated and illegal. So nothing new, but I do think that there are entire categories of work products that we need to go think through. What does that mean?

• 28:49Harry Stebbings?

  Do you not think there's like an inherent challenge in terms of the opacity of models? Like when we think about the regulatory challenges, we can't continue to have such opacity to get to outcomes. Will there not need to be more transparency in models to show people the pathway to answers?

• 29:06Speaker 00

  So a hundred percent. Two examples, one again from last week, IBM was announcing their own Gen AI models. And they were talking about being fully transparent on the data that went into the models. That's a big step in lineage. The other one is talking about how they trained it. So they're in tune with this concern and they want to surface it in a more open way. The other example is at Snowflake, we acquired this company, Niva, that was doing consumer search based or augmented by L L Ms. And a lot of the principles that they had when they created this product was you want to have citations and quotations and attribution of, even if it's a summary, every snippet, what was the source that it came from, which is a way to control for hallucination.

• 29:50Speaker 00

  So that is the world that we're headed towards, at least in the enterprise. Forget the creative side of things, but at least in the world where you need correct answers, you need to be able to attribute where things came from.

• 30:01Harry Stebbings?

  I totally agree with you in terms of that attribution. I also was one of Shredhouse's first investors with Niva. So that was great to hear. Can I ask you that, you know, obviously, Snowflake is an incumbent. You've mentioned IBM there last week. We've mentioned also OpenAI and Anthropic, which I don't know which camp you put them in, startup or incumbent? Probably incumbent now, given funding amounts. How do you think about the question of where does value accrue in this next decade of AI? Is it the startups building or is it the incumbents who have the distribution?

• 30:30Speaker 00

  I would bias towards incumbents that have the data. Think of all the data that a company like Google has. That's public data or not semi-public, but a lot of user data. Or think all the private enterprise data that companies Snowflake has and that's obviously a self-serving comment. But I do think that data is what powers outcomes. Those are going to be the companies that at least are best positioned. But to make things harder is, I think all of the folks that have data are creating platforms for startups and others to come and leverage that data and deliver innovation. So it's complicated, but I would bias towards the incumbents.

• 31:06Harry Stebbings?

  I have to say, I do agree with you. I also don't think you can underestimate, especially in the short term, the power of distribution. And I think like the co-pilot strategy is a very incumbent focused strategy, which will win in the next two to three years for sure. But I'm a VC, so we pontificate on stuff we don't know about, Christian. But you'll learn more about that, as you know. Can I ask the other question that comes up is open versus closed? We mentioned opacity, we mentioned transparency. How do you think about whether closed systems versus open systems will be more dominant in the next 10 years as the predominant way to build the best models?

• 31:40Speaker 00

  I think there is nuance on what open means. It usually refers to the weights of a model are available. But beyond that, there are questions on are there use case restrictions? For example, Llama 2 is very significant as a development in the industry. But they were very clear, thou shall not use Llama 2 for training other models. Thou shall not use Llama 2 for, I think it's 700 million user use cases. So you can say, is that fully open or not? And sure, because the weights are available, you can say meets the bright line definition. But there are other items that define what is more or less open. I do think that open weights creates a lot of opportunity for research and further innovation.

• 32:21Speaker 00

  Much has been said of how much excitement and interest is happening around the open model. So from that perspective, I think there's a lot of value of those models. But I also think that commercial solutions will largely be hosted cloud services. In which case, in the same way that it is true with open source, I think it will be true about open weights. I have no idea if it matters or not. At the end of the day, who has the best answers or the best service integrated for customers?

• 32:48Harry Stebbings?

  One final one that I am thinking about a lot is just like the speed of adoption. Everyone seems very worried about, you know, impact on jobs, impact on economies that AI brings. Are you concerned by the speed of adoption? I guess, how should businesses prepare for the shockwave of AI?

• 33:04Speaker 00

  All of this is harder than people realize. The demos are awesome. The productization takes longer time. So I would say all of us should go forward as fast as we can because there's going to be natural difficulties that will just throttle us. I also, as I mentioned earlier, I don't think that it's a mass firings happening next week. It's more hopefully incremental productivity boosts happening over the next six, 12, 24 months. And then over time, you decide whether you take those productivity gains and you turn it into fewer employees versus more productively deployed employees.

• 33:40Harry Stebbings?

  You're a product OG, Christian, and you wouldn't say it, so I can. I think that with AI, you see the reducing value of UI. You will see personalization and customization according to each user. And actually, you see this kind of chasm between device and user in a way that wasn't before. Do you agree that UI importance will reduce with the increasing prominence of AI?

• 34:03Speaker 00

  I would say that for certain use cases, that's entirely true. If I have a UI to configure a cluster, like I don't need to know all the options. I just can specify what I want. But I would say there are many use cases where you may want a richer way to interact with data or with whatever is the problem space that I don't think Gen AI would dramatically change it. It does continue to help with personalization. And we've been on the journey of personalization for a long time. So I would say, yes, some use cases, it shifts the value of UI, but many others has the opportunity to continue to enrich them.

• 34:36Harry Stebbings?

  What do you think Snowflake's biggest challenge is in terms of embracing and getting your arms around this next wave of iteration around AI? I know it's a continuation, given the data-first strategy and mindset. But if there was a challenge or an internal, hey, we're going to solve this or how are we going to get around this, what do you think that is?

• 34:54Speaker 00

  Probably perception. I talk to folks on a regular basis and many of them still think of us as data warehousing. We expanded from those origins six years ago. We need to make sure that organizations across the world understand that they can do AI and Gen AI close to the data within Snowflake without having to copy the data to a different platform.

• 35:14Harry Stebbings?

  Why don't they already?

• 35:16Speaker 00

  When you're very successful with some positioning, that comes and bites you later that you are too successful with that positioning. And for many years, we said Snowflake is the data warehouse built for the cloud. And that still keeps getting repeated over and over. And it's a journey. If you think about, most companies end up stuck with their original use case for a long time. We see a little bit of that here.

• 35:36Harry Stebbings?

  You know what's funny? We can edit out stuff, but it really fucks me off, Krishan, because I started this show when I was like 18 and a kid. And that was like 10 years ago. And people still think of me as like a kid podcaster, despite managing like $600 million and not being a kid anymore.

• 35:51Speaker 00

  Yeah, it's exactly the point. Go, go look at Salesforce will be CRM for a long time. Like ServiceNow is the ticket company, so it sticks around for a while.

• 36:00Harry Stebbings?

  Okay, there's one final one before we move into a quick fire. I've so enjoyed this, but when we think about your leadership style as the product leader that you are today, how do you think your style of product leadership has changed over the years?

• 36:11Speaker 00

  I've been more willing to push opinions in a slightly more top-down way as more time has gone by. Earlier on, it was, I need to be a great manager and listen to everyone and accommodate everyone's opinion. And I'm not trying to say that's not important, but in some instances where you want a product to come out with a consistent view, as if it came from a single unified set of principles and individuals. Sometimes you have to go and push for something like, hey, this is what we're doing. I think that confidence comes or evolves over time.

• 36:42Harry Stebbings?

  I had Gustav, who's the CPO of Spotify on the show, and he says, talk is cheap, so we should do more of it. How do you think about the balance between internal debate on product and product ideas, iterations, versus just speed of execution and getting it done?

• 36:58Speaker 00

  I think it depends on the nature of the technology or the nature of the product. At Snowflake, we have both types of technology. The core subsystem that does clustering of data on disk. I think you want to design that thing really, really well, measure 100 times and cut once, because nobody wants their data to get corrupted or their results to be wrong if that thing is not built the right way. But if you want the UI for a query editor and you have Ten different ways and how you could do suggestions for customers. There's no right and wrong. Might as well go quickly, iterate, learn from from users. Both are the right tools and the right approaches, depending on what you're trying to do.

• 37:37Harry Stebbings?

  I want to move into a quick fire round. So I say a short statement, you give me your immediate thoughts. Does that sound okay?

• 37:42Speaker 00

  Sounds good.

• 37:42Harry Stebbings?

  Okay, so what do others not know that you know to be true?

• 37:47Speaker 00

  I don't know if others don't know, but I for sure know that it always comes down to people.

• 37:51Harry Stebbings?

  In terms of hiring, in terms of customers?

• 37:53Speaker 00

  Everything, the the results, relationships, how things are going. Everything is just people.

• 37:59Harry Stebbings?

  Can one succeed as a PM today without being deeply technical?

• 38:03Speaker 00

  For the most part, no. There may be a few types of products that you might get by, but I I like deep technical PMs.

• 38:10Harry Stebbings?

  What's your biggest piece of advice to a PM starting a new role today?

• 38:14Speaker 00

  Learn the product that you're a PM off. Go be a user, go as deep as you can know the technology.

• 38:21Harry Stebbings?

  Do all founders need to be in the valley who are innovating in AI?

• 38:25Speaker 00

  Absolutely not.

• 38:26Harry Stebbings?

  What makes you say that?

• 38:27Speaker 00

  Well, there is amazing talent throughout the world. Even though the valley has something special from the community and the ability to bounce ideas of one another. It's very clear right now, there's a lot of innovation happening elsewhere.

• 38:41Harry Stebbings?

  What's the best product decision you've made and how did you learn from it?

• 38:45Speaker 00

  Focusing Snowflake and ease of use.

• 38:47Harry Stebbings?

  What did you learn from that focus and ease of use?

• 38:50Speaker 00

  It is something a little bit counterintuitive that you may put out a product faster if you just say, I don't know if we should be used this way or that way, so you just surface choices to users. And sometimes it takes longer for us to take the automatic choice and simplify for our customers. So it may be counterintuitive that faster is not necessarily better if simpler is what is being treated off.

• 39:13Harry Stebbings?

  What's the single biggest element that you've most liked to change about the AI community?

• 39:18Speaker 00

  I will double down on they need to know that Snowflake is a great platform for AI.

• 39:21Harry Stebbings?

  Listen, always be selling, baby. Tell me, Quentin Clark said, were you right about Satya in the beginning?

• 39:27Speaker 00

  I was super wrong. When Satya came into the enterprise business, this is before he was CEO. He came in in very short order, made A lot of really difficult decisions, how the org was structured, how contractors were hired, how we thought about the cloud versus the on-premises products. And at the time, my thinking was like, I don't think that he understands all of this. And obviously with the benefit of hindsight, or actually shortly after, it was very clear, he understood it better than all of us and he's proven to be a brilliant leader.

• 39:55Harry Stebbings?

  What do you think makes Satya such a brilliant leader?

• 39:58Speaker 00

  He is very clear what the needed outcome or desired outcome is. So he's a clear thinker would be the attribute. And then he can relentlessly drive towards it and not get encumbered by all the hundred reasons that we all make up for ourselves.

• 40:14Harry Stebbings?

  I love Frank Sutman, okay? I love him because he's no bullshit. He says how it is in a world where no leader says how it is. What have been your biggest lessons from working with Frank?

• 40:25Speaker 00

  He is also such a clear thinker. He has that commonality with Satya. He becomes a clarifying force. Oftentimes, if you're just picturing yourself telling Frank about a problem and a couple of options, just in the formulation, it becomes very obvious that you don't even need to get his opinion because you know where he's going to stand. So that type of clarity that he has simplifies decision-making, accelerates decision-making. He's also an amazing, amazing leader.

• 40:51Harry Stebbings?

  Final one for you, my friend. Now it's 10 years. What role does AI play in society then?

• 40:56Speaker 00

  Productivity boost on pretty much everything we do. Everything is going to be simpler, easier, faster.

• 41:02Harry Stebbings?

  What impact does that have on GDP? Is it like 2%? Is it like 10%?

• 41:06Speaker 00

  I don't know the magnitude, but for sure net positive.

• 41:09Harry Stebbings?

  Should we have a bet? What do you want to bet on? Would you do 10 or 2?

• 41:13Speaker 00

  2.

• 41:13Harry Stebbings?

  Yeah, I would too. Bugger. Well, listen, I'll buy you dinner in London next time you're here to thank you for this anyway.

• 41:18Speaker 00

  Wonderful.

• 41:19Harry Stebbings?

  Arjun, this has been fantastic. You are a star. Thank you so much for joining me today.

• 41:23Speaker 00

  Harry, thank you for having me. It's been fun chatting with you.

• 41:28Harry Stebbings?

  I love it when we can really go deep on a particular topic like we did there. And if you want to see more like that, you can head over to YouTube and search for 20 VC, where you can find all of our interviews in video form. I always love to see you there. But before we leave you today, you know all those mind numbing tedious tasks that seemingly take up half your day? Well, Coda is here with their new AI-powered work assistant that helps you and your team not just finish tasks, but make progress. So your product team can bring a feature to market faster by using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions and more. Your sales team can engage more customers by bringing in data from sources like Salesforce and then use Coda AI to suggest action items, meeting agendas or lead scores.

• 42:14Harry Stebbings?

  And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes and writing mock press releases to visualize taglines. With Coda AI, you can reimagine your to-do list and how you collaborate. So you're not only finishing tasks, but really making progress. If you want a work assistant that lets you get back to work, you can get started with Coda AI today for free. Head over to coda.io/20vc. That's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings. But now you can reduce costs up to 30% and actually reward your employees. How do you do this?

• 42:57Harry Stebbings?

  Well, Navan rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game-changing all-in-one travel corporate card and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc. And last but by no means least, we need to talk cash. As of the 29th of June, you can get 5.5% yield on your cash with 26-week Treasury bills. But buying Treasury bills, it's not that easy and you have to navigate a website that... It looks like it was made before I was born. Enter public.com.

• 43:40Harry Stebbings?

  Their treasury accounts make it simple to earn a high yield on your cash. And it takes 20 seconds. Here's how it works. Sign up at public.com. Easily purchase 26 week treasury bills that automatically roll over at maturity for a compounding yield. Plus, there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account. Of course, to receive the full guaranteed yield. Of course, to receive the full guaranteed yield, you do have to hold to maturity. But here's the thing. These are T bills, which means your investment has the complete backing of the US government, making it one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4% yield on your cash.

• 44:25Harry Stebbings?

  As always, I so appreciate all your support and stay tuned for an incredible episode coming on Monday with the one and only Mark Benioff.