Ep 29: Salesforce AI CEO Clara Shih on Future of Slack, How Gucci Uses AI and Working with Marc Benioff

There’s an ongoing debate about where the most value will accrue in AI between incumbents and startups. Of the incumbents, few have shipped product faster than SalesforceAI. Today on Unsupervised Learning we had on Clara Shih, CEO of SalesforceAI and one of Time Magazine’s 100 Most Influential People in AI.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast. I'm Jacob Afaron. I'm joined by my co-host, Pat Chase, and today we had an awesome episode with Clara Shi, the CEO of Salesforce AI. Salesforce is one of the incumbents that probably sits on some of the most interesting data and has moved fastest of folks out there. We talked about some really interesting ways that AI is showing up in the Salesforce products today. We talked about how companies like Gucci ended up leveraging AI. And then we also hit on all the interesting kind of barriers to enterprise adoption today and what actually is required to get enterprises using AI. I think folks are really going to enjoy this. Definitely make sure you stick around for the debrief after. Thanks so much for coming on. Really appreciate you taking the time.

• 0:38Speaker 01

  Thanks. It's great to be here. Cool space.

• 0:40Jacob Effron?

  Yeah, you know, it's fun. I imagine Salesforce has a much more glamorous podcast studio, so I appreciate you coming in and playing an away game, you know, in our podcast studio.

• 0:48Speaker 01

  You know what? Anytime.

• 0:50Jacob Effron?

  You know, I guess to start, like, I'm wondering, like, when, you know, this next generation of people comes into the workforce like 10 years from now, and they look back on, like, how we work today. Anything that comes to mind is something that they'll just think is, like, so ridiculous that we used to do things in, like, X way. Because obviously, we're all thinking about, like, how AI changes things. And I think it's easy to see in this, like, six, 12-month, the things that work here and now. But I'm wondering, as you think about, like, way in the future, the kind of stuff that we're, you know, that is just involved in being in the workforce today, that is going to feel like it was from a different, completely different era?

• 1:20Speaker 01

  10 years is a long time, and I bet it's going to be everything. I mean, it's like us looking at how people worked in the early 90s, and they filed papers in filing cabinets. A lot of people didn't even type. They would dictate to someone. I'm sure 10 years from now, which is an eternity, people are going to look back and think that pretty much about of everything we do.

• 1:39Jacob Effron?

  Yeah, I mean, I'm curious of the stuff you've seen today, like, the early examples of how folks are using some of the Salesforce AI capabilities. Like, anything that's jumped out to you is, like, that's a really cool, specific example?

• 1:49Speaker 01

  Oh, there's so many. So last summer, we shipped our first set of generative AI applications under the Einstein GPT brand. So we have Service GPT for customer service, Sales GPT for Sales Cloud sales users, et cetera. And I think just the one that's really blown everyone away, especially me, has been service reply recommendations and case summaries.

• 2:10Speaker 01

  Because those are two very time-consuming tasks that no customer service person enjoys doing. And on the other side, customers hate having to wait for an answer. And so just being able to do, you know, basic, you know, rag, basic type of query, everything grounded in the customer's data that's in Salesforce, and for it to work out of the box, I think it's felt really magical for a lot of customers.

• 2:32Speaker 02

  I feel like customer support is such an amazing use case of LLMs where it's like, you can have humans do the thing that they're good at and AI do the thing that they're good at. And it's cool that you guys ship that so quickly.

• 2:45Speaker 01

  I couldn't agree more. And the amazing thing for us was that we went into this thinking that we would use AI for customer service. We'd bring AI into sales, we'd bring AI into marketing. But what we've learned is that the AI can blur the lines. And so the example that I was sharing earlier around ServiceGPT, we rolled that out to Gucci, which is one of our first customers on the product. And their service team, which is split between Florence, Italy, and New Jersey, they did see a reduction in average handle time. But then, instead of hanging up with the customer or ending the chat, they've actually been able to up-level and get augmented by the AI in real time to become salespeople and

• 3:24Speaker 01

  become brand storytellers. And it's just completely changed the traditional role definition of what it means to be a service representative.

• 3:33Speaker 02

  And so that's like, the AI could recommend something, you know, they may, a customer may have come for an issue they're having with the belt, or I'm going to show how little I know about fashion.

• 3:42Jacob Effron?

  I was excited to see how many different products you could name there.

• 3:44Speaker 02

  Yeah, belt was the first one. What does Gucci make? But then the customer support representative could say, you know, hey, have you checked out this other line? Or like, some, is that the right?

• 3:53Speaker 01

  Exactly. So exactly, they solved the issue with the customer's belt buckle not working. And then they noticed that you've been on their website browsing handbags, or you have one on your, in your cart, and you've had it for several weeks, but you haven't, you haven't checked out. And now the service representative is coached in real time by the AI to become a salesperson. And we're seeing this actually work, and the conversion rates increased by double digits. And this cost center, which was their customer service team, is now becoming a revenue center.

• 4:22Speaker 02

  So cool. That's amazing story.

• 4:24Jacob Effron?

  I love that like whole, you know, I feel like people are always like, oh, is AI going to take jobs or whatnot? And it's like this idea of being able to repurpose people into, you know, higher value activities for the business is so powerful.

• 4:35Speaker 01

  Yeah, it's like the opposite. It's like people, you meet these customer service representatives. It's really emotional because they feel empowered and they say that they're doing the best work of their careers. And that their careers are taking on a new life that they never thought was possible.

• 4:49Jacob Effron?

  I also love that you have like Gucci as a case study. I feel like if people are curious what our generative AI is actually making its way through. It's a pretty sweet example.

• 4:58Speaker 02

  It sounds like from, you know, just the example that you gave that it's, you know, not just AI showing up in Service Cloud, but across all of the Salesforce products. And congrats on the big GA announcement. Maybe you can, for our listeners, can you walk through what went GA and what some of the newest products are?

• 5:14Speaker 01

  Yeah, we were really excited to GA this week, both our Einstein Copilot, which is our new natural language conversational assistant. That's automatically grounded in all of that customer's data and metadata and flows that they have in Salesforce. And then in parallel, we also GA'd our Copilot Studio, which is it's the ability to customize your own Copilot. It's a prompt builder where you can create and save golden prompts that reference data, specific data fields in Salesforce. And it's our model builder, which lets, you know, the customers who want to fine tune or bring their own predictive models, they can bring their own and plug it right into their Salesforce AI stack.

• 5:55Jacob Effron?

  How do you structure the team to do that? I mean, obviously, I feel like AI touches every product within Salesforce. Like you could imagine having a totally separate team that like, you know, there's probably shared infrastructure across everything you're doing. There's also like within each product team, probably like AI getting infused. Like, how do you think about this, you know, having a horizontal team versus like having AI folks within each existing product team?

• 6:17Speaker 01

  We have continuously evolved. We started off with AI folks largely decentralized in each application cloud. I was running Service Cloud for the last two years, last year when this happened. And we were building our own AI in Service Cloud. But then we realized that what we were starting to build for generative AI, you know, our Einstein Trust Layer, our model gateway to various providers. We knew that every app would need that within Salesforce and it didn't make sense for every cloud to do it on their own. So that's when we we splintered off and we took the Service AI and Sales AI teams. And we created a new shared services AI platform team around that to build out the Trust Layer, build out the Gateway. And that's also the team that's been building Prompt Builder as well as Einstein Copilot.

• 7:01Speaker 02

  So then what still lives within like the the specific, like the formerly existing product teams?

• 7:06Speaker 01

  So they own predictive AI for their use cases. And then they're also now building on the platform. So every cloud, so Sales Cloud is building sales actions for Einstein Copilot. Service Cloud is building service actions for Einstein Copilot. Slack is building their actions on Einstein Copilot.

• 7:23Speaker 02

  Right. And on the Trust side, yeah, it seems like Trust is huge for you guys and you, you know, you obviously serve the biggest companies in the world. How have you thought about kind of putting guardrails around the AI that you're deploying? And, you know, when you're going into really high stakes use cases like support at Gucci or, you know, these types of things where you're talking to real customers, you know, what what guardrails have you guys put in place there?

• 7:45Speaker 01

  We really think about it on three levels. The first is from a technology standpoint. We've engineered Trust into the product through our Einstein Trust Layer that has data masking, data grounding with our data cloud to reduce hallucinations. We have citations, audit trail, prompt defense and zero retention prompts. So there's a whole bunch of features there that really mitigate the data security, data privacy and ethical risks. The second layer of this is our acceptable acceptable use policy. So for example, customers that deploy our AI bots to their customers, the AI is required to self identify as an AI versus otherwise a consumer might not know whether they're talking to a real person or not.

• 8:29Speaker 01

  It's not technically required, but it's our policy. And the third layer is from a stakeholder engagement standpoint. We have, we've developed an open source, a set of trusted AI guiding principles around accuracy, honesty, empowerment, et cetera, that we've shared with the whole industry and also with government regulators.

• 8:47Jacob Effron?

  It's super cool. And I guess that in that first bucket, like of data masking and citation and all, I feel like these are like the problems that everyone's trying to figure out right now. Like, did you guys build most of that in-house or like, have you found that there's, you know, folks you've been able to partner with on that stuff? Like, I feel like, yeah, it's amazing that you've solved so much of that. And I feel like there's probably a lot of our listeners wondering how they, how they do the same thing.

• 9:07Speaker 01

  Yeah, I mean, it's a combination of a build and partner. And so for the startups out there, if you have something really special, please call me. I want to hear about it. And it's like, there's so many, there's an endless number of things that we know we want to put into the trust layer. But just for example, specifically for data masking, it's something that we've had even during the predictive AI days. Because if you think about certain types of, of information, like someone's, even their name, their gender, their zip code, this can introduce bias for both predictive and generative AI. And so for many years now, we've had these, they're called sensitive data fields that we actually proactively surface for our customers. And we suggest to them, we recommend to them that they choose to mask these data fields in any type of AI model.

• 9:50Jacob Effron?

  Obviously, you had to build a lot of this from scratch. Like, I'm curious, like, what's on your wish list of like, if only there was a company that helped us do this or that. Or, you know, as you think about like, uh, doing this, you know, the next time around, what would make it easier? Like, what were some of the thorniest things you guys had to figure out?

• 10:06Speaker 01

  I mean, everything's thorny. It's just like so new.

• 10:09Jacob Effron?

  Yeah.

• 10:09Speaker 01

  And especially when it comes to co-pilots, it's like, we're, we're shifting from this deterministic world to this, to one that's stochastic. And it's just a different way of operating, a different way of thinking, it's a different way of building. And so I think it's, it's, every day is a challenge, but I think in a really good way.

• 10:25Speaker 02

  Kind of moving from deterministic to stochastic, have you had to do any user education around how to use these tools? Or, um, you know, something like the support that's so deeply ingrained in their workflow? Like, how do you think about educating users versus kind of, um, you know, building the product that they can figure out over time?

• 10:43Speaker 01

  There's tons of user education. And I think especially with, there's a narrative out there that will AI replace certain jobs? And I think there's a fear, and I've seen that in some of the customers where we've deployed Einstein GPT and now our co-pilots. And so it's really working with the users to understand, okay, what do we need to build into the user experience and the onboarding and the product itself, the UX itself, to make it very clear that we're here to empower the user and not replace what they're doing. And we want to replace what they don't like of their job.

• 11:16Jacob Effron?

  Yeah. How do you do that?

• 11:18Speaker 01

  It's through ethnographic research. It's sitting in the call center with these users, I've done these visits, and understanding and hearing their concerns. And as they crop up, making sure that we address those either through the product itself or through one of our learning modules.

• 11:33Jacob Effron?

  Is there like a change to the product you've made from some of that that like, you know, you've like particularly resonated as a good way to do that?

• 11:40Speaker 01

  Yeah, I think when the initial message that you, that pops up when the feature first used, it kind of describes what it is. And it's really, even the naming of co-pilot, right? It's not autopilot, it's a co-pilot to be your coworker that you can talk to in natural language and ask any question or ask any request for it to action on your behalf.

• 12:00Jacob Effron?

  I mean, one thing, as you talk about this like movement from, you know, the deterministic, you know, previous AI generation to stochastic models we have today, I wonder like, you know, I'm sure there was a really good Salesforce AI roadmap like before, you know, the explosion of GPT three and ChatGPT. Like, to what extent does this feel like a continuation of that roadmap versus like a, wow, there's just this entirely new set of capabilities, you know, let's, let's maybe shelve some of this stuff we were thinking and move to this other set of things?

• 12:28Speaker 01

  You know, the cool thing is that it was both. First, our Salesforce AI research group, which has been around for 10 years, they're a bunch of, you know, mostly Stanford, former Stanford CS professors and AI researchers. They've been working on large language models since 2018. So shortly after the famous Google paper, our team began publishing papers, including one on the concept of prompt engineering. So that was a radical concept at the time because before that you had to code to be able to access the AI. And so we had this, we've had models and we've been playing around with them. And the summer before ChatGPT was launched, our team was also playing around with GPT, I think two and 2.5. And it was just getting exponentially good.

• 13:13Speaker 01

  And it caught everyone's, it caught my attention. And then we also started working with Gucci back then.

• 13:18Jacob Effron?

  Wow, Gucci, a pile of customer.

• 13:19Speaker 01

  Yeah, Gucci was.

• 13:20Jacob Effron?

  Early adopter for AI. I would not have guessed.

• 13:22Speaker 01

  They were, they, they visited, they visited Silvio Savarese, who's our chief scientist and me, November 2021. I remember because it was the first in-person meeting that I took after getting vaccinated. And we were all super scared. We had like double, double masks on. And we were talking and Silvio was showing off our code gen model. And everyone's jaws were just dropped, right? Mine included.

• 13:47Jacob Effron?

  Yeah.

• 13:47Speaker 01

  And Gucci was game. They were so excited to test this out. They wanted to be the guinea pig. They wanted to co-create with us. And so a year later, or a year and a half later, fast forward to when everyone wanted a large language model, we were able to ship Service GPT in a matter of 60 days. And everyone was like, how'd you do in 60 days? Well, the truth is, we didn't. It wasn't 60 days. It was a year and a half in the making.

• 14:11Jacob Effron?

  And how do you think about like, you know, where to kind of, you know, where it makes sense for that research team to focus today? I mean, obviously, like, you know, you have an amazing group of researchers. You then, you know, probably have the first draft pick of any other set of researchers you want to work with or other companies. Like, how do you think about like where the internal research efforts make sense?

• 14:30Speaker 01

  It's a really exciting question that Silvio and I have actually been working on. Because we were so successful with Gucci and then productizing that, we've actually formed a new team together that's jointly run by the two of us. It's called our Frontier AI Group. And there are certain areas that we've decided that we'll, we'll prototype and actually prototype off of the Salesforce stack. So as if we were a startup and we're just, it's like 30, 60, 90 day type of experiments that we're going to continuously run. And the ones that are successful, we're going to graduate into our core product.

• 15:04Speaker 02

  Hey guys, this is Jacob. Hope you're enjoying the episode. We're trying to get the word out as much as possible, so please share this episode if you think others will find it interesting. Now, back to the show. Have there been any learnings from kind of working off the stack and then things that you wanted to then bring on or maybe make more part of the Salesforce platform?

• 15:22Speaker 01

  So many. It informs what the core stack needs to be able to support.

• 15:27Speaker 02

  Yeah, yeah, very cool. One other thing I was super curious about, you know, I feel like you hear everyone say whoever has the best data is going to have the best AI, right? And that's a big part of the narrative today. And I think, I think that's, you know, Salesforce is kind of the canonical example of that. And you own customer record. And then from that, you, there's all these other things that you can do on top of it, even before AI. Then, you know, you could, all the other tools wanted to integrate with Salesforce to get that data. Now, probably even more because they want to, you know, train agents and models and those sorts of things. So the thing I was curious about is, you know, how do you think about who you want to partner with and give away that valuable data potentially versus building something yourself or building an AI, you know, all of these AI products in-house?

• 16:12Speaker 01

  It's such a good question. So first of all, it's not our data. It's our customers' data. And our, and our customers' data is not our product, unlike a lot of consumer companies. So that's a really important distinction. This data question is so interesting. You can really think about four different types of data that Salesforce has that's really unique and important for AI. The first is what you said is it's a typical data records around CRM. It's these structured data largely is Salesforce's heritage. The second is really where we've pushed over the last two or three years and unstructured data, whether it's knowledge articles that sit inside of Salesforce Service Cloud, or it's the exponentially growing amount of conversation transcripts that we have from

• 16:56Speaker 01

  Slack channels, we have from contact center, voice calls, from chats, from emails, et cetera. And we're now expanding our data cloud to have a vector search and hybrid re-ranking to work across both structured and unstructured data. We also have a zero ETL partnership with all of the major data lake providers, including BigQuery, Redshift, Snowflake, Databricks, et cetera, so that all of the data across a customer's enterprise environment can be brought in. So those are structured and unstructured data. Then we have our metadata layer. Which was created back in the day, 25 years ago, in order to support multi-tenancy.

• 17:38Speaker 01

  And who would have guessed that that same metadata layer would be so important for giving context to AI to know which data objects, which tables, which functions to use. But it's just been amazing. And then the last type of data that we're building is feedback data. Because you can think of Salesforce not only as a world's largest database of customer data, we're also the largest database of customer outcomes. What's the sales opportunity stage? Did it advance to the next stage? Did it dead out? Did the marketing campaign result in an open, a click, a conversion? That data becomes a reward function for any AI model. It doesn't matter if it's predictive or generative. And we're capturing that as a data pipeline in Data Cloud.

• 18:24Jacob Effron?

  So cool. I feel like, yeah, just the sheer breadth of data that you have is incredibly powerful and to the point of like building these feedback loops into the models. It's like incredible to think about the ways you can, you can kind of get smarter and smarter on kind of the advice you're giving or the co-pilot, you know, working with people and helping, you know, upscale them. And I guess across, I mean, a bunch of those players you mentioned, you know, build their own models. Like, how have you thought about both internally, like which models to use? And then on your customer side, like, you know, both in just the closed source versus open source models and any other kind of considerations?

• 18:54Speaker 01

  Yeah, I mean, there's so much happening in the model space, as we all know. And it's very early, it's too early to call a winner, right? There might be somebody who's at the top of the leaderboard now, but it might change in a few weeks. And so we, we've been taking this open architecture approach when it comes to models. So customers can choose their models that are on our service. They can bring their own models and they can bring their own third-party models or they can bring their own first-party models. That's our model builder. And over time, we believe that different models will be used for different tasks and use cases. And so our own Salesforce research model, we are building and fine tuning for industry-specific and domain-specific use cases.

• 19:37Speaker 01

  For example, code generation, Salesforce flow generation, financial services, sales, high-tech sales, et cetera.

• 19:46Speaker 01

  And then our customers, they're selecting whether it's Google or OpenAI or Azure OpenAI. Doesn't matter to us. It's really about choice and also for us to be able to track and benchmark which model is the best cost, performance and latency for each task.

• 20:02Jacob Effron?

  How have you thought about doing that? I feel like that's a problem that I feel like comes up again and again in these podcasts.

• 20:07Speaker 01

  Yeah, well, it's great to be, we're very focused, right? So we're able to create sales-specific benchmarks for the pharma industry and sales-specific benchmarks for wealth management. And so that's what we're doing.

• 20:19Jacob Effron?

  That's super cool. So you have a team within each vertical that like is the expert on that space and knows like what a good what a good response looks like or what a good outbound looks like. And that then becomes like the eval data set that just whenever there's a new model, you can kind of run it through and see how they do.

• 20:33Speaker 01

  That's right. It's not done yet, but that is what we're doing.

• 20:36Jacob Effron?

  That's really cool.

• 20:37Speaker 02

  It would be fascinating data to see how, I don't know, Llama performs in the pharma industry compared to OpenAI or, you know, what customers have built themselves. Yeah, I'm sure it's really interesting.

• 20:49Speaker 01

  And it's complicated though, because it's the model, but then you fine tune it. And so it depends on how you fine tune it and with what data. And then it's entire RAG pipeline that you have to fine tune. So there's a lot of potential variables that affect the performance, but we want to benchmark every step of the way.

• 21:03Jacob Effron?

  I thought that feels like the future, like even in the data labeling space, it feels like that's moving more and more toward like specialist expertise, you know, within verticals. And I think we haven't solved general eval yet, but even if we get a little progress there, it feels like inevitably most of your customers, they probably care about, you know, they don't care about how the eval is across a million different spaces, just like how does it perform in our world? And so that's really cool that, you know, it makes total sense that you guys have a lot of people that are really, you know, experts in those spaces and can build that.

• 21:30Speaker 02

  On the cost side, you know, I feel like there's been a lot of talk about the cost of AI in the news and how expensive it is, but also it's, you know, going down 10x year over year for inference. And I think there's a similar curve for training. What are some use cases or if there's any that you might be holding off on today because they're maybe too expensive, but if AI was 10x cheaper, then you might explore them.

• 21:54Speaker 01

  I mean, I guess if it were a lot cheaper or close to free, you would just test everything on every model every time.

• 22:00Speaker 02

  Yeah. Yeah. Yeah, I guess that's a good point. Yeah.

• 22:03Speaker 01

  Yeah. And you wouldn't have to think about cost optimizing strategies. You wouldn't have to ever cash anything.

• 22:08Speaker 02

  Is cost a big consideration for you all when it comes to inference or training or? You know, maybe, I mean, Salesforce is obviously a behemoth, so tons of research, resources that go with that.

• 22:20Speaker 01

  Well, we, we pass the costs on, we capture the costs in our, our product SKUs. Cost is a consideration for customers, not when they're initially piloting and testing it out, but as they think about scaling out to their entire contact center, for example, and multiple use cases, it is a big consideration. And so that's, that is why we are looking at that multi-model, multi-model size architecture and being able to be able to route the right task to the right size model on our customers' behalf. And then also why it's so important for our customers to be able to define clear business cases for why they should invest, right? Just going back to the Gucci example, it's about reducing

• 23:04Speaker 01

  the average time to resolve a customer issue. And also, what's the incremental revenue lift that you can drive with these AI inferences?

• 23:12Speaker 02

  Amazing. I'm gonna know a second Gucci product by the time we get, yeah, it's gonna have belts and handbags, I don't know.

• 23:19Jacob Effron?

  I'll show up to the next recording, I'm like, a full fit.

• 23:21Speaker 01

  Gotta do some shopping after this.

• 23:22Speaker 02

  Yeah, I know. I'm gonna test out the customer support, I'm, I think they'll have a great experience.

• 23:26Speaker 01

  Well, they're right down the street from here.

• 23:27Jacob Effron?

  Exactly, it's just, it's just AI research, right?

• 23:30Speaker 02

  Yeah.

• 23:30Jacob Effron?

  At that point. I guess like, you know, the cost thing is so interesting and it comes up all the time. You're talking to enterprises every day, like, if you were to stack rank, like, you know, there's, there's obviously barriers to going from trying something out to, to doing it at scale. Like, is cost, you know, the top reason? Is it actually like really making that business case? Is it something else? Like, you know, I feel like there's this feeling a lot that, you know, a lot of people try this stuff out and then, you know, there's maybe the select few that are really scaling it. And so I'm wondering what you're seeing in terms of some of the barriers to, that folks have in, in terms of rolling this out more broadly.

• 24:06Speaker 01

  I think the biggest barriers are on trust because it's scary, right? There are real data security, data privacy risks. And it's not just about data leaking out of an organization, although that is also not acceptable. But even within companies, you think about different employees and different departments have access to different workflows and different sets of data. And so really having to honor those sharing rules and entitlements in the AI is really important. I think once we get customers comfortable around that, then they're usually very excited to start testing and playing around in our prompt builder, in our playground. I think the second one after that is the business case, and it isn't just about cost, right? It's about productivity and margin expansion.

• 24:49Speaker 01

  And if they can make the equation work out, then cost doesn't matter if they're getting more in return in ROI than they're spending.

• 24:56Jacob Effron?

  Have you found any good ways to baby step people? Because I feel like one thing we had, you know, Des from Intercom on, and one thing he was saying that we thought was interesting was, you know, he's like, well, we tell people, well, turn it on for like weekends or like, you know, nights or times when like they're, you know, and baby step into it that way. And, you know, I'm wondering if you guys have seen anything where it's like, you kind of recommend, like, if you don't fully, you're not fully sure yet, like, try this part first.

• 25:20Speaker 01

  Yeah, I think having those turnkey AI use cases that we launched last summer was really important because I think otherwise you're asking a customer to write their own prompts and not everyone knows how to do that. And so having those turnkey prompts that just literally takes five minutes to get up and running with the service GPT, sales GPT, commerce GPT use cases, people see immediately what the business value could be. And then what they've been asking us for the last few months, since these products have been out is, how do I make this my own? I love the service reply recommendation, but how do I ensure that it captures my brand voice, which at Gucci is going to be different than AAA insurance, is going to be different than Ford Motor Company? And then they'll always want to reference their custom data, their custom field in

• 26:05Speaker 01

  Salesforce, right? In Gucci, they have a custom field for what your style is, right? And how frequent of a shopper you are. Ford is not going to care about your style. They're going to want to know the make and model of your car and how many Ford cars that you own in your in your family. And so it's just being able to to reference that data really makes it their own. And that's why I think Prompt Builder and Copilot as AI platform features is going to make it even more relevant.

• 26:30Jacob Effron?

  And behind the scenes for like a Ford versus a Gucci, like is it just you fine tune a model differently? Or is it like the RAG pipeline works differently in terms of what you're retrieving? Like, how does it? How do you end up like actually making the the Ford voice sound different than the Gucci voice or?

• 26:42Speaker 01

  You can just put in the prompt.

• 26:43Jacob Effron?

  Yeah.

• 26:43Speaker 01

  I mean, certainly you can fine tune a model, but it's it's much easier just to to put that in the prompt.

• 26:47Jacob Effron?

  You were alluding to this earlier about, you know, you get you give a ton of different options to customers to like choose which model they want to use. And I'm curious if you noticed any trends in like, you know, customer, like certain types of customers, like certain models or for certain use cases, it makes sense to use a larger model versus a smaller model or open source for closed source or anything you've kind of seen as as people have been choosing from this kind of ecosystem of models that you make available.

• 27:11Speaker 01

  Yeah, I mean, we don't play any favorites, but I would just say from a trend standpoint, unsurprisingly, if you're an existing large Google Cloud customer, you're generally using Workspace, you're using BigQuery. You tend to prefer Gemini, right? It's just easy and you already have have an open PO with with the company. I think another trend is, you know, OpenAI's models are still the the cutting edge. And so for very complex types of use cases, agentic planning, plan generation, GPT-4 is still best in class that we're seeing.

• 27:46Jacob Effron?

  One thing that I think gets debated a lot is kind of value accruing to incumbents in AI versus startups. And I think Salesforce is, you know, one of the things that people point to a lot of case for incumbents winning, right? You have all of the data and layering on these amazing AI features on top of that. Like, how do you think about, you know, Salesforce and then versus like some of the newer players? There are some ex-Salesforce people, I think, that are out there with Brett Taylor and his new company. And I don't know, like, any thoughts on kind of the big companies versus newer companies and how that all shakes out?

• 28:20Speaker 01

  I know that's a fun debate to have, but it's it's just, I think there's so much space for innovation and value creation. And I think we saw this in previous waves of software that if you were an incumbent, sometimes that that advantaged you. Sometimes it disadvantaged you because you were slower. Brett's a very smart guy. He's a former classmate of mine. I'm a big fan. And I just think there's going to be a lot of different players in the space. And the other thing is, you know, you asked a question earlier about Salesforce Ventures. I mean, we're investing in companies that we think will become all multi-billion dollar, some of them already are multi-billion dollar valuation companies, maybe trillion dollar companies. And I think it speaks to the room in the ecosystem for lots of value creation.

• 29:03Speaker 01

  We hope startups build in the Salesforce ecosystem and on the platform. We do think that there's a lot of value that our data cloud and our UI can offer to these startups that's really great for them and also great for us.

• 29:15Speaker 02

  Totally. Yeah, I think that makes a lot of sense. And we believe there's tons of opportunities for the sales forces of the world, but also new companies that will, you know, be multi-billion dollar companies one day, as you were saying. And I think that kind of follows what we've seen in other trends of, you know, cloud, like there may be some new winners or mobile, there might be new winners, but then also Google and Amazon. And I think a lot of those companies, their position was reinforced by riding that wave as well, so.

• 29:42Jacob Effron?

  Good for our job to have startup opportunities.

• 29:44Speaker 02

  Exactly, yeah. We got to be a little bit biased, you know?

• 29:46Jacob Effron?

  Exactly. If we're, uh, it was just the incumbents, we'd have to pack up and shut the podcast studio down. It'd be very sad. I mean, obviously, you know, you've been a super successful entrepreneur. Like, I'm curious, you know, if you weren't at Salesforce today and you were kind of like, you see so many things happening in generative AI, like, what would you go build right now?

• 30:07Speaker 01

  I have 20 ideas. If I didn't have as great of a job as I currently do, I mean, I loved being an entrepreneur. I founded Hearsay Systems in 2009, was CEO for 10 years. We were actually in the AI space. The hear part of Hearsay is mining signals from your connection, social media posts and text messages to look for lead generation events. The say part of Hearsay is predictive AI on what you should say to whom at the right time, based on those signals. And so I think there's a lot of, again, room to create really meaningful businesses. And I have a lot of friends doing it, and I'm trying to help them out.

• 30:44Jacob Effron?

  If Hearsay was being started today, like, how do you think that looks different with, like, you know, these large models? Like, to what extent is that, like, yeah, how would you think about that?

• 30:54Speaker 01

  Well, first of all, the the hearing part of hearsay is much easier now.

• 30:58Jacob Effron?

  Yeah.

• 30:58Speaker 01

  The team has remade it. I mean, we were doing regular expressions like that was our our NLP.

• 31:04Jacob Effron?

  Yeah.

• 31:04Speaker 01

  And now, of course, it's just much more accurate to be able to mine. And you think about there are millions of text messages that the regulated financial advisors and insurance agents that use hearsay, they're sending back and forth on social platforms as well as via text messaging and MMS. And so now they're able to mine all of that for insights on a client basis, on the advisor basis, and then on a firm and regional basis, which is just unlocking a totally new set of insights that was never there before.

• 31:31Jacob Effron?

  I guess it feels like today, the like killer use cases for generative AI in the enterprise are like chat your documents and customer support. It feels like those two are the ones that are taking off. I'm curious like

• 31:39Speaker 02

  Co-generation.

• 31:40Jacob Effron?

  Yeah, co-generation. Very fair. Shout out to Paul said.

• 31:43Speaker 02

  Yeah.

• 31:44Jacob Effron?

  But I guess like I'm curious, as you think about, you know, the next year or two, like what's going to join that list of like, you know, we'll be sitting here and yeah, when we're rattling off the obvious ones, there'll be, you know, one or two other ones.

• 31:56Speaker 01

  I think every department, I think everyone in company needs to rewrite their job description. And it starts with the functional leads and first line managers, teaching people and and sharing those expectations, just like in the 90s, I imagine, like being in the office, you know, it was before I worked, but it was like, I bet people had to learn how to use the internet and learn how to use email and learn Google search. I think the same is going to be true of AI. I think about some of the things that we're doing now at Salesforce is we're doing our V2 mom planning as our vision, values, methods, obstacles, and measures. So it's our version of an OKR. And now we're using our secure version of generative AI in Slack to be able to be able to help brainstorm what that V2 mom is for my department, for my role, for for the whole company.

• 32:41Jacob Effron?

  How's that work?

• 32:42Speaker 01

  It's great. It's like you put in your ideas and all these initiatives and it can help summarize what the methods are.

• 32:48Jacob Effron?

  How else do you guys use generative AI internally at Salesforce?

• 32:51Speaker 01

  In every way. I mean, we were thinking about how we launch our new Einstein co-pilot product or our co-pilot studio. And we just, we use the AI as a co-pilot in Slack because then you can have lots of people all on the same channel and you're just brainstorming ideas with another teammate.

• 33:11Speaker 02

  I really like the little the Einstein guy.

• 33:13Speaker 01

  Thank you.

• 33:14Speaker 02

  And I feel like he will be AI generated very soon if he's not already. So, but yeah, I do the the videos are are awesome. So.

• 33:21Speaker 01

  Thank you, and he is very cute.

• 33:23Speaker 02

  Yeah.

• 33:23Jacob Effron?

  Yeah, it's really interesting. I feel like that is like an emerging trend of like you, I mean, you mentioned Slack, like you have a channel with people and also like an AI co-pilot. And I feel like even in the consumer world, like, uh, character has those like chats where it's like actually your friends, but also like some, you know, AI bots as well. And I wonder whether it's like an interesting way where it's not just you chatting one-on-one. It's like there's a community of people as well, but also some like, uh, you know, and AI side to like augment that conversation.

• 33:51Speaker 02

  I think it's really interesting, this kind of group chatting with AI and you could imagine group coding with AI where Jacob and I are both coding and then we're tagging an LLM on, hey, what do you think about this? We can both see that response. And I would imagine there's so many things that you can do around customer data with that sort of workflow as well, whether it's like trying to close a sales account and the AI is chiming in where everyone can see like, hey, have you thought about doing this? Or this may increase, you know, our chance of winning by X percent if we do this. So I think it's a, this kind of team plus AI will kind of be the future.

• 34:24Speaker 01

  I agree. It's really neat to think about and Slack is, is a perfect place for it to happen.

• 34:28Speaker 02

  Yeah.

• 34:29Jacob Effron?

  Probably everyone listening to this uses Slack. And so like, I'm sure everyone's intrigued, like, you know, how are you thinking about kind of like AI's role in Slack? And, um, you know, what do you imagine that product looks like in the future, you know, given some of the stuff we're talking about?

• 34:41Speaker 01

  Well, we've launched Slack AI, which includes conversation summaries and channel summaries, which is really awesome. You should check it out if you haven't. Just like saves me a ton of time and catches me up. And then what we're working on now is bringing Einstein co-pilot into Slack. So that everything we've just been talking about, whether it's a service reply recommendation, or asking a salesperson asking to summarize everything that's happening at a customer, all the open support tickets, all the, the marketing web engagement and events engagement, and summarizing all of that before they walk into a meeting. They'll probably want to do that in Slack so that the whole account team, to your point, gets to see that.

• 35:18Speaker 02

  I never thought about how amazing a UI Slack would be for that. But yeah, that makes a lot of sense. And Jacob slacks me all the time. So being able to get a summary of his would be great.

• 35:28Jacob Effron?

  I want co-pilot suggestions on how to respond to your ridiculous. Somebody someone could could help draft that.

• 35:34Speaker 01

  You can just have AI generating all of your Slack messages and then you would have AI summarizing all.

• 35:39Speaker 02

  I know. And then I would make my AI really verbose so that it wastes Jacob's AI time.

• 35:45Speaker 01

  But then he would have an AI to summarize you.

• 35:47Speaker 02

  That's a good point. And then, yeah.

• 35:48Jacob Effron?

  You're just drive up my inference costs like that?

• 35:50Speaker 02

  I don't know. Noisy channel, setting bits over the, I don't know, something like that. Yeah.

• 35:56Jacob Effron?

  I love it. You know, given so much of the focus is is on, you know, or one of the main areas you guys do is customer support. I thought the question that that everyone always asks is, you know, what percent of like customer support questions do you think can be answered by AI today and like in the future?

• 36:10Speaker 01

  It really depends on the industry and the company. I mean, so it's hard to give like a blanket answer. But the other thing is, the really neat thing about AI is it's not just good for question answering. It's also good for workflow orchestration. So if someone wants to initiate a return, they can empower their co-pilot to initiate that RMA and send the shipping label to the customer via email or whatever workflow the customer has built into Salesforce.

• 36:38Jacob Effron?

  It's amazing that you have, you know, actual action taking, because I feel like that's usually the precipice that people are too scared to cross. So they're they're afraid, you know, I guess what that happened with Air Canada where they the chatbot was like, you know, just issuing refunds left and right.

• 36:51Speaker 01

  Well, I think it's really important to have that layer of governance and control. And so it's not that the co-pilot doesn't automatically get authorized with all of the flows in an organization. The admin has to go into the Einstein one co-pilot builder and they can pick from their existing flows and Apex, which ones they want to designate that their their co-pilot has access to. And of course, you would want to be cautious and maybe start with the the least harmful, maybe the lookups.

• 37:17Speaker 02

  Yeah.

• 37:18Speaker 01

  Versus issuing refunds. And then they can also create new ones.

• 37:21Jacob Effron?

  Totally. I feel like one thing I'm struck by is is just like. It makes so much sense. There's just such requirements for permissioning. There's such requirements for like, as you step in cautiously but optimistically into, you know, AI landscape as an enterprise, like you're going to want to enable some things, not enable other things. And it feels, you guys obviously have all this set up already to just seamlessly allow people to do that. And it feels like a much higher hill to climb for like, these startups that are trying to do net new things in some of these spaces.

• 37:48Speaker 01

  Well, they can build on a platform and be able to access the same things. And that's the really neat thing about these co-pilots is that, you know, they, they, they might look the same on the surface, if you're using what was formerly known as Bard versus ChatGPT versus Einstein Copilot, but it's what you don't see that makes it so special. It's all the data and the metadata that we were talking about, which means that if you and me are both employees at the same company, you asking Einstein Copilot, what's your forecast for the quarter, is going to yield a different answer than what I ask, because we were owners on different opportunities, and we have access to different data and different business processes.

• 38:24Speaker 02

  What if Jacob's always sandbagging? And so you ask Einstein, what will Jacob tell me that then I need to add 20% to? I guess the AI could learn that as well.

• 38:33Speaker 01

  Yeah, exactly.

• 38:34Jacob Effron?

  No more, no more sandbagging for me.

• 38:35Speaker 02

  I know, I know.

• 38:36Speaker 01

  Yeah, party's over.

• 38:38Jacob Effron?

  Indeed. Well, we always like to end our conversation with kind of a quick fire round, where we get your quick take on, on different things. And so one question we always like to ask is, one thing that you think that's overhyped and one thing you think that's underhyped in the AI world today?

• 38:52Speaker 01

  It's so hard to answer, because I think everything in AI is really hyped right now.

• 38:58Jacob Effron?

  We certainly feel that.

• 38:59Speaker 01

  But I also feel like we haven't seen anything yet.

• 39:02Jacob Effron?

  Yeah.

• 39:04Speaker 01

  I feel like it's 1998 for AI, except it's happening much faster.

• 39:09Jacob Effron?

  What's kind of a across the horizon that gets you most excited?

• 39:12Speaker 01

  I just think we're on the precipice of the most exciting, not just technology, but societal change that we will have ever experienced as a civilization. And just for us to get to be a part of it, not just to shape the future, but to make sure that we're doing it responsibly, and that the tremendous benefits that come from AI get accrued to everybody, not just a select few.

• 39:34Jacob Effron?

  You work, obviously, super closely with Marc Benioff, who's like this iconic entrepreneur of Silicon Valley. Like, any good stories about working with him, or what's, what's that been like?

• 39:44Speaker 01

  There are a lot of stories.

• 39:46Jacob Effron?

  What's he like to work with?

• 39:48Speaker 01

  He's really inspiring and fun. And I'm amazed by his ability to grasp a lot of different domains. Like, a lot of people think, oh, he's just a sales leader, or he's a marketing person. But he's also understands engineers and code. And he understands what's happening with AI in a really fundamental way.

• 40:09Jacob Effron?

  I imagine, like, you know, obviously, just given the extent of what you shipped already, that like, everyone's been pretty involved in a lot of this AI planning. And it just feels like such a top priority for the company right now.

• 40:21Speaker 01

  Yeah, it's taken a village.

• 40:23Jacob Effron?

  Yeah, no, it's really impressive. Well, I mean, I think, you know, would love to kind of leave the last word to you. Like, where can people go to learn more about you, about all the stuff that you just GA'd? I guess it won't be yesterday by the time we release this, but pretty recently, you know, with Salesforce, what's the best place for people to go?

• 40:39Speaker 01

  So people should go to Salesforce.com/ Einstein.

• 40:42Speaker 02

  Thank you so much for joining us. This was really fun and appreciate you taking the time.

• 40:47Speaker 01

  Thank you. This is a lot of fun.

• 40:48Jacob Effron?

  Pat, that was a fun one.

• 40:50Speaker 02

  Yeah, it was super fun. It was awesome.

• 40:52Jacob Effron?

  I feel like that was the first time we've done one of these podcasts where Gucci came up. I think, you know, five, six times.

• 40:59Speaker 02

  Gucci came up and Taylor Swift did not, which was surprising because we've had Taylor Swift come up several times.

• 41:05Jacob Effron?

  It's very off-brand for us.

• 41:06Speaker 02

  It's the first time that Gucci's come up. So, yeah.

• 41:08Jacob Effron?

  I was kind of maybe, I was hoping that Salesforce was maybe supporting like the Taylor Swift Eros Tour or something. There was like some way it could have been adopted.

• 41:15Speaker 02

  Some collab, yeah, we should, next time we got to fit it in. I thought it was really cool, the tangible use case that she was talking through with Gucci and how they're using AI for customer support. And I think it really shows how good customer support is as a use case for AI. And, you know, we were talking about it as customer support, code generation. These are kind of some of the core use cases where AI will.

• 41:42Speaker 02

  Climb into the enterprise and I think already is and so that was cool.

• 41:45Jacob Effron?

  Yeah, I mean, one of my main takeaways, and I think I mentioned this to her is just, you know, obviously as enterprises think about using this stuff, it's amazing just the complexity of like, what do you do permissioning for? Like, maybe you want this workflow to go through your co-pilot, but not this other workflow. And, you know, they're obviously just in such an advantaged position from having, you know, having a lot of that logic already built out. So they're like, oh, we know your, you know, 20 workflows. Like, let's, and you think about, you know, kind of the hill to climb for a net new company versus, you know, Salesforce sitting on top of that. But it feels that there's just going to be a real need for like startups and other companies to help figure out this like permissioning and like the how you like partition workflows to go through the models and not go through the models. Like, it feels that this is a common

• 42:30Jacob Effron?

  problem we hear across the board.

• 42:32Speaker 02

  Totally. Yeah, it was, it was super interesting to hear about how they solve all those things. And one other thing that stood out to me was just how, one, they have moved quite quickly in the space to ship a bunch of AI stuff. But then, two, I think it was Benioff that said, always be launching, or, I don't know, that seems to be one of the Salesforce-isms. And every single time something came up, it was like, we just launched X and we just launched this. And, and, you know, they're not just shipping, but they're also pushing this narrative about all the stuff that, that they're, they're putting out there. So I thought that was interesting.

• 43:08Jacob Effron?

  Another thing that stood out to me is the power that they have of, you know, obviously having these industry-specific teams and being able to build these vertical-specific eval sets. I mean, you know, hell, we talked to enterprises and people are trying to figure out just general eval data sets. And they're already like, we have like the life sciences, sales, eval data. And it makes total sense that like, eventually we'll get into a world where you'll have like some sort of way to evaluate models on your very, very specific use case. And obviously, you know, I think much of the energy right now is just toward generally like, is model X better than Y? But it's really like, is model X better than Y in, you know, telling the person that had a belt support issue and upselling them to, you know, a handbag?

• 43:45Speaker 02

  Totally. And the models, I mean, what the models are asked to do today is such a huge surface area that it's hard to know, unless you have an evaluation for a specific domain, it's hard to know which model is really better. At least that's, that seems to be. What what we've heard and so yeah, having the domain specific evaluations, I think will be it'll be really interesting. And that's probably where most people will go because it could be like, oh, this, you know, llama outperforms Gemini on this question answering set. But maybe that has no bearing on the task that I actually want it to do.

• 44:21Jacob Effron?

  Totally. And even just being able to try your fine tuned models, you know, across the specific specific tasks you do. I liked your question where you were like, any early signs that like, you know, Gemini just crushes life sciences and like, you know, uh, llama is like the perfect finance model, but I guess too early to, given they G eight it yesterday. Uh, too early to tell.

• 44:37Speaker 02

  Yeah, yeah, it makes sense. Um, well, thank you for the kind words.

• 44:41Jacob Effron?

  Yeah, every so often.

• 44:42Speaker 02

  Yeah, you know.

• 44:45Speaker 02

  I thought her point on Slack being the perfect interface for generative AI was very interesting. Cause I had been thinking more about Salesforce from the standpoint of CRM and the data that they have with CRM. But I do think Slack is is quite interesting because as you, you know, can chat more and more with AI. You also have your team there. And I do think a lot of the workflows will involve your team and AI. Um, whether that's coding, you know, coding with your team and AI or asking AI to generate a block, a block of code and having your team comment on that. I think, um, a lot of the workflows will look like that where you kind of have people and your team and AI involved. And, and so Slack would be really cool interface for that. And we've seen a bunch of startups that are kind of building bots into Slack

• 45:29Speaker 02

  or into email. And I think it's, um, it's really cool. I think some of those will be successful, but also, you know, Slack has an advantage by already having all the data.

• 45:40Jacob Effron?

  And I think just interesting, like, you know, I feel like so much of the way we interact with AI tools today is like, single player, where it's like just you and the, and the AI hanging out. And it feels like inevitably you'll move into this like multiplayer world where, uh, you know, we get to hang out with our favorite, uh, AI, uh, help.

• 45:53Speaker 02

  Soon it will be you, me and Tony Stark on the podcast and we'll have.

• 45:57Jacob Effron?

  Yeah, we'll just have him represented in your sweet socks.

• 45:59Speaker 02

  Exactly, exactly. Um, maybe that's a good segue into, we announced a few new investments.

• 46:05Jacob Effron?

  We did. It's funny. Sometimes it's, uh, it's quiet for a while and then we just, uh, unload a bunch at the same time.

• 46:12Speaker 02

  You want to chat, uh, you know, tell the listeners a little bit more about a bridge and, um, why you guys got so excited there?

• 46:18Jacob Effron?

  Yeah, I mean, look, we've been meeting like every AI application under the sun, right? Ton in legal and finance and healthcare. And I think, you know, I think the potential for AI in healthcare has always been clear. But, you know, I think what's really incredible about a bridge is it's basically a technology that, you know, listens to doctor-patient visits and then, you know, puts the summary of those notes in exactly the format that's needed in electronic health record for the doctor, gives the patient a summary of that visit in a way that they can understand it. And you think about like where that can go long term. I mean, there's just, everything flows from that doctor-patient visit. And the technology is finally good enough for, for speech recognition, for summarization. But it's something that saves doctors tons of times, prevents them from burning out, it's something that helps patients better understand what's happening. It's, uh, it's

• 47:03Jacob Effron?

  hard not to get inspired about a mission like that.

• 47:05Speaker 02

  Totally. I think it's an amazing application. And, you know, one that will drive a bunch of value, but also will really help people and help doctors. And I think it's, it's a super cool company. And yeah, it'll be, I'm excited that, that we can be a part of it.

• 47:22Jacob Effron?

  Totally. I mean, what's nuts is like, you know, you probably have doctor friends too. It's like they have this, there's this thing, it's literally a term for it. It's called pajama time. And the idea is that basically like, you already work like crazy as a doctor, and then you come home and you're gonna want to have dinner with your family or something. And it's like, nope, three hours of just typing in notes, you know, and it's, it's just incredible that a bridge can basically get rid of that.

• 47:42Speaker 02

  Totally. And that's definitely something that AI should be doing and not humans, right? Like AI should be great at taking unstructured voice and conversations and then turning that into structured data for your EHR. And so it makes a lot of sense even to, you know, someone like me that doesn't know much about the healthcare world at all. So.

• 48:02Jacob Effron?

  Now, and then, you know, we, we, you know, in addition to hosting podcasts, I guess occasionally we, we get to work together on investments, which is, which is always the dream.

• 48:10Speaker 02

  Always the dream.

• 48:11Jacob Effron?

  And so tell our listeners a little bit more about, about why these Iron Man socks are getting you thinking.

• 48:17Speaker 02

  Yeah, so we announced our investment in Ideogram and Ideogram is a modern text to image model today, but in the future, kind of reinventing how people think about generative media, starting with images. And so the team is incredible. That's the team that built text to image models at Google and the Imogen team and they, you know, the de facto experts in the world and built an amazing model that that really shines for images with text, which is something that a lot of people want for marketing type use cases, but also just really high quality images. And so it's been been really cool to be a part of and learn from Mohammed and

• 49:01Speaker 02

  the team. And yeah, kind of amazing application of generative AI. And they're, you know, they're both building the model and the consumption experience. So they have, you know, really awesome community of people that are building and remixing each other's images with AI. And then, you know, on the back end, state of the art AI model to support all of that.

• 49:25Jacob Effron?

  So totally. And I feel like we were talking about evals earlier, like, you know, I think I mentioned this on this podcast before, but it is worth noting, I think you have an eval data set of, you know, one prompt for image models, which is always just Iron Man surfing a wave. Yeah, it's like always, you know, that's, that's how you measure quality.

• 49:40Speaker 02

  The advantage of just having one prompt is it's really easy to compare model to model. And so then I can see how they do on the new Iron Man surfing a wave prompt. And Ideogram, they just launched 1.0. It did very well.

• 49:54Jacob Effron?

  It's come a long way. Iron Man is really crushing that wave.

• 49:56Speaker 02

  It does very well on the Iron Man surfing prompt. In particular, I was very impressed with the hand placement. It was.

• 50:04Jacob Effron?

  It's an expert surfer.

• 50:06Speaker 02

  Iron Man was in a very athletic looking stance, kind of hands forward and where you would expect him to be on the wave. So I thought that was very cool.

• 50:14Jacob Effron?

  I don't know if I have the, I'd be bad for that evaluation because I'm not sure I totally have the surfing form down myself. You know, in other AI news, I mean, I feel like Google's been in the headlines a lot. Obviously, the open source Gemma, which is a pretty powerful open source model and cool to see them, you know, decide to open source something.

• 50:31Speaker 02

  Totally.

• 50:31Jacob Effron?

  And I think great for the ecosystem to have yet another really powerful model. And then obviously, you know, received some more negative headlines around, you know, some of the biases in their model and their image generation models. And it seems like they're, they're working on fixing it, but obviously, you know, I think took what was a potentially very good news week for them and unfortunately a little less exciting.

• 50:54Speaker 02

  Yeah, totally. I think became more mixed news for them, but I think the Gemma thing is really interesting from the standpoint of there's now another tech giant that's investing in open source. And, you know, Lama's been doing the, or Facebook has been doing the Lama thing for a while. And I think Zuck committed to $10 billion in compute for, to continue to invest in those, those models. And now we see kind of Google also stepping up to the plate and investing in open source. So I think it's, it's great for the open source ecosystem. It'll be interesting to see if it's kind of a, you know, one and done, or if they want to continue to put out models into the open while keeping, you know, probably the, the best models proprietary like Gemini or Gemini 1.5.

• 51:37Jacob Effron?

  It was a good few weeks for open source. I mean, the Mistral-Microsoft partnership, like I think there's, there's clearly a lot of like enterprise demand for these open source models and exciting to see. We love open source.

• 51:47Speaker 02

  Totally.

• 51:49Jacob Effron?

  Well, Pat, that was a lot of fun. I'm very excited to see, I think our, our viewers will all be expecting you to roll into our next one with like a full Gucci fit. And so I hope no one is going to be too dissatisfied here.

• 52:00Speaker 02

  I would temper your expectations if, if that's what you're hoping for, but, but yes, I, I definitely need to research more about Gucci and maybe up level my, my fashion game, so.

• 52:11Jacob Effron?

  Yeah, at least next time we need to be able to name more than, than two articles.

• 52:15Speaker 02

  Yeah.

• 52:16Jacob Effron?

  Awesome. Well, that was a fun one.

• 52:17Speaker 02

  Super fun.