Ep 41: Head of AI at Snowflake Baris Gultekin on Why They Built Their Own LLM, Governance as a Moat, and the Most Common Enterprise Use-Cases

Snowflake sits in a unique position in the AI landscape: They enable fast, secure, and scalable proprietary data access for thousands of customers, many of whom are building AI tools. They also maintain their own suite of AI products, increasing the utility of their platform and empowering customers who may not have the resources to build their own. That’s why it was so fascinating to speak with Baris Gultekin, Snowflake’s Head of AI, on our latest Unsupervised Learning. Baris has helped Snowflake launch several key products, including Cortex, AI Data Cloud, and even Snowflake’s own LLM, called Arctic. Baris was a founder himself — he joined Snowflake through an acquisition of his blockchain API startup, nxyz. He has a unique window into the future of AI via his role building key infrastructure. It was really fun to dive deep into this side of the AI ecosystem with Baris.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast. And today, we have on Barris Giltekin, Snowflake's Head of AI. Barris leads Snowflake's wide-ranging AI efforts, including products like Cortex, AI Data Cloud, and launching their own LLM, Arctic. Typically on this show, we interview founders building AI products. Barris is a little different. He's working on the infrastructure that supports AI tools in production, and was able to share a lot of really interesting insights about the types of use cases he's seeing most, and commentary on when things are working like fine-tuning and rag. This was a really fun conversation with one of the most important infrastructure players in the AI ecosystem. Here's Barris. Thanks so much for coming on.

• 0:31Speaker 01

  Of course. Thanks for having me.

• 0:32Jacob Effron?

  I feel like there are so many different places we can start because you have a really broad kind of AI product portfolio at Snowflake. Maybe the best place to start would be with just Arctic LLM and obviously your kind of efforts on the open source model side.

• 0:43Speaker 01

  Yes.

• 0:43Jacob Effron?

  What goes into like an effort like that? When did you decide like, hey, we kind of need our own model? And how many folks does it take to kind of get something like that up and running?

• 0:52Speaker 01

  So that effort was relatively small compared to some of the others in the market. We started in December or so timeframe when we wanted to have our own model because a lot of our customers really want to build BI-type experiences using AI. And there, of course, text to SQL, natural language, from natural language to SQL to data really matters. So we wanted to build a large language model that can do SQL really well, that can also follow instructions really well. And a lot of the models out there are great, but when it comes to SQL, there's a lot of complexities. And so we wanted to invest in the space broadly.

• 1:33Speaker 01

  We were really fortunate to get amazing researchers from DeepSpeed as well as VLLM, like these are kind of founders of these two projects. And once we got the team together, and it's a relatively small team, we were able to build the model in a...

• 1:52Speaker 01

  Three, four month timeframe and then release it.

• 1:56Jacob Effron?

  Yeah. And one of the things I think you said when it came out, which I liked is that you were like, look, this is an LLM that's specifically for enterprise needs, you know, we were not going to be your best on composing poetry. But we'll probably be pretty good on the on the sequel co-pilot and high quality chatbot. Like, what goes on behind the scenes to make those optimizations for a model that's particularly good at that? And maybe isn't going to do our Shakespeare sonnets?

• 2:16Speaker 01

  Surprisingly, the, you know, it does those as well, but so there were a couple of things that went into it. First of all, the team had a pretty innovative architecture. We really focused on making the model really efficient, very efficient to train as well as efficient to do inference as well. You know, we were able to build the model at one eighth the cost of kind of similar models out there. And once we get the architecture right for that optimization, it was all about, do we get the data recipe? How do we make sure that the data recipe is correct? And we spent a lot of time on that and then getting that data right. And then I got the model ready.

• 2:56Jacob Effron?

  And what are some of the ways that people are using Arctic today? There's kind of, you know, the BI side of things, SQL generation, but also you allow people to run the LLMs within their Snowflake or on their Snowflake data.

• 3:08Speaker 01

  Yeah, that's right. So at Snowflake, we have this product called Cortex. Cortex is our managed service that allows us to run all these large language models, our own Arctic, as well as many others, like from Mistral, from Meta and so forth. And customers tend to do three major things. And one is, we talked about AI for BI type use cases. The other one is, you know, the traditional, you know, build a chatbot for, you know, talking to their unstructured data, documents, PDFs. And then perhaps uniquely, Snowflake, we have a lot of structured data in Snowflake as well as semi-structured data. So a lot of our customers kind of use natural language to extract data from text and then process it later.

• 3:53Speaker 01

  So a lot of text analytics, natural language processing, you know, understanding what is happening in sales call logs.

• 4:02Speaker 01

  And you know, customer support tickets and you know, all sorts of other kind of semi-structured text. Processing all of it in batch is a big use case.

• 4:11Jacob Effron?

  I guess for two of those use cases, like maybe the BI use case and the data extraction use case, might be helpful just to level set from you on like, where are we today? And like, when does it work? When does it not work? Like, what are you kind of seeing on the ground?

• 4:22Speaker 01

  Yeah, I mean.

• 4:22Speaker 02

  It always works.

• 4:25Speaker 01

  I'll start with the BI one, because that is the one that is the trickiest. We call that a iceberg problem. And it's very easy to build demos.

• 4:34Speaker 02

  Yeah.

• 4:35Speaker 01

  But the real world is incredibly messy, right? You know, you have today, a lot of people look at Spider, for instance, for SQL performance. But Spider is relatively simplistic versus the real world data. You know, we have customers that have tens of thousands of tables, you know, hundreds of thousands of columns with weird names. You know, Rev1, Rev2, what are you gonna use? What is the revenue and so forth? So it's very, we focused a lot on getting the quality right. And turns out, it's not just get an LLM ready and ask the question and LLM does the job for you. Instead, you need to stitch together a series of

• 5:11Speaker 01

  systems so that ultimately you increase the quality. So we built this product called Cortex Analyst, which is a combination of, you know, three, four LLMs working together to answer BI questions. It knows when to ask for clarifications, when not to answer questions that it can't answer. And then, you know, we have these systems that are kind of self-healing, generating SQL, checking the validity of that SQL as well. So it's a pretty complex system. Getting it right is hard. We were able to get the quality in our own efforts up to, you know, 90, 95% range. It's good for many use cases, but not still for all. You know, when you're talking about BI type use cases, you don't want to just put that in front of the CFO and completely trust that the revenue number is always right.

• 5:53Speaker 01

  So we're also building in things like, you know, human in the loop systems to make sure, you know, our customers are building it correctly.

• 6:01Speaker 02

  And is all of it, it's work today? Does it do the SQL generation, but then that can be edited by a human? So it's kind of, or is there, does it do multi-step stuff?

• 6:10Speaker 01

  So it does multi-step stuff and then. Then there is a, there's a feedback loop in, in that customers can basically create what we're calling verified queries. Uh, and then, you know, if we can answer a query, um, using a verified query, we tell, uh, you know, we respond, return that in the, uh, in the API. So that there is increased confidence.

• 6:31Jacob Effron?

  Very cool.

• 6:31Speaker 01

  And then there is a, um, you know, workflow component of it that you can keep adding verified queries.

• 6:37Jacob Effron?

  Where do these models kind of like fall on their face today when the, in the BI space?

• 6:40Speaker 01

  Uh, in the BI space, uh, the hardest thing is getting these complex joins, right? You know, each company has their own, um, uh, you know, data model. And then there's an art to creating these data models that are, that are unique to these companies. Um, and, uh, you know, semantic understanding of that whole space is, um, is, is a big area of, of focus. So, um, these models tend to get, uh, you know, they, they might hallucinate on, um, you know, column names. For instance, they might, uh, get the joins wrong. Sometimes they write queries that are not going to execute and so forth. So there, there are a lot of things that might fall, um, you know, fall flat.

• 7:21Speaker 01

  Uh, so a huge amount of work goes into, how do you go from, you know, a hundred thousand column names to the right ones to provide to the L L M? How do you make sure that when the L L M creates, uh, you know, the response, it is, you know, it is relevant? Uh, when somebody asks a question, how do you know that that can ask, actually be answered? So there are a lot of, you know, gotchas to get right. And, uh, you know, L L Ms will gladly give you an answer even when they should not be. So there is, uh, yeah, there's a lot of work to do there.

• 7:48Jacob Effron?

  Yeah. I mean, it sounds like you've stitched together a lot of different things to make this work end to end. Like what are, you know, I guess, what are some of the things you've learned over time that you've added on like to, to make that kind of stitching better?

• 7:58Speaker 01

  So we've very consciously focused on, um, precision, uh, and, and, and sacrificing on recall. And that, that's a, that's a product choice that, that we are consciously making, you know? We should not be answering all the questions. Instead, we should know when not to answer. We should know when to ask for clarifications, for instance. So that's, that was a, we did not start there. We started a lot broader and yeah, we can do all the joins and then we can, you know, ask us any. Anything and that's actually an okay route for an analyst. You know, be very open ended. You can ask questions of your data. You can explore. And it's okay if you're not right all the time because the analyst knows if you're, yeah, you got the, you know, eighty, ninety percent of it right, I'll correct the rest.

• 8:36Jacob Effron?

  Totally.

• 8:37Speaker 01

  When it comes to business users, you don't have that luxury. So, you know, you need to narrow it down quite substantially so that you increase the quality.

• 8:44Jacob Effron?

  That makes a lot of sense. And then on the data extraction side, I guess, which is another key use case you guys have. Where's it work today? Where do you, where do you see the models kind of fall on their face?

• 8:52Speaker 01

  So there I actually think things are pretty good. The types of interesting use cases we're seeing is, you know, you have a large data set and you want to not just ask questions for each of the cells, but rather all of the cells combined. So, you know, a recent interesting example is we ran a employee survey. And then there's an open ended set of questions in the employee survey. So now you can ask the L M. Of all the questions, just summarize it. What are the categories? Give me examples, you know, give me example quotes and so forth. So something that would have taken a couple of hours of kind of HR digging in can now be a prompt.

• 9:32Speaker 01

  And that just goes through everything and then summarizes it. So I think there's a lot of super interesting opportunities to, you know, make this work well and work easily. So one of the things that we really focus on is how do we make sure whatever we do is from a customer perspective, very, very simple and easy to use. And that's an ethos Snowflake has, you know, make everything super easy. So when it comes to an analysis like this, you know, we've been building, you know, task specific functions that just don't require customers to do a lot of the prompt as well as the data pipeline work. Instead, they can just, you know, use one of those functions off the shelf.

• 10:14Jacob Effron?

  Well, I know we've hit on a bunch, just in the early part of this conversation, but it might be helpful to just take a step back for our listeners and kind of contextualize. You guys do a lot of different AI things at Snowflake. So, you know, maybe just kind of give us the surface area of different AI products at Snowflake today.

• 10:28Speaker 01

  Yeah, we're doing quite a lot. So at the core of what we do is Cortex, which is our inference engine. It's running all the large language models. And then on top of that, we have two products. Like one is Cortex Search. We focused a lot on having a very high quality search. It's really important for RAG. And then the other one is Cortex Analyst, the BI product. So we think of that as talk to your unstructured data and talk to your structured data. For Cortex Search, a huge focus has been on quality. So we have our own embedding model, Arctic Embed, which is the state of the art. We focused on that model to be not only high quality, but also very efficient.

• 11:11Speaker 01

  So Arctic Embed is a quarter the size of OpenAI's embedding model, while still having a higher benchmark score compared to some of the other competition out there.

• 11:22Speaker 02

  And obviously, it seems like smaller models have been a focus. Is that like mostly latency and speed for your customers? Or does cost actually end up being enough of a factor here to really drive it?

• 11:31Speaker 01

  I mean, it is all of it. So cost is a big factor. And we're seeing the market come down quite a lot on cost. And the way the market's coming down on cost is by distilling these large models into smaller and smaller models. And these small models are incredibly effective and efficient. Yeah.

• 11:47Speaker 02

  And on the search side, is that more internal? Like the Snowflake example you said, where I have a bunch of data and I want to search or run some queries over it? Or are people building kind of external user-facing RAG-type applications on it?

• 12:02Speaker 01

  We're seeing three different kinds of applications. We're seeing external RAG applications, definitely. And there, the focus is on, of course, quality, reduced hallucinations. Knowing when to abstain from answering the question there is also really important. Cortex Search is a hybrid search system. It does both vector search as well as lexical keyword search. So it can reduce hallucinations. Another use case that we're seeing, we're seeing a lot of internal use cases, of course. Internal use cases are the most common ones. They are increased productivity, but low risk from a hallucination perspective.

• 12:35Speaker 01

  A third kind of surprising use case is the just regular enterprise search. So enterprise search is back, you know, Glean is doing a phenomenal job. For instance, we're seeing our customers really interested in how do I, you know, upgrade my stack and Cortex search helps there as well.

• 12:52Jacob Effron?

  And I guess like, you know, I feel like such a strong part of the Cortex story is the fact that you sit on top of a lot of interesting data already. You know, I think a lot of folks are thinking through like, what do we have to do with the data that we do have, the data that we get in production, we deploy these models to kind of get the data flywheel going like, what are you learning on the ground in terms of, you know, what makes people effective at doing this?

• 13:11Speaker 01

  A lot of large companies have these governance boards, AI governance boards. So the first step is making sure that the company feels comfortable with the platform of choice for running AI. So, you know, trust, making sure that, you know, the data is secure, the data governance is respected and, you know, the policies are acceptable for the company. So a huge focus actually goes there, ensuring, you know, the company feels comfortable. With Snowflake, running all these AI systems right next to the data, we can answer some of these, you know, questions and provide reassurances for the company. But that's the first step.

• 13:52Speaker 01

  So once the company feels okay, then the question is, okay, so I have all my data here. How do I build a high quality, a chatbot or something else? And then how do I have this, you know, data flywheel?

• 14:10Speaker 01

  We've recently acquired a company called Truera.

• 14:14Jacob Effron?

  Yeah.

• 14:14Speaker 01

  And.

• 14:15Jacob Effron?

  You guys done a lot of acquisitions, yeah.

• 14:17Speaker 01

  That's right, yeah. So Truera is a, has this product, open source called TrueLens. TrueLens is an observability LLM evaluation platform. So we're finding that to be great for our customers. There's a lot of, you know. You build a system and then you, you know, evaluation is a problem. You know, you ask a couple of questions and then you don't know if this is good, this is bad. Is it, is it better than another model that I should, I should be using perhaps? When we bring a full stack solution from an evaluation perspective, like that, that eases, you know, concerns and then you can run evaluation at scale with L L Ms as judges. So, so that, that usually is a core part of how our customers are starting to think about production systems.

• 15:02Speaker 01

  You know, so far up until recently, everything was just P O Cs and, and there you don't need to think about any of them. Are we seeing a transition yet? We're, we're eagerly awaiting. We are seeing transitions. We're seeing transitions slowly, right? You know, for internal use cases, for productivity use cases, for analysis, we're definitely seeing transition from P O Cs to production. For external use cases, I think there's still more, you know, confidence that needs to be there before, you know, large scale production systems get built.

• 15:29Speaker 02

  What, what do you think the timeline is for like widespread external deployments? And then what like, what would need to happen to get there? Is it just, you know, people have to spend a little more time with technology and get comfortable or there's some actual changes in capabilities or in guardrails that, that need to happen?

• 15:43Speaker 01

  I think it's both. I think people are becoming more comfortable with the technology. I think there is product innovation needed as well to get the end user to be comfortable with the answer not always being right possibly and then, and then kind of mechanisms to check whether this is right or wrong. So there's, I think there's an opportunity there. And then of course, technology keeps getting better, but it's not quite there yet. There are still risks for hallucination, which at some, in some cases, especially in regulated industries can be quite catastrophic.

• 16:17Speaker 02

  I would imagine on the, the privacy and security side, you guys have a huge advantage, right? Cause you're already storing everyone's data. So then it's a good story to go to everyone and say, hey, you can run, you know, the, you can run our model right next to your data. Does all of the inference happen within Snowflake Cloud? Do you allow people access to models outside of that or, and then how do you think about that going forward?

• 16:39Speaker 01

  Yes, it's a huge value proposition to run everything right next to the data inside Snowflake. So all of our models are running inside Snowflake right now. For customers who want to use external models, we also have solutions, but that's, that's not default. So they'll up, you know, they can configure it so that they can use external models if they choose so. The Governance of the data plays a huge role. So at Snowflake, governance is built from the ground up so that our customers are setting up very granular access controls. The example I'd like to use is, let's say that a company builds a HR chatbot. Then you ask a question, who got promoted last week?

• 17:24Speaker 01

  So first of all, depending on who asks that question, you need to give very different answers. And then there is very little room for hallucination, beta leakage, right? So that points to the importance of getting the governance right. And we're seeing having that be a core part of the platform from the ground up is a huge advantage.

• 17:45Speaker 02

  Yeah, that's a much better example than the one I was thinking of, which I don't want Jacob to access, my SQL query, because, you know, he's going to mess it up with whatever he's doing. But yeah, that makes a lot of sense on the RAG application side.

• 17:58Jacob Effron?

  How do you actually build that, like, access control into a model?

• 18:02Speaker 01

  So first of all, even before AI came into the picture, we had granular access controls required. At Snowflake, it's a very granular system. You can basically grant any object, a database object access.

• 18:18Jacob Effron?

  I guess what new things do you have to do when an LLM comes into the picture?

• 18:21Speaker 01

  So I'll use search as the example. So for instance, we had to build access controls very deeply into search. So when you search for a document, that filtering needs to be there so that you can only, the search system only gives you access to the documents that you have access to. So for AIs, for search, that's a big part. And then we rely on already existing access controls for the data part. So when you ask a question, if you can't access the data, then you don't get that data. But having that be already built in is a huge advantage. A lot of our customers spent a couple of years getting the governance of their data in place, so now they can benefit from that and then build AI on top of it.

• 19:05Jacob Effron?

  Makes sense, and good luck to the next person trying to recreate that.

• 19:07Speaker 02

  Yeah.

• 19:09Jacob Effron?

  I guess, you know, obviously a big part of you're probably working hand in hand with these enterprises who are building these applications. What kind of advice are you giving them or what are you noticing on like when it makes sense to just take a model off the shelf, fine tune it a bit, maybe train your own model from scratch, like what are you kind of seeing?

• 19:24Speaker 01

  I mean, we're recommending starting with the large models and then building, you know, POCs. And usually a large model plus a RAG solution, you know, gets you quite far. And then once everything is in place, then the optimization starts for the production system. Then you can start fine tuning a smaller model that can give you kind of scale advantages from a latency perspective, from a cost perspective. For companies that have a large amount of unique data set, then custom models also makes sense. Having built Arctic in a very efficient way, now it doesn't have to be outrageous to build your own very specific custom model for your own needs.

• 20:09Speaker 01

  So for some, you know, small number of customers who are interested in kind of building their own models, pre-trained models, we also work with them to do that.

• 20:16Jacob Effron?

  What are some of the like examples that jump out to you as like, yeah, it probably makes sense to build your own pre-trained model there?

• 20:20Speaker 01

  Usually in regulated industries, this becomes, this becomes a desired thing. You know, our customers will want to have full control over what data goes into the model, for instance, as well as, you know, they have, you know, healthcare is a good example, you know, unique language that they use. So again, the French on it isn't necessary, but understanding the business is important.

• 20:44Jacob Effron?

  And for those, are you, are, are they training their own model within Snowflake? Is, are you guys supporting training today? Or it seems like kind of inference, at least a lot of the messaging has been more on like, you can use any of the models and we focus on inference, but like, how do you think about the?

• 20:57Speaker 01

  From a volume perspective, a huge part is inference. That's, that's, especially as these models get very, very capable, I think that that works for most customers.

• 21:07Jacob Effron?

  Totally.

• 21:08Speaker 01

  Yeah. And, and for those kind of small number of customers, so fine tuning is part of it. So you can fine tune and then use that fine tuned model. We make that very simple to do as well. And then for, as I said, small number of customers, we'll work with them to train and then, and then use that trained model inside Snowflake.

• 21:26Speaker 02

  Very cool.

• 21:26Jacob Effron?

  Yeah, I mean, you said it must be an interesting place to sit where you see what people are selecting on the model side, like any patterns you're noticing, you obviously support a bunch of them.

• 21:35Speaker 01

  Yeah, we support a bunch of them and maybe not surprisingly, but somewhat surprisingly, customers start selecting the models based on brand versus capability, right? You know, like we have very capable models that aren't used as much because the brand recognition on those models aren't there, even though the models are super capable. So that is one relatively surprising thing.

• 22:01Speaker 02

  Engineers are human, I guess.

• 22:02Speaker 01

  Yeah.

• 22:06Speaker 01

  Customers are asking for assistance on how do I select these models, of course, and we have started kind of facilitating that. I think the LLM evaluation framework being easily accessible will help in that direction as well.

• 22:22Speaker 02

  What does it take to support the new models? You know, Mistral came out with a new model last week and Llama, I think, the week before that. Are those easy to incorporate into Snowflake? Do you guys need to update your infrastructure to support those?

• 22:36Speaker 01

  So we're very fortunate to have the VLLM team be part of Snowflake. Well, the founders of the VLLM team are now in Snowflake. So we need to make sure that the inference stack supports the models. And because VLM is relatively popular, you know, when the models come out, they come out with support for that. But last week's, last week, previous week, you know, Llama 3.1, 405B model came out. And there we built a series of optimizations into VLLM so that that very large model can be run with a lot of efficiency and multi-node inference. For instance, we started supporting a lot of.

• 23:19Speaker 01

  Optimizations there, both on inference as well as fine tuning. And then we upstream those upgrades to VLM, working with the VLM team. So overall, usually we, the running new models is relatively straightforward. But of course, when you know, large models, new architectures come up, we need to upgrade our stack.

• 23:44Speaker 02

  Did the VLM folks, did they join as employees or is it a project out of Berkeley, right?

• 23:49Speaker 01

  It's a project out of Berkeley, so we have a couple of folks who are employees, yes.

• 23:53Speaker 02

  That's awesome. Well, congrats. I'm sure there's a lot of big companies chasing the hot AI talent, so.

• 23:59Speaker 01

  Yeah, they're great.

• 24:00Speaker 02

  Yeah.

• 24:01Jacob Effron?

  Do you think that 405b model will be widely used or is it like almost too big for a lot of these end use cases?

• 24:06Speaker 01

  It is a very capable model. It is a large model, so it's slower. However, it's from a cost perspective, it is still not unreasonable. So we're seeing usage of it. We're also seeing usage of the model for, you know, fine tuning purposes, distilling other models purposes. It's a very, very capable model and broadly available. You can use it for synthetic data generation. You can use it for kind of distilling your model. So overall, having that more flexibility in the terms to be able to use that model for other purposes is a huge advantage for the ecosystem, I think.

• 24:45Speaker 02

  Yeah, I thought that was the coolest part, that you could do model distillation with it and you could have this big model. I would imagine even for things like SQL generation, you could say, hey, big model, generate some SQL for me and then maybe have that train a smaller model or fine tune something else. So, super cool.

• 25:01Jacob Effron?

  Like, it seems like today, there's, obviously enterprises are kind of just searching for use cases where this stuff really works. And it seems like you found a few and I'm sure people are looking for more. On the cost side, are there, are you finding that there's actually places where enterprises have identified a good use case, but it's like too expensive at scale? Are we still mostly in use case discovery mode versus having found those use cases but still waiting for the cost to come down?

• 25:23Speaker 01

  I don't think, I'm not seeing a lot of, hey, this is too expensive for me to turn into production yet. But as I said, In the P O C phase, everyone is using these large models and them doing the proof of concept. And because a lot of the current use cases are internal focused, the volumes aren't massive to be super worried about costs. And the costs have been coming down very rapidly. Yeah. Haven't seen, I know everyone is very aware that these models cost a lot, but because the number of massive production use cases have been relatively low, it hasn't, I haven't seen it be a huge blocker. I don't think people are at a point where, hey, I want to deploy this very broadly, but cost is an issue.

• 26:08Jacob Effron?

  What's the future have in store for like Arctic L M? Obviously, you've got other open source models that are investing a ton in compute and

• 26:13Speaker 01

  Yes.

• 26:14Jacob Effron?

  like, you know, do you want to keep up with the Joneses with that? Or like, you know, was this enough to kind of get it off the ground?

• 26:20Speaker 01

  Yeah, our goal is not to build a fully general purpose model that competes with G P T five. That's not the goal. It is to focus on snowflake customers' specific needs. So we are focusing on kind of sequel generation. We're focusing on kind of RAG quality. These are areas that we're doubling down and Arctic is helping there.

• 26:40Jacob Effron?

  I think we are really interested, you've talked a little bit about like Cortex Guard, and we've kind of alluded to some of this, like the stuff that comes around the deployment of these models on the security side. Wondering, one, like just kind of some of the emerging patterns you're seeing around privacy, you know, protecting against prompt injections, all sorts of things that, you know, enterprises are thinking through.

• 26:56Speaker 01

  Yeah, it's super interesting. I was actually just today having a conversation about open source models. People are wondering, is, you know, are the guardrails on open source models, you know, better or worse than the proprietary models? And then turns out, you know, Meta has done a great job in aligning Llama, but then providing something like Cortex Guard using Llama Guard underneath as a additional guardrail is very much resonating with the market. Because this is uncharted territory for many businesses, you know, the responses are completely

• 27:35Speaker 01

  Unscripted, so they want reassurances, they want guardrails on what these models can do. Many companies want to make sure that not only from a policy perspective, everything is lined up, but also from a kind of brand perspective, everything is lined up. So I think there's a lot of additional guardrails that are being built into these models.

• 27:55Speaker 02

  Yeah. What are we like missing today? I mean, of all the kind of enterprise concerns you had, I'm sure some of them you're like, no, no, we're good there, we can solve that. And other ones you're like, yeah, that is a legitimate concern.

• 28:07Speaker 01

  I mean, we've touched upon a couple of them. Hallucinations are still a concern and measurement is still not mature for many companies. Quality, when it comes to more complex scenarios like Texas sequel is still a concern. And then, of course, everyone's hot topic, the agentic systems are coming, and this is just uncharted new territory. It's super exciting. There's a lot of potential, but it's just getting started.

• 28:37Speaker 02

  We'd love to talk a little bit. You mentioned the True Era acquisition around evaluation. More broadly, like, how do you think about acquiring companies in the AI world versus partnering with them? And then, I'm also curious with Neva, like, how you've incorporated that into the product?

• 28:57Speaker 01

  Yeah, I'll start with the second one. Neva acquisition was instrumental for-

• 29:03Speaker 02

  The guy, you see, yeah.

• 29:03Speaker 01

  Yeah. The guy, our CEO. And-

• 29:07Speaker 02

  Do you have to say it's instrumental because he is the CEO now?

• 29:12Speaker 01

  It kick-started a lot of our efforts. Cortex Search, for instance, uses Neva technology. The underlying embedding model uses Neva technology. A lot of the folks that we brought in, in addition to the CEO, are core valued partners of engineers in the team. So yeah, Neva technology is a big part of our offering. To go back to the first part of the question.

• 29:38Speaker 01

  We are partnering with a lot of model providers today. For instance, Mistral, Rekha, AI21. These are good examples. We're investing, investors in some of them as well. In addition, we're partnering with companies like Lending AI that have not just models, but also full end-to-end solutions. And we're bringing these partners as applications on top of Snowflake. So that, you know, Snowflake customers who want to make sure that their data is still secure and they don't want to take their data out of Snowflake, they can just still use these very capable end-to-end products within the Snowflake boundary. So, you know, our partners can run applications inside Snowflake that directly can access Snowflake data.

• 30:20Jacob Effron?

  I'm also curious like how you're using LLMs and AI internally for your teams at Snowflake.

• 30:24Speaker 01

  We are using LLMs in a series of different places. Our sales teams are using it to understand from, from sales conversations, getting summaries of kind of why are we winning? Why are we losing? That's a very internal use case. That's interesting. We use, we have, you know, employee assistants that are built inside the company so that you can ask questions of internal data documents. We have use cases that are, you know, documentation oriented as well, just, you know, regular, you know, chatbot type use cases.

• 31:04Speaker 01

  And then AI being a core enabler of technology now, a lot of teams are taking AI and then incorporating it into, into the full stack, you know, for anywhere from, you know, optimization of, of our SQL engine to our marketplace, bringing AI and AI capabilities into the marketplace. So we're using it both for kind of internal purposes as well as product development.

• 31:31Jacob Effron?

  One thing I know our listeners will be interested in. So I'll ask about is just how you think about like comparing, contrasting the Snowflake AI strategy to Databricks and what they're doing.

• 31:40Speaker 01

  I'll say at the core of what differentiates Snowflake is the ease of use. We think of ourselves as a single product and everything works well with each other. And that makes it difficult to develop, but very delightful to use the product. So AI is no different. You know, we've made AI incredibly integrated and easy to use.

• 32:05Speaker 01

  And then on top of that, because of some of our acquisitions like Neva and our investments, when it comes to quality of these systems, either text-to-SQL system with Cortex Analyst, as well as Cortex Search from a RAG perspective, we have very high-quality systems that our customers can use very easily. I'd say those are main ones. And of course, governance is an important part. Both companies are investing in governance. And then we have a pretty simple-to-use tools that we've been investing over the long run for governance.

• 32:41Jacob Effron?

  Do you think the vision of the end-to-end platform is kind of the same? Or actually, is there slightly different approaches in how you're each going about it?

• 32:48Speaker 01

  Both companies are approaching it from slightly different angles. For instance, when Cortex launched, we started showing Cortex with SQL. I think we are one of the first companies that integrates AI into SQL as the first thing. So depending on where the customers are, where the audiences, the products are slightly different.

• 33:07Jacob Effron?

  And I guess I'm curious because it feels like you're building such a comprehensive end-to-end offering in Cortex. You know, are there opportunities for, like, as we think about, we see infrastructure companies pop up all the time and there's folks doing like bits and pieces of what you're, you know, what part of Cortex is or what future parts of Cortex. Do you think there are opportunities for startups like within the AI infrastructure stack? Or is the kind of future converging on these like end-to-end platforms that kind of consume almost all these different parts of the stack?

• 33:33Speaker 01

  Yes, absolutely. There are opportunities. I think the market is massive and it's only going to grow. And the opportunities for startup at various aspects of the stack will be there. I'm already seeing really awesome startups in the infrastructure parts of it. We're talking to them. We're working with them as well. There is value to end-to-end from a end-customer perspective. We talked about this AI governance boards choosing the platform. However,

• 34:07Speaker 01

  They're still, we're seeing a lot of innovation in the whole stack. So there's always opportunities for startups.

• 34:14Speaker 02

  You guys have obviously shipped so much on the AI front already. What are the big pieces of functionality or products that you're most excited about? Or kind of the next iteration of them?

• 34:24Speaker 01

  I've been talking about just using natural language to talk to data broadly, talking to unstructured data, talking to structured data. We are going to be enabling agents as well to plug into the system. I'm really excited about using the full ecosystem from a customer's perspective. And enabling a lot of what has resonated for our customers on Snowflake is these things like making it very, very easy to share data. So because it's such a huge data platform, data cloud, it's easy, for instance, for an asset manager to have access to the financial ecosystem, SMP or FactSet data. And as all these companies turn their data into AI-ready data,

• 35:08Speaker 01

  now a company like Morgan Stanley can easily build a chatbot that plugs into all of this. So bringing the whole ecosystem together, building on top of the data that's already built in is a huge opportunity, making that not only seamless, but also rich as an ecosystem play.

• 35:29Speaker 02

  Makes sense. Yeah, it's fascinating because I think

• 35:33Speaker 02

  there's very few companies that have the opportunity to do that because they, you know, it's hard to get the data sharing working. But since it's all under Snowflake, then you can say, hey, it's already all on Snowflake. And then we can build a model on top of it. So.

• 35:45Speaker 01

  That's right. Exactly. So it comes with a lot of governance and security guarantees.

• 35:49Speaker 02

  Yeah. What are the categories of like AI infrastructure that you find interesting or looking at the ecosystem and kind of excited about?

• 35:56Speaker 01

  I mean, the. Inference stack is still developing so fast and you know each improvement results in you know reduction of costs and that's that's really welcome. So I think there's a lot of innovations still happening there. So I'm super excited about it because it immediately translates to cost. Yeah.

• 36:13Jacob Effron?

  Do you think there's like long term businesses to be built on the inference side?

• 36:16Speaker 01

  It's a hard question.

• 36:18Jacob Effron?

  For us to.

• 36:19Speaker 01

  There are a lot of companies clearly in the playing in the space. The costs are coming down dramatically, but I think there's a scale play. So I think there's businesses to be made, but I don't think it's going to be many successful ones.

• 36:32Jacob Effron?

  I feel like, you know, in addition to your fascinating role at Snowflake, you've done a lot of cool things in your career before. And one of them, I think especially relevant to this moment is you ran Google AI on Android, right?

• 36:42Speaker 01

  Yes, right. Google Assistant.

• 36:43Jacob Effron?

  Yeah. And so I feel like, you know, your role is fascinating here. I'm sure if you were still doing that, that would be really interesting. Like, how do you think about what's going to happen in that world? And what would you be doing if you were still over there?

• 36:54Speaker 01

  I, so a long time ago now, I started this product called Google Now. So Google Now was a very early, you know, AI company at the AI product at the time. And then that morphed into Google Assistant. And I used to joke, after a decade of working on this, it was it Google Assistant essentially used to be a, you know, series of use cases that are kind of handcrafted, right? And then LLMs came and now it's a little more general purpose. And I thought, okay, finally this thing works. It, you know, it and it's super exciting that it actually, you know, going from handful of curated use cases to something that that is not going to fall, you know, through the cracks that are naturally going

• 37:38Speaker 01

  to show up in a kind of use case by use case scenario. I am very excited about that world. Like, I think what has captured the imagination with all these assistants is finally starting to show signs of utility. And then, you know, ChatGPT is doing a phenomenal job there. But as all these, you know, so far, these assistants have taken action. You know, you can set alarms, you can set timers, and you can start playing music. Now with LLMs, we're starting to get there. As these LLMs are able to call all these APIs to take action, I think it becomes incredibly powerful.

• 38:12Jacob Effron?

  What do you, what do you think those like assistants will look like in five, 10 years?

• 38:16Speaker 01

  You know, her movie has captured the imagination, right? So and

• 38:19Jacob Effron?

  I think that comes up in about in half the episodes we do.

• 38:22Speaker 01

  There is that part.

• 38:22Jacob Effron?

  A crushing movie.

• 38:23Speaker 01

  I actually really like, not necessarily

• 38:27Speaker 01

  The her, I like her vision too. But this idea that you have many small, specific agents that you've set using natural language that are just going off and doing things on your behalf. And I like that idea quite a lot because when you look at anyone, everyone has very unique, specific things that they're trying to do. And that is why, like being able to set alarms applies to everyone. Sure, sounds good. But then when I'm doing something specific at work and I want this AI to be checking whatever on this website on a regular basis to see if something becomes available, that is maybe unique to me. So being able to create these assistants, these agents that are very easily created.

• 39:11Speaker 01

  I think that there's an opportunity here that's that I get very excited about.

• 39:15Jacob Effron?

  Yeah. Are you seeing any of your customers like begin to implement agents or is it still early, early days of that?

• 39:20Speaker 01

  It is early, but our customers are starting to do this with something like BI, for instance. It's an agentic system. You need to figure out, okay, when am I going to generate the sequel? When am I going to evaluate the sequel? I'm going to run it, get the data and then reason about it. So that is one very specific BI use case that that is agentic. But but it's it's just starting to emerge.

• 39:45Speaker 02

  Yeah, I've been very impressed with ChatGPT's data analysis, which is kind of a BI light type workflow, but you can give it screenshot of Excel and then it will, you know, go back and say, hey, we tried this Python transformation didn't work. Okay, now we go update it. And you're kind of following along, but you're not doing that much. And it's, I don't know, an interesting glimpse into the future of BI and data analysis.

• 40:10Jacob Effron?

  We all thought Pat was doing a lot more work, but it turned out.

• 40:12Speaker 02

  Exactly, yeah. I just I've just been using ChatGPT for a few months longer than than some folks. So that's my secret.

• 40:19Speaker 01

  And I use LLMs to write very complex Excel formulas, for instance, and then brag about it. Look, look at this.

• 40:28Jacob Effron?

  Yeah, I guess it really, it definitely equalizes a lot of that. We always like to end our interviews with a quick fire round where we we get your thoughts on a few things. Maybe to kick it off, would love one thing that's overhyped and one thing that's underhyped in the AI world today.

• 40:42Speaker 01

  I think underhyped is evaluation.

• 40:47Speaker 01

  Overhyped, based on where we are today, is.

• 40:53Speaker 01

  I was gonna say agents, just because it's just starting. There's a lot of hype about it, but I think it will match that hype. But we're in the early innings.

• 41:02Speaker 02

  Maybe agents in the short term.

• 41:03Jacob Effron?

  I always joke that I feel like we've gone through a full hype cycle already with agents. There was like this peak of excitement and then trough of disillusionment and now we're like kind of

• 41:10Speaker 02

  Now we're climbing back up.

• 41:13Speaker 02

  What's been the biggest surprise in building Snowflake AI? Maybe one thing you thought that would work that didn't, or one thing that you thought wouldn't, but ended up working in the end?

• 41:21Speaker 01

  The text to see cool part.

• 41:25Speaker 01

  We knew how complex and challenging it would be. And defining the exact problem to solve has been hugely, hugely unsurprisingly, but it's a core part of the solution. So to me, that was perhaps not surprising, but really it's a process that you have to go through. And then going through that process was very informative.

• 41:49Speaker 02

  Yeah, we've actually heard that from a couple of folks, which is kind of, you know, the classic machine learning things of defining the problem and what good looks like still very much ring true in LLM world.

• 42:01Speaker 01

  Yeah, exactly.

• 42:02Speaker 02

  Yeah. People always like to do the closed source versus open source comparison. And obviously, it's not one or the other. But curious like how ubiquitous you think, you know, open source models will be versus kind of these state of the art closed models in the future?

• 42:16Speaker 01

  I mean, companies like Meta especially, but also Mistral have been hugely

• 42:23Speaker 01

  influential in communicating that it's not only a, an industry that only has two players. So having all these open source models and then flexibility for the whole ecosystem to emerge around it and then to take these models and then do awesome things with it has been really, really welcome. So I think there's a lot of gratitude towards, you know, Meta especially. So that's, that's great.

• 42:49Speaker 02

  Which AI startup outside of your space are you most excited about right now?

• 42:53Speaker 01

  I'm still very excited about model startups. When I look at Mistral, for instance, it's a very small team of very capable folks building amazing models and then moving super fast. And they've also done a great job of bringing these models and then creating awareness for them. So I'm super excited about Mistral.

• 43:18Jacob Effron?

  If you were to go build an AI application like tomorrow, what would you go build?

• 43:23Speaker 01

  So I, you know, my background is all in assistants and I'm super excited about how this whole space will shake out, so I'll probably do something in that space.

• 43:32Jacob Effron?

  Have you seen anything, like anything on the startup side that you found particularly interesting in the assistants world?

• 43:37Speaker 01

  No, I've been paying attention to a couple, but nothing, nothing yet. The tricky thing about the assistant space is it wants to be a platform, but as a startup, you cannot start as a platform. So yeah, there's a tricky chicken, chicken and egg there.

• 43:52Jacob Effron?

  Is there a place for startups to start there?

• 43:54Speaker 01

  I think the only place for a startup to start is to nail a couple of really deep use cases versus I'm going to just do many, many small ones and then hopefully it'll work. So identifying a couple of deep use cases is really important. Devon, for instance, has gone, okay, I'm going to go build an engineer and then have that, have that be a thing. I'm seeing startups that do something similar. Here's a salesperson, here's a SDR and so forth. So that's, that's an interesting thing. Rather than selling quotas in APIs, you're, you're selling, okay, do you want one Devon or two Devens, which is interesting as a concept.

• 44:35Jacob Effron?

  We're waiting for the here's a podcast host and then Pat and I can just go.

• 44:38Speaker 02

  Yeah, I personally, I'm, I'm very excited about travel on the consumer side. I think that would be a make for some killer, killer agents, I guess. So the, what company besides Snowflake do you think it would be most interesting to run AI right now?

• 44:54Speaker 01

  Platform, I find platform companies very interesting just because of my background. So having worked at Google for a long time and then.

• 45:03Speaker 01

  Platform has an opportunity. These capabilities are incredibly exciting, but they are waiting to be to become platforms. They are not yet platforms. So I think any of the main players who are kind of waiting to build platforms are super interesting from a large company perspective. They are not startups yet. From a startup perspective, I would, you know, the application space is just forming. And I think over the next two years, when I talk to customers, we're now transitioning from here are a bunch of building blocks, go build whatever you want to. No, I actually want the product end to end that works. And then these areas are becoming more and more clear. You know, customer service is the most obvious one that was there, but increasingly there are other ones.

• 45:48Speaker 01

  So customers want end to end solutions. So I think application space is going to be the next. Hopefully we'll see a lot of flowers bloom in the next, in the next two years.

• 45:56Jacob Effron?

  Yeah. Well, this has been a fascinating conversation. I'm sure there's a lot of threads folks will want to pull on. So I'm going to leave the last word to you. Like where could folks go to learn more about you, about all kind of all the stuff Snowflake's doing in AI? Feel free to point people wherever, wherever you'd like.

• 46:08Speaker 01

  For Snowflake, I think the, you know, our, you know, Snowflake AI website is the best place. And we've just recently did our summit event and there are a lot of awesome videos on YouTube that talk about all the things that we do on Snowflake.

• 46:25Jacob Effron?

  Awesome. Thanks so much for joining us. Thanks so much. This is a really fun episode.

• 46:28Speaker 01

  Of course, no, thanks for having me.