Ep 15: Snorkel AI CEO Alex Ratner on What's Needed for Wider-Spread Enterprise AI Adoption

Jacob sits down with Alex to discuss how Snorkel grew from an open-source project in a Stanford AI lab to a $1B company. Alex shares his thoughts on why data development is at the heart of AI development, why enterprises are slow to deploy LLM applications, and the importance of academia in the future of AI development.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast, and I'm Jacob Ephraim, one of your co-hosts. Today, we had a fascinating conversation with Alex Ratner, co-founder and CEO of Snorkel AI. Alex started Snorkel as a PhD at Stanford to automate data labeling, and they've since expanded into helping companies fine-tune, pre-train, and align their models. The company is recently valued at over $1 billion, and Alex and I hit on a bunch of topics, including how companies are actually using generative AI today.

• 0:21Speaker 01

  There's all this chat GPT excitement coming down from the C-suite, and then shrewd data science, AI, ML leaders are going to actually divert that to more classical workloads that already have proven evaluation metrics, proven business value.

• 0:32Jacob Effron?

  Model evaluation.

• 0:33Speaker 01

  The bigger thing comes down to just making sure that you have a fast way to build custom benchmarks that reflects the actual business or use case needs.

• 0:41Jacob Effron?

  And how Alex thinks academia will play a role in AI going forward.

• 0:44Speaker 01

  There's all the formalization, the deeper understanding, the creative thinking that has to happen on the academic side. So there's plenty of room left there, would be my perspective.

• 0:51Jacob Effron?

  I think you'll really enjoy the conversation.

• 0:53Speaker 01

  Jacob, thank you so much for having me on the podcast. I really appreciate it.

• 0:56Jacob Effron?

  It's a real pleasure. And I guess to start would just be curious, you know, I'm sure a lot of folks would love to know how you kind of went from your PhD program and kind of working this stuff in academia to actually deciding to start Snorkel as a company.

• 1:09Speaker 01

  That's a great question. And before jumping in, I just want to note that it's very amusing to me that the name of your podcast, Unsupervised Learning, and my whole area is supervised or weekly supervised learning.

• 1:20Jacob Effron?

  We're an equal opportunity, you know, we can't discriminate too much here.

• 1:24Speaker 01

  Fair enough. I feel appropriately welcomed. I was always interested in having an applied focus, and part of that was what led me to work with now my co-founder, my PhD advisor, Chris Ray, who's still over at Stanford, and his lab, even back when I joined around 2014, was very applied and focused. There was a kind of design mentality and philosophy that I became a big proponent of as well, and it was there when I joined. You start with an inkling of a theoretical and a conceptual idea. Chris has a very deep theory background. The lab had a very deep kind of statistical learning and actually a broader theory background, and I certainly got run through that wringer. But you start with a little bit of a high-level concept, a bit of theory maybe,

• 2:07Speaker 01

  but then you apply it via real software to facts with end-to-end collaboration with real users

• 2:15Speaker 01

  before you ever kind of go write the fancy ML paper, because too many of the latter just get thrown over the wall, and then unfortunately, no one cares about them, and they may, you know... Not actually be addressing things real. So we always had that that spirit of being applied and mixing both more formal theoretical stuff and applied stuff. And to that end, we, we, you know, we released dozens and dozens of open source repos. We did a lot of collaborations. For many years, though, the goal was not at all to make a company out of it. We were, you know, very, uh, we put everything out in the open, both academic papers and open source. We actually got approached pretty early on the project, maybe like 2016, 2017, with, you know, multiple unsolicited bids for funding, which, when you're on a,

• 2:56Speaker 01

  you know, a grad student salary is, you know, eye-popping numbers, uh, that, you know, I guess now I would view as, you know, small pre-seed funding offers, I guess, but because I've been corrupted by capitalism. But we were pretty diehard of just, you know, the kind of, um, community we had and, and the newness of the ideas, which we'll talk about in a bit, that we were studying, really, all this programmatic labeling, but more broadly, this idea that the data development was actually at the heart of AI development, potentially, not just some upstream janitorial task that, uh, you know, you, you, you ignored and, and, and treated like it was, uh, like it was janitorial. We kind of, you know, kept our heads down and, and ignored any of the kind of, uh, you know, siren calls of the outside

• 3:40Speaker 01

  world. It all started to change a bit when we were doing some of our first big deployments. We did, we published a paper with Google. We did a bunch of big deployments there in, in very high-impact scenarios. We did a bunch of stuff with, with Intel, with the Stanford Hospital, a bunch of others. We began to see how much work was, it sounds trite to say it this way, but we got to, we went, we began to see firsthand how much work, even with the maturity of an infrastructure and environment like a Google, how much work was required to go from this concept of programmatic labeling and data-centric, I will talk more about in a second, to actual product. And then our users, because our open source community started, you know, getting a lot more popular than we'd ever expected. We had weekly office hours. Kind of all of the issues that started bubbling

• 4:21Speaker 01

  up the stack rank were, could you just make a damn UI? Can you, like, take care of first mile and last mile stuff? Can you, you know, can you give us, you know, a workflow tooling for how to go through this, this, this data-centric workflow, et cetera? So that was kind of what shifted things in our mind of saying, look, the biggest way to drive it forward is, there's always going to be an academic component. There, there still is. You know, the extended Snorkel team published dozens of papers so far this year. So it still continues. But the biggest issue is sort of became, okay, we got to prioritize this. And then, once it comes time to prioritize something, that's not, that's not the appropriate use of, of, you know, grad student hours, right? It's not the optimal or appropriate for, for grad students. So that was when we started thinking about spinning out.

• 5:04Jacob Effron?

  No, I mean, it's, it's super interesting. I think probably a good segue to just talk a little bit about what kind of Snorkel was originally as a company. And then kind of also, you know, the evolution that you guys have had over, over the last, you know, years.

• 5:16Speaker 01

  Yeah, so I mean, maybe I'll start at the, the, um, the top level kind of umbrella goal of the project and then of the company, which, which gets to our long-term company vision as well. And it's what we call data-centric AI development. And I'll talk a little bit about that arc at a very high level. So when we were starting, data center development, for those that are, I'm guessing, most people who are listening to a podcast called unsupervised learning are a little bit, a little bit, you know, deep in the weeds. So can probably appreciate that when we say AI today, we usually mean machine learning. And machine learning, of course, literally by definition is learning to automate things, learning to do AI from data. So the term data-centric AI might sound a little tautological because data is at the heart of how we do AI.

• 6:01Speaker 01

  But really, we meant it, you know, initially. And now, we mean it from the standpoint of where does your workflow center around? Is your workflow very model-centric? Or we often used to call this the Kaggle-centric kind of era of machine learning where, you know, your journey as an ML engineer or data scientist starts by, you know, clicking download on a nicely curated labeled data set from Kaggle or getting a nice, you know, getting an Excel spreadsheet that's been all carefully selected and curated from some of your line of business partners in, in an industry. And that data is kind of treated as a fixed input and you iterate on the models, the algorithms. That's how data science has been taught. That's how data science is still structured in many enterprises, like down to the how, how the, the, you know, the, the org structure is, is, is printed out.

• 6:44Speaker 01

  And our original question was, what if a lot of that machine learning loop, the algorithms, the models became automated and standardized?

• 6:54Speaker 01

  Basically push button. And that was almost, you know, heresy back then because there, I mean, especially in academia where, you know, the ad industry or their entire fields of people who that's their job. And that should describes most of the, you know, last five, 10 years of ML tooling. But, you know, that's our job in academia is to ask, you know, a little bit more interesting questions. It was also just surprisingly boring to some people on machine learning. I remember being given earnest advice by someone saying, why would you like? You sure it's a good career move to do a project on data and machine learning? Like that's janitorial work you're never going to publish in a machine learning conference. Uh, but luckily I had the, the, the, the support and, and, you know, uh, urgings of, of my advisor at the time, Chris. So we went ahead.

• 7:38Speaker 01

  We said, what if all the ML stuff, all the stuff that, you know, people are paid millions, billions of dollars to do and building entire stacks and companies? Just turns into, you know, kind of a commodity button push. What happens then? Well, the natural bet would be if you're going to get that stuff to be automated and push button and standardized. It's going to come out a trade off. You're going to have to make bigger models that are more data hungry. They're also going to be more black box by definition. So the data is not just going to be something that you need more of to feed them. It's also going to become the effective interface. So we said, okay, what if that happens? And by the way, that's what's happening. That's what's. Yeah, I was about to say it's pretty, pretty prescient. Yeah. Again, it's not just us who are saying that, but, but, but that's, that's what's happened, right? It was less sure at the time. So I think when we started, you know, experimenting, I said, what if, what if we flip it entirely?

• 8:23Speaker 01

  So what if the model's fixed as an input and the data is the thing they're iterating on? You're labeling, you know, again, now labeling, you could be writing prompts. I would consider that data center iteration maybe all self-serving, but that's certainly how we position with our tooling today. And, and add, and add actual value.

• 8:42Speaker 01

  That data development, all that janitorial work, what if that's the center of the workflow? Basically, what if data scientists have to become, you know, data engineers? But again, very, you know, non-standard idea at the time. When we started working on this, the first big wave that's kind of people have been aware of in the last like 10 years. So that was coming out was, was deep learning. So that was the kind of first hype cycle in, in recent memory around AI. And that really did push the models to become much bigger, much more black box, much more standardized, much more data hungry. Put a lot of pressure on the data labeling. So that actually started kind of lifting the project up and, um, That ultimately is kind of the current that led us to spit out. But when we started the company around 2019, 2020,

• 9:25Speaker 01

  you know, the pitch was data is now a thing that's also a pain point. The really exciting, interesting thing that's happened in the next, you know, last couple of years has been data is not just a problem anymore. It's arguably the only thing that you can deal with. Because when deep learning started coming out, people, there's still a lot of tinkering and tuning, and you tweak these hyperparameters, you pick this model, that's how you get it to work. Now, who's going to claim that you can go and tweak some of the parameters of a 500 billion parameter model? Because it's messing up on a certain subsegment of patients or a certain type of object in a satellite image or a certain type of document that the model has to parse.

• 10:07Speaker 01

  If you believe that the foundation models or GenAI LLM's foundation models, I'll get into maybe a little bit why we call them foundation models, but whatever you call them, if you believe that's the future like we do, data is not just a pain point that blocks, it's effectively the new programming interface for AI. So in that context, what we've been working on is all of the tooling and the workflows for how you do that kind of programming. And I say it that way because a lot of how it's done today is very manual, very zeros and ones. Let's label some data points to fine tune or to put into a prompt or let's tweak the syntax of a prompt or let's manually kind of put together the right mix of data to dump into pre-training one of these models. And our goal has not been to make that go away because if you believe like we do that AI is, that data is the,

• 10:52Speaker 01

  data development is the programming language, the interface to tuning and building AI, you can't just make it go away. You can't make it auto magic. But we wanted to make it kind of lift it from the level of, you know, click, click, click, let me teach this model how to parse this type of legal document by digitally highlighting or asking my line of business colleagues to digitally highlight for months to something that looks more like software development.

• 11:13Jacob Effron?

  Totally. And to what extent has that required a change in the product? And obviously, you started the business before this LLM craze and ChatGPT happens. And I'm sure every conversation with customers is some remnant of a discussion around that. Like, did that end up changing kind of product roadmap or things? Or were you already kind of focused on a lot of this stuff? Maybe would love to hear a bit about what those first few months after ChatGPT were like internally.

• 11:39Speaker 01

  Yeah, well, it's a super interesting time to be in the space and it's both in strong positives and also lots of. Yeah, head spinning and confusion amongst the market and customers and, you know, so I'd say, but taking a step back, you know, a lot of what we've been building at Snorkel has focused on the data labeling and doing it programmatically rather than manually. So again, that means kind of writing software, we call them labeling functions. So, you know, a simple example, you know, if I see this kind of pattern in, I don't know, customer call complaint transcript at a bank, label it this way rather than that way versus, you know, asking people to click one at a time. And this programmatic labeling is sometimes called weak supervision because

• 12:24Speaker 01

  when you ask people to kind of programmatically or write programs to label data, the cool thing is you can do that via code. We have Jupyter Notebook and SDK. You can do it no code. You can even do it via natural language, which is something that LLMs have accelerated our ability to do. You can write prompts to do this. Snorkel's job is to help you develop that and then also to kind of clean them up because we don't expect them to be perfectly accurate. You know, if you say, if you give the machine a heuristic to teach it, it's not going to be as accurate as one carefully chosen label, even though it'll be radically more efficient and transparent. So this is in the literature article called weak supervision, just because I'd mentioned that before, because we don't expect these, you know, 10, 100 plus X faster kind of labeling functions to also be as accurate. We clean it up using theoretically

• 13:07Speaker 01

  ground techniques. So that was where we started, this loop for programmatically labeling data. And so then what changes when these, these models come out? Well, in some ways, a lot didn't change because, and I can give a little bit of insight why. First of all, was always our positioning. We've always kind of thought about a spectrum of use cases. People like to think about AI just applies to anything. I think the most important thing that determines success or failure of an AI project is problem selection. You know, what's hard, what's easy, what's measurable, what's not, what belongs with this tool versus that tool. It's a very underemphasized art still today.

• 13:47Speaker 01

  We we always, you know, from how we built the product to how we, you know, thought of, you know, filtering inbound, targeting outbound, enabling our sales team, thought about, you know, picking the right use cases. And we always wanted to pick use cases that were, you know, harder, that required more data labeling, because there have been pre-trained models, I guess now we call them medium or small language models for years, that were getting pretty good at the easy tasks. There were large hyperscalers that were paying people to label data to train off-the-shelf models for the simple types for a long time. And we did a, Azure is one of our big partnerships, we did a, um, uh, uh, the CTO of Azure AI, uh, uh, gave a nice presentation a while back before this craze of, hey, if you have

• 14:30Speaker 01

  a government contract, this is for the form recognizer product that we have, uh, we had one of our first collaborations with Microsoft around. If you have a, a, a government form, probably can just use an API off the shelf from cognitive services or whatever, right? If you have, you know, an institutional form, some PDF in the balance of the bank, you probably need to label data and tune a model with circle. So there was always this trade-off space. We always knew that there was going to be an easy segment that got driven to commodity, and there was going to be a more bespoke segment that was harder, often involved private data that's not on the internet. So, you know, the acceleration of that commoditization kind of leapt forward faster than we would have expected, if you asked me to kind of place a bet. I mean, I knew these model, we, we all knew these

• 15:12Speaker 01

  models were going to scale up. The, the, the, the scaling curve for ChatGPT and others blew a lot of us away. Um, and it's a really just incredible achievement. Also due to data, of course, not algorithmic chicks. So, so also something that we, we love because it, it, you know, plays to the party line. We were always aiming to position for these harder, higher value use cases, or actually harder use cases were more private bespoke data, more subject that our expertise needed, more challenging. These were always the problems where our, you know, better labeling and data development was going to provide a higher delta. And also they tend to align with the ones that are higher value, uh, for enterprises to solve and more white space in the market. So that positioning already kind of left

• 15:55Speaker 01

  us in the realm where still, you know, a, a, a GPT four, GPT five, et cetera, whatever comes out are going to have a little bit more trouble because it doesn't look like web data. And these are, you know, high value and hard because they need to be very, very accurate, right? And I think, still think about that kind of quad chart. Talk about a lot of, you know, think about, you know, how bespoke or complex the data is and how high of an accuracy bar you have to hit. If you're in the setting where it's kind of generic data and maybe you don't even know how to measure accuracy because it's human loop. Actually, I think, like Andreessen had an article, they were extolling GenAI recently. They were saying part of the wonderful thing is they don't even, a lot of the applications don't even need to measure or have high accuracy because they're human loop or they're creative.

• 16:38Speaker 01

  If you're in that setting, it's right. You actually can use these models, which are miraculous and their capabilities out of the box without much configuration. If you have, you know, private data at a bank and you're shipping something that has to run at 99% accuracy, pass through five model risk management committee reviews because it's going to go touch, you know, customers or regulatory topics, that still requires a lot of tuning, you know, generally on some kind of labeled data. So part of it's just positioning. And then part of it is the same kind of loop of labeling data, finding error modes. The tools that you connected to change. So we, you know, released end of last year, our foundation model suite, which had a lot of the connectors of how do you make sure you're starting

• 17:23Speaker 01

  with a state of the art foundation model? So you're not labeling data that already knows how to cover. How do you, you know, think about efficient ways to shrink it and distill it and lots of stuff we do there? The ways you talk about it change. So the syntax around all of this changes in a market shift like this quite radically. And that's a challenge in and of itself because people are confused. The data science teams we sell to are rewriting their strategies, having major reorgs, you know, the marketing becomes one of the most challenging pieces and saying that we're and others still working on the education.

• 17:58Speaker 01

  One other thing that I would say has changed is that the game. Is less about volume of labeled data or data in general, and it's more about finding the right data. And actually, a number of studies both from ourselves at UW, Stanford, and others show that all the way from pre-training these models to fine-tuning them, better, smaller, but carefully chosen data that kind of picks the error modes is how you win.

• 18:23Jacob Effron?

  I think you're really on the front lines of like figuring out, you know, what data is required to improve, kind of like pre-training, fine-tuning. Like, what have you kind of learned in these like, you know, initial months of deployment around, you know, what's actually required to, or, you know, and to what extent has your thinking changed or the way you work with customers changed around, you know, how you go about gathering the types of data that are going to be particularly effective in moving the needle on these models?

• 18:47Speaker 01

  Yeah, it's a great question. I think there's a, there's just a, there's been an evolution of the toolkit that we're still trying to, you know, everyone's kind of going through an education phase right now. So, you know, one thing that's come up is, is, you know, retrieval augmented generation. And this is where you see this, you know, kind of flush of vector databases and, you know, that idea of kind of plugging context in. And so a lot, you know, one question we often get is, okay, how do I think about ragging something, which I'm told is the term the cool kids are using, or at least I was told that by some Microsoft execs at Build. So I guess, yeah, I'll go with it, ragging it. When do I fine-tune a model? When do I just get away with prompt engineering?

• 19:30Speaker 01

  And, you know, being able to kind of situate there, like most things in ML, these tools are complementary. The metaphor I like to use, just to dig into that for a second, because it comes up often, is imagine a kind of off-the-shelf foundation model is like a, you know, an early med student. And let's say a medical setting. I know nothing about this, but, you know, so humor me if you have some medical background. But imagine they're trying to diagnose a patient. Some things, neither they nor a fully-trained, you know, expert doctor in their subspecialty could diagnose unless they have access to the patient's chart or latest tests. That's kind of like bringing in context from a standard database or any other number of metadata tables or from a vector database. It's just...

• 20:15Speaker 01

  Like no one theoretically could make a the decision without this required information. Right. And then there's the actual decision making where the med student needs to learn to function as well as a highly trained subspecialist in the field. And that's kind of like the fine tuning or prompt engineering. And, and so that's where you're actually trying to not just give the model some information, it just. Anyone, any human, any model would need to have to make the decision. You're actually trying to make it make better decisions. And so both of these are complementary tools. And so anyway, I'd say we're learning how to piece these together. We're learning that a lot of these models.

• 20:53Speaker 01

  They just don't need as much volume of data. They need carefully chosen data. So it's a lot more around. The error analysis loop, which. We've always built, but it changes how we emphasize and how, and how we said our build, build North Star. It's more about finding the errors or hallucinations. If you, you want to use the scare tactic term. And then correcting them in a targeted, you know, efficient way rather than. Just purely trying to crank up volume of, of, of data or labeled data.

• 21:21Jacob Effron?

  Totally. What are the best people that like are, are kind of, uh, that are, that are figuring this out? Like, how do they, you know, structure this when they get like errors or hallucinations? Like what kind of tricks have, have our folks learning about like, you know, the right types of data to feed back in to, to kind of get around this?

• 21:34Speaker 01

  Well, you're, you're setting me, you're setting me up to set, to sound way too self-serving by saying, oh, they're using snorkel flow and they're, you know, they're.

• 21:40Jacob Effron?

  I'm setting you up so you can, you can go for it.

• 21:42Speaker 01

  Well, I, I, I, I won't go that, that, that head, that, you know, head first in, but I will, you know, I'll, I'll. I'll share some of the workflows that we think are best practice and that we, you know, we build our product around, but, but they aren't confined to our product for sure.

• 21:58Speaker 01

  And a lot of it's going to sound very simple. It's, it, a lot of stuff just really comes down to trying the simplest thing first and then just. Doing rigorous error analysis and actually looking at the data. And that sounds really generic, but it remains very true because it's. It's very seductive to, um. You know, that's, that's it. Way too strong of a term, but again, we're on the nerdy podcast here. It's very, you know, attractive to just say, I'm going to go look at some loss curves, or I'm going to tweak some algorithms, you know, anything but look at the actual messy data. There's almost an allergy to it. Obviously, a lot of our mission is to make that at the center of your workflow and

• 22:40Speaker 01

  to make it less allergic and less painful. But usually it comes down to being very rigorous and iterative around your error analysis and remediation process. And so you start by, you know, just looking, I mean, the first thing is you have to be able to define your problem and success measure rigorously. It actually all starts with benchmarking and evaluation, which we're still far behind on and actually working on some, some cool new stuff there at Snorkel. What, how do I measure accuracy? What is ground truth for this? What are the specific things I want this chatbot or copilot to actually do?

• 23:16Jacob Effron?

  How do you even begin to crack that problem? It feels like a tough problem. I know a lot of folks have, have put forward some ways to go about it, but I'm curious how you guys think about it.

• 23:24Speaker 01

  Yeah, I mean, a lot, some of this stuff is, is upcoming because a lot of the problems that we work on in our GA platform and the bulk of our business today are purposely chosen to have more concrete evaluation metrics. That's a whole nother thing we could talk about. A lot of the use cases that people are trying out in the enterprise today are going to almost be doomed to failure because no one knows how to measure success or ROI on them. So we're placing some bets there and I'll get into a little bit what we're doing there in a second. I'm just noting that still a lot of where we see the durable value in the enterprise where people actually have to report on a, you know, an ROI to their

• 24:07Speaker 01

  AI expenditures. And a lot of where our work still is with customers is around, you know, classifying or tagging something or extracting something that has a clear definition that you can measure an accuracy against. And even there, it's not so trivial. But there, at least you can get a number and you can know, is this good enough to ship to production or not? Am I automating something or not? Now, the interesting setting is some of these kind of chatbot copilot use cases, which we're starting to support our customers, a bunch of customers and doing. There, you know,

• 24:40Speaker 01

  People like they often do in machine learning are kind of looking for these magic tricks. You know, what kind of exact configuration of, um, you know, semi automated, uh, uh, you know, using a G using G P T four to grade G P T three or, you know, doing it in this kind of way like. These are valuable. There are, you know, semi automated ways to do evaluation. There's also a whole area called active validation. Uh, that we've done some work on around being smarter about which subsets you pick to take a look at, um. That all matters, but I think the bigger thing comes down to just making sure that you have a fast way to build custom benchmarks that reflect the actual, you know, business or use case needs. Uh, and so I think right now, um,

• 25:26Speaker 01

  you know, my, my co-founder Chris is very involved with the helm benchmark and you have these kind of at Stanford, you have these big kind of generic benchmarks. But I think a lot of what's it's going to come down to, and we're, as you could probably guess, building some stuff here is really just custom micro benchmarks that actually, you know, enable you to rigorously and iteratively say, these are the types of things I want to make sure this, this thing doesn't mess up on. Uh, and that is part, you know, M L magic and U I U X stuff speeding up. A lot of it is just rigor and discipline in the actual practice of applying A I, which it, it, you know, is, is, is just needs time to, to build up. And then a lot of the interesting thing there is working backwards. It's just, there's a lot of, of new tools. When, when do I, um,

• 26:11Speaker 01

  when do I need to just plug in some rag, uh, uh, you know, vector database caller or standard database call because my model is missing some info. When do I need to fine tune or dump a bunch of effectively fine tuning data into the prompt? When do I actually need to pre-train a model? And so what we're trying to do with our tooling and our work with, with customers is um, actually be, enable people to be rigorous.

• 26:35Jacob Effron?

  It makes total sense that like evaluations, a, a natural prerequisite to be able to, to do any of that.

• 26:40Speaker 01

  And I think there's going to be real value all across the line. Like we just, we just, um, one of our customers, a large e-commerce company just actually, you know, gave, gave a readout to their board of some awesome results, results we got of, of helping them uh, curate the data. So this is not labeling as much as it's shaping, sampling, filtering, some of the stuff that we recently announced. Um, in, in, uh, in early preview, uh, and they were able to get, um, Previously, they'd been doing hours and hours of of of fine tuning of a model to get to this like seventy one on average. And then they were able to custom pre-train a model and start at eighty six with their kind of, you know, we call it, you know, GPTU, not GPTX. So um, there are real reasons to go really deep and build really custom.

• 27:25Speaker 01

  But you want to be rigorous. You want to try the simplest thing first. And some problems, it's going to work. And then you want to zero in on the problems where you actually need to go deeper. And then maybe you go fine tuning. And then you find the subset of problems still that aren't good enough. And then you go and actually start building more and more from scratch. It's really just rigorously working backwards. And um, no one wins, even us as a software vendor, if you, you know, apply a an overly heavyweight hammer to something that's a, you know, a a tiny problem.

• 27:49Jacob Effron?

  I mean, it's interesting because you're working, you know, I feel like you have a a who's who of of customers and across some really interesting industries. And, you know, uh, you were kind of alluding to it earlier, but it seems like, you know, even with uh, everyone talking about generative AI in the boardroom. You know, a lot of the the main use cases folks are still using uh, these models for more classic like machine learning use cases. Um, you know, classifying labeling and whatnot. And and I'm I'm curious, like, you're obviously on the front lines, like. Are you starting to see some of these, you know, uh, large enterprises, you know, actually implement L L Ms? And like, what are some of the most interesting use cases you're you're seeing?

• 28:21Speaker 01

  Yeah, I mean, so it's we're definitely, I mean, we're definitely seeing, yeah, I I don't think it's going anywhere. We we've certainly. You know, to be clear, we're not, we're not a a foundation model pessimist. We're we're quite the opposite. We put all of our chips down across Stanford, U Dub, Snorkel. Uh, we're just, we're just, you know, realists. We believe we're going through a a hype cycle curve, right? Actually, shameless plug, I think we're the only uh, uh, uh, you know, software vendor uh, that's in two categories on the the Gartner uh, recent uh, AI hype cycle report. But um, the um, you know, the viewpoint is really just that there's, there's a, there's a little ways to go before these these hit production. So what we're seeing is enterprises are definitely working to adopt this.

• 29:03Speaker 01

  You know, some of these, you know, the a lot of the customers we service, you know, we're working with uh, right now, five, uh, six of the top ten US banks, uh, government agencies, uh, pharma, healthcare, et cetera. These people have to. Be very careful with what they put out there because it has significant impact, whether, you know, anything from patient health to massive financial ramifications to regulatory implications. And so, you know, there's legitimate caution in terms of bringing this in. There's actually quite a bit of concern of, I think people are aware of kind of the concern that enterprises, especially larger ones with very unique private data have of sending stuff out to say a foundation model API. They also have concern about bringing stuff in when they don't know what the training data lineage is.

• 29:48Speaker 01

  Right? They're not indemnified against, you know, some code copilot auto suggesting something that's actually copyrighted and then it moves in. So there's a lot of concerns they're working through and it's a lot slower than you might expect. But there's definitely, you know, definitely this stuff is moving through. I kind of already shared this viewpoint, but I think a lot of what's going to happen in at least the kinds of enterprises we work with is what we sometimes call capture and divert, which is, you know, there's all this like, you know, chat GPT excitement coming down from the C suite. And then, you know, shrewd data science AI ML leaders are going to actually divert that to more classical workloads that already have proven evaluation metrics, proven business value.

• 30:31Speaker 01

  You know, so imagine, you know, a lot of what we're seeing is, you know, I have like a a model that, you know, classifies loan documents or pulls out of customer packets. And I'm just going to accelerate that by building on top of a large language model, rather than I'm going to go straight to like a copilot or chatbot. And I think what we're seeing on the latter front, it's definitely real. Some of these newer kinds of use cases, building a copilot chatbot. Most of our customers are using them internally. So you're seeing a lot more, you know, internal IT help desk support bot, nothing, nothing customer facing in most of our, a lot of our customers.

• 31:08Speaker 01

  And I think people are realizing as they go through the hype cycle curve that it's a little bit more nuanced. One customer of ours, large bank, actually just finished their first AI pilot. We were not directly involved, but heard that they were, the report was that it was a copilot for regulatory reporting. That was specific to this this organization. And their line of business partners thought it was about five times slower than writing the reports from scratch, because they found that checking for all the, you know, all the citations and the references for these very technical reports for potential hallucinations of which there were enough to require this was actually just net slower and more cumbersome. Are they turned into pessimists? Absolutely not.

• 31:51Speaker 01

  But there, you know, there's a lot of folks who are just realizing, hey, there's a category of use cases where maybe it's, you know, marketing SEO copy or creative digital or this, that or the other, where this stuff is pretty magic. Or some of the easier use cases now, it's it's just kind of works out of the box. And then there's a big category of ones, especially ones that are higher value and closer to significant, you know, consequences that they're seeing there's a long pathway to, I mean, to even measuring and understanding, let alone operationalizing these things.

• 32:21Jacob Effron?

  Totally. I mean, you've hit on a lot of the barriers, right? Obviously, like, you know, there's, there's barriers in just pure evaluating and measuring success. There's barriers in terms of knowing the data lineage of how some of these models have been trained and copyright issues. You've you open yourself up to the ability to kind of deploy in your own environment. But I'm curious, you know, any anything else you're seeing that's kind of a barrier to enterprises, you know, deploying some of these generative AI use cases? And, you know, also curious kind of what that means for the snorkel roadmap, as well as, you know, the broader startup ecosystem in terms of what's kind of important to build here?

• 32:51Speaker 01

  Well, I mean, the first thing starts with just, just training and inference cost. That, you know, there's the the chip shortage that we're all aware of in the broader environment that's leading to some, some crazy, crazy hijinks in the space. In a lot of enterprises, most enterprises are still working on migrating to the cloud, right? Let's keep that in mind, both as a kind of general yardstick of how fast things percolate out into the into the the field in terms of tech waves. But it also has effects of it's it's harder. It's often, there's often just blockers around, but the governance perspective, but also just the hardware availability perspective. And so one interesting ramification this has had, which is not surprising, but it's

• 33:34Speaker 01

  actually exciting because it's happened faster than we thought, is interest in what's I've called model distillation, or basically all forms of how can we, you know, get a smaller, more specialized model to work in these settings? I'll just double click on that for a second. We actually, um, one of the awesome students I get to co advise at U Dub and working with some Google AI collaborators recently published something called Distilling Step by Step. So a paper on a new technique here. This is actually an old idea that you can take a large, I mean, it's the basic intuition is simple. Take a large, massive generalist model that can maybe answer anything from, you know, write me a poem about a coding interview to, you know, I don't know, whatever people do on the internet with these things.

• 34:18Speaker 01

  It's, it's, it's, it's far surpassed my ability to keep up, but down to, you know, you a specialist for the specific task that you need to perform, you can massively shrink the model. So I'll give an example of this. Um, we, we published a case study anonymized from one of our customers recently where they were, um, trying to train a model to do very specific types of product attribute extraction, uh, from actually both from image and text. This particular example is over text. So it's probably managed a massive catalog of, of, of products that they have for both backend management and then front facing user experience. So very important problem. Interestingly, this is all public data that's on the internet. It doesn't match heuristic test number one that I shared earlier of where might GPT four struggle.

• 35:00Speaker 01

  But it's really because it's not just couch or chair or TV or, or, or DVD or whatever. It's, it's very fine grained categories that do with their catalog. So they applied GPT four and the number of these problems, it gets kind of, you know, forty F one, which is a type of accuracy measure. They did some prompt engineering, actually using snorkel flow. So looking up error modes and then, you know, coming up with examples, which they were putting into the prompt, uh, which you can do rather than fine tuning, um, that increasingly kind of there are those things are going to converge. Got to about seventy, then they finally actually did some fine tuning and they got up to the mid eighties, which is where they were fine to start using it.

• 35:40Speaker 01

  The interesting kicker though, this mid eighties model was a model that was 10,000 times smaller than GPT four. So often as you become more specific and accurate at a task, you can also shrink down the model. And this is actually, it's no surprise that we did this research with Google. I mean, Mature, long time users of massive language models in production, i.e. Google and Google, have been doing distillation for years. So that's been one cool thing that we thought was like a nerdy footnote that people would take a while to kind of come around to. And it's actually gotten a lot more excitement. I guess unsurprising in retrospect because of just the constraints on compute in real enterprise environments.

• 36:17Jacob Effron?

  That's awesome. And I guess you guys like have a product around distillation as well.

• 36:21Speaker 01

  Yeah, it actually falls kind of out of our normal process because once you start labeling and refining data, you have a pretty straight path to now train a smaller model from that.

• 36:30Jacob Effron?

  One thing I'm struck by is just the sheer breadth of like, you know, your product suite, right? It seems like whatever the problem is around, you know, large language models or traditional ML use cases like you guys are releasing or have products around it. Like, how do you think about like the boundaries of like, you know, do you want to be everything for like these enterprises and their ML solutions or there are places where you're like, you know what, we don't ever need to build X, Y, or Z. Like, how do you think about what the snorkel footprint of the future looks like?

• 36:54Speaker 01

  One thing I'll note is that, and this, you know, we dig into right away in terms of our pipeline when we start engaging with potential customers is that we do have specificity around data and use case types. So people think of AI as any task, any data type, it's all the same thing. And that's a really exciting, that's true from the model centric perspective. Models these days, like whether it's image or video or graph data, it can all get dumped in as a vector to a transformer and it works pretty well. That's unheard of, even if you go like four or five years back. And it's the reality today. And this has been a big dream of machine learning. So from a model centric perspective, kind of anything works with the same tools. And that's awesome, right?

• 37:38Speaker 01

  And that's part of the scalability of how we can do lots of things. From a data centric perspective, whether you're looking at, you know, extraction from PDFs or, you know, object detection in images is a very different toolkit from a data centric and from a snorkel perspective. So one thing I'll note is we do roll out new capabilities around specific data and machine learning task types. And Because we're building a whole programmatic stack around those, we actually do, you know, we have, it takes us some time to navigate those, right? So that's one way in which we're narrower. The second thing is we really do try to stick to developing the data. We include some basic tools. About half our customers are using Snorkel Flow to train models. But it's not actually a lot of development on our side.

• 38:23Speaker 01

  It's really just because our whole thesis, and it's not some clever Trojan horse thing, or it's not a, it's not a quiet one. It's something that we've said for years, if you get the data right, nine times out of 10, you're probably fine with just a standard open source model and just go tune that. So we hook into standard libraries. We believe it's important to have some kind of model training in Snorkel to give feedback as you iterate on the data. But our focus is really just on the data. The labeling and now actually also the sampling and selection, the filtering, the cleaning, which is what we've been expanding out to. But that's really where we keep our footprint and we aim to be as interoperable as possible with your favorite model training platform with, you know, upstream and downstream. So that's another way we stay focused.

• 39:05Jacob Effron?

  I'd be really curious, you obviously in your current, you know, role with University of Washington, you're advising students, you started Snorkel with some of the research you've done out of, you know, your own PhD program. Like, how do you think about the role of academia and AI development going forward? I mean, increasingly, obviously, we're in an ecosystem where the largest amounts of compute exist outside of academia. Like, how do you think about the role of, and especially as someone who is going to be an academic, like, I'm curious how you think about the role of academia in the space going forward?

• 39:31Speaker 01

  Yeah, it's a, it's a topic that a lot of people are thinking about and it's, and it's, you know, a lot of things, more things than people think, I would say, have not changed with this latest wave. But, you know, many things have and there are lots of existential crises going on and, and that includes in terms of, you know, I'm a new PhD student coming in, for example, or master, or I'm, you know, I'm a seasoned researcher. Where do I fit into this when there's such a big, you know, compute bill for so much of this, of this work?

• 40:01Speaker 01

  You know, I, I, I think that, um, I think when the dust settles and it's already settling, there's, there's a lot more room than, than people realize. I, I do think there is some stuff that we should be thinking about as a, a community to level the playing field a bit. I, I think actually there's a new act that just got, uh, I think put up as, or got proposed in Congress. I saw something from the Stanford, uh, HAI Institute in my inbox about this. Uh, so not directly involved, but something about, you know, basically giving, subsidizing, um, uh, academic institutions, research institutions for compute budgets. I'm very biased, but I think this is a prudent thing to do because if you look at, I mean, you know, the attention is all you need paper came out of Google, but attention as a mechanism

• 40:44Speaker 01

  came out of academia. The transformer architecture, same thing, you know, uh, new architectures that may beat transformers, again, same thing, like a lot of the progress that we all now rely on comes out of academia or, and I would include, you know, pure academic research labs like at Google and Microsoft and others. If we, um, if we kind of starve the funding, no one's going to benefit. But I do think there's plenty of, of, of areas to look at where you don't need, um, to train a massive model from scratch. And, and the kind of obvious reason for that is most users of AI are not going to train, uh, uh, you know, uh, uh, an eight-figure, nine-figure training run from scratch. So all you have to do, and I'll say this directly to, you know, folks who are maybe entering grad school and looking

• 41:29Speaker 01

  for new topics is just put yourselves in the shoe of a user, uh, who is, you know, not working on, you know, working on maybe a, you know, a more interesting, non-standard setting. They're not just trying to have, you know, chat about coding interviews on the web. They're trying to do something that's a little bit more bespoke or interesting. They don't have a massive compute budget. Start there. You'll find out that there are a lot more problems that, uh, you know, not only are fun to deal with from the kind of tool builder side out in industry, but are fascinating from the academic side.

• 41:59Speaker 01

  There's plenty of last mile to deal with, and that's obviously what gets us excited as a tool builder. But anywhere where there's that, there's all the formalization, the deeper understanding, the more creative thinking that has to happen on the academic side. So there's plenty of room left there would be my perspective.

• 42:14Jacob Effron?

  Awesome. Well, we always like to end our interviews with a quick fire round. So I'll give you a few questions just to get your quick response. Would love your take on what's overhyped and underhyped in AI right now.

• 42:23Speaker 01

  Overhyped, let me give a pitch that's different from my, you know, because I guess the standard answer would be overhyped is, you know, GPT-4 out of the box. I'll give you two answers, right? One is just straight talking in my book, both commercial and academic, which is, you know, overhyped is GPT-4 or GPT-X out of the box. Underhyped still, I think, is what we've called GPT-U. So anything that's actually tuned on your data, whether you're actually, you know, going from scratch like a Bloomberg GPT, or you're just kind of doing a little bit of fine tuning the last mile. You know, we think that the landscape, especially in the enterprise, is going to be more like a thousand task-specific, smaller specialists rather than, you know, one big off-the-shelf model.

• 43:04Speaker 01

  But the second thing I'll say, just to go a little bit different, is I think co-pilots, in terms of their near-term value, are pretty overhyped right now. Not because I don't think they're cool, but because I think there are some very classical challenges that people still have to run into. Everyone got excited about coding co-pilots. Coding is actually an easier problem in some ways. Lots of data, lots of built-in checks. I mean, you have whole teams that are writing unit tests, you have compilers, you have a user that's used to this kind of auto-suggest-based workflow. There's classical challenges here that we still have to deal with that are tough, right? There's the operator complacency challenge. The more accurate the model, the co-pilot model gets, this has been studied for decades,

• 43:46Speaker 01

  the more complacent the human operator gets and the less likely they are to catch the areas where it messes up. Anyway, so co-pilots, I'm hugely optimistic on long-term, but I think their near-term, like, real business impact is overestimated because there are

• 44:02Speaker 01

  Deeper challenges lurking to actually deploy them.

• 44:04Jacob Effron?

  And then I guess, you know, obviously, there's a lot of talk about AI regulation right now. Like, how do you think AI regulation should change going forward?

• 44:11Speaker 01

  Well, I'll fall back to my book again here. I think that I think that data has to be in the picture. And actually, one of the motivations that we've had at Snorkel, and we've talked about this with various folks on the, you know, the federal regulatory side as well as on the academic side is, you know, bias and errors can come in at any stage of the system. And often, you know, at the first or last mile. One of the biggest ones that were places they come in is the data. And if you have a kind of ad hoc manual way of curating your data set, everything from, you know, what distribution of, you know, which gender of resume did I collect to dump into my, you know, resume GPT? To how you label it.

• 44:54Speaker 01

  If you do that in an ad hoc way, the challenge of identifying, let alone correcting biases, for example, is actually theoretically harder than the problem you're trying to solve in the first place. So obviously, if you take a more structured way, like a programmatic approach, and that's a lot of the tooling we build, our motivation is not just about better accuracy. It's about better governance, better, you know, de-biasing, better, you know, auditability of the data that informs so much of what AI models do. So I guess my quick take, skirting some of the thornier issues that I'm not an expert on, is that whatever we do from a regulatory standpoint, we should be mindful that, you know, the biggest delta in how these large language models perform today is what data sets they were trained on. We should have regulation about maturity of management and curation of that data.

• 45:39Jacob Effron?

  Totally. Well, Alex, it's been a fascinating conversation. I feel like we've hit on a lot of really interesting topics. I'm sure folks are eager to learn more and dig in. What's the best way for them to learn more about you and Snorkel?

• 45:48Speaker 01

  Yeah, well, our website's at snorkel.ai. Lots of stuff up there. If you're really masochistic, I think we hit something like 95-ish peer-reviewed publications. So you can go deep with links on the website. But I would recommend just, you know, some of the top level kind of, you know, YouTube videos or product demos to give you the basic idea of what we focus on. And obviously, you know, if you want to go deeper than that and talk again, one of the benefits of us being still a small startup is we're We're always eager to just find the right fit. So if you want to have a conversation, reach out to us. We'll either find a good overlap with our platform or try to point you in the right direction. And yeah, looking forward to some conversations.