Getting in the Flow with Snorkel AI

Braden Hancock joins Chris to discuss Snorkel Flow https://snorkel.ai and the Snorkel open source project https://snorkel.org. With Flow, users programmatically label, build, and augment training data to drive a radically faster, more flexible, and higher quality end-to-end AI development and deployment process. Sponsors: DigitalOcean https://do.co/changelog – Get apps to market faster. Build, deploy, and scale apps quickly using a simple, fully managed solution. DigitalOcean handles the infrastructure, app runtimes and dependencies, so that you can push code to production in just a few clicks. Try it free with $100 credit at do.co/changelog https://do.co/changelog. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. LaunchDarkly https://launchdarkly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Power experimentation at any scale. Fast and reliable feature management for the modern enterprise. Featuring: Braden Hancock – Website https://www.bradenhancock.com, X https://x.com/bradenjhancockChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Snorkel AI https://snorkel.aiSnorkel OSS https://snorkel.orgSnorkel Blog https://snorkel.ai/blogSnorkel AI | Twitter https://twitter.com/SnorkelAISnorkel AI | LinkedIn https://www.linkedin.com/company/snorkel-aiSnorkel Best of VLDB paper https://link.springer.com/article/10.1007/s00778-019-00552-1Snorkel Drybell collaboration with Google https://dl.acm.org/doi/abs/10.1145/3299869.3314036Jerod recommends Getting Waymo into autonomous driving (Drago Anguelov) https://practicalai.fm/103Building the world’s most popular data science platform (Peter Wang) https://practicalai.fm/101Achieving provably beneficial, human-compatible AI (Stuart Russell) https://practicalai.fm/85Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  These noisier sources of supervision can be much more scalable and much faster to execute, easier to version control and iterate on than individual labels are. And if you can layer a number of these on top of each other and basically then let their votes be aggregated by an algorithm, one that we developed at Stanford, you now have the ability to get, maybe not 100 perfect labels, but 100,000 pretty good labels. And it takes about the same amount of time. And as we've seen time and time again in recent years, the size of the data set seems to keep winning the day when it comes to getting high performance with these models.

• 0:29Speaker 01

  Bama for change log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com. And we're hosted on Lino cloud servers. Get $100 in hosting credit at lino.com/changelog. This episode is brought to you by Digital Ocean. Droplets, managed Kubernetes, managed databases, spaces, object storage, volume block storage, advanced networking like virtual private clouds and cloud firewalls, developer tooling to robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150,000 businesses that develop, manage,

• 1:13Speaker 01

  and scale their applications with Digital Ocean. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 1:41Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:07Chris Benson?

  Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a principal emerging technology strategist with Lockheed Martin. And unfortunately, Daniel was not able to join us today, but I have a guest that I'm excited to talk to. Today, I have Brayden Hancock, who is the co-founder and head of technology at Snorkel AI. Welcome to the show, Brayden. How's it going today?

• 2:30Speaker 00

  Thanks. Glad to be here. Doing well.

• 2:32Chris Benson?

  Well, I was wondering if you would start off telling us a bit about your own background, and let us understand how you got to where you're at. And then I'm looking forward to asking you more about Snorkel AI.

• 2:41Speaker 00

  Yeah, absolutely. So yeah, as you mentioned, I'm currently a co-founder and head of technology at Snorkel AI. The company's been around for about a year and a half now, maybe coming on two, and it's been a blast. Before that, I was a Stanford PhD student, along with all the rest of my co-founders. That's actually the origin story of our company. And then that's what brought me to the Bay Area in the first place. Was actually not in computer science originally. I came from mechanical engineering, and just found myself consistently being drawn to machine learning. And then finally, you know, saw the writing on the wall and made the jump myself going into grad school.

• 3:18Chris Benson?

  So I'm just curious, because we hear that kind of story a lot where people are coming in from an industry that you may or may not expect that to happen. When you were still doing mechanical engineering, what was the draw into machine learning for you at that point? I'm just curious what it was that started that that process of sliding over.

• 3:34Speaker 00

  Yeah, no, I completely agree with you. I think we see a lot of people make that shift, and it's perhaps not too surprising, just given how rapidly the field is growing, that the people have to come from somewhere. But for me, I think part of it's just how much faster the science is in computer science. The fact that you can iterate so much more quickly. An experiment can be run in seconds or minutes, certainly set up and run in a day. Often, you know, for certain experiments compared to when you've got a mechanical rig and there are parts and, you know, one bad transistor and the whole thing's kind of suspect. And there's just so many more failure modes and such longer timeframes that I kept coming back to, I want to be able to answer questions quickly and I could do that so much faster in CS.

• 4:16Chris Benson?

  And do you think that's going to be a situation that we see over and over again with people in various industries pulling in? We've seen a certain amount of that. I tease Daniel a lot. We end up talking to people that come from physics a lot and one way or another, they found their way over. So do you think that's going to be very typical with mechanical engineers constantly finding the need to use machine learning to get their jobs done and whether or not they jump over to the dark side or not?

• 4:40Speaker 00

  I'm sure we'll continue to see plenty of people jump in from over there, you know, for different reasons probably, right? I think for a lot of people, you end up finding that the best way to do your job is to use machine learning and then you realize, hey, this is actually a really cool tool. I think I'd like it to be more than just a tool for me. And then you really lean in and start, you know, diving in a more permanent way rather than just in sort of an applied sense.

• 5:01Chris Benson?

  Gotcha. So I'm just curious, what was your first experience as you started getting into machine learning? Before you made the full jump, what was that thing that was drawing you in? What kind of models were you doing? What was your tooling that made you think it might be time to make a shift?

• 5:17Speaker 00

  You know, this maybe just shows how thick I am that it actually goes all the way back to high school that I was, that I had my first dabbling in machine learning and loved it and just didn't even, didn't realize then that I should have just embraced it full, wholeheartedly from the get-go. But there was a lucky break for me. There's an internship program for high school students near Wright-Patterson Air Force Base in Ohio where I grew up. And so I was on a project using MATLAB, of course, the lingua franca of mechanical engineers, not the Python of machine learning engineers. But yeah, so I was, the task that I was, you know, assigned to was using genetic algorithms to design better airfoils. So some non-gradient based optimization. And I thought it was so cool that even after I lost my MATLAB license, you know, during the school year, I had to go back to high school.

• 6:02Speaker 00

  This was after my junior year. I used Excel and you know, I had a separate tab for each generation of the genetic algorithm and like tried to recreate it there because I was still, of course, a lousy programmer, but just thought the ideas were so neat.

• 6:14Chris Benson?

  Very cool. So is it genetic algorithms were what actually pulled you in and going from that and thinking all the way to now as a co-founder at Snorkel AI? What was the crossover right there that got you to Snorkel AI?

• 6:28Speaker 00

  Yeah, I'd say, you know, from the very beginning, the one of the ideas that drew me in was there should be this different interface for for getting things done, for transferring information from an expert into a program that can now do work for you. And I think historically, that's very imperative code, very much like describe exactly what you want done step by step. And that was was less interesting to me, felt a little bit more like your job's just to translate, you know, from one language to another. But the cool thing about machine learning or AI in general, I think, is that you get more of a sense in the right setup of if you can tell me what's good, then I can find it. There's like, there's this better synergy between the human and the computer where now I can show you what I want, even if I don't know how to get there and you can get there, you know, where you here is the computer, of course.

• 7:13Speaker 00

  So I think that's the broader idea that was really appealing to me all along the way that had me coming to machine learning. And then throughout my PhD, I kind of dove into that problem much more deeply of what really is the best interface for getting domain knowledge from an expert into a model? And that's, you know, those are themes that I explored for multiple years that along with my co-founders ended up, you know, being what led us to Snorkel and then Snorkel AI now.

• 7:36Chris Benson?

  Just to dive in there a little bit, was there a particular itch that you were scratching in that context that actually led to Snorkel AI? Was there something you can relate where it's like, well, guys, we got to solve this particular issue. This is something that we need to dive into that might have been the the specific genesis or?

• 7:54Speaker 00

  Yeah, so I'd say one thing that my PhD advisor was fantastic about was Chris Ray at Stanford. And he, I think, is very good at making sure that the problems you're solving actually will matter to people, actually solve real problems. And part of the way that you do that is by, you know, on most papers, we would try and have real-world collaborators work with another, you know, company or research organization or government entity or something where we could make sure that like this actually solves your problem. So people are more likely to care. This is likely going to stick and, you know, have a potential to make real impact. And so very early on in my degree is we were looking at what is the effective bottleneck for new machine learning applications? What is it that stops people from solving their problems quickly, as quickly as they'd like to?

• 8:38Speaker 00

  The realization came that that bottleneck is almost always the training data. You know, we saw kind of the writing on the wall, deep learning was blossoming right about then. We saw these super powerful models, feature engineering is becoming a lot less necessary. A lot of that can be learned now, but with the one caveat of you can do all this if you have just mountains of, you know, perfectly labeled, clean training data, ready to go for your specific task. And that in reality never exists, of course. And so that I'd say was the real impetus for this line of work was, you know, this is what stops people. In academia, it's download the data set and then do something cool with it. But in industry, it's

• 9:16Chris Benson?

  Get the data.

• 9:16Speaker 00

  I mean, steps like one through nine is where am I going to get my data? And do I have enough of it? And is it clean enough? And these annotators are doing the exact wrong thing. I can clarify the instructions. Is this good now? It's iterating and 80% of the work is making that training set. After that, like pulling off some state-of-the-art model in the open source and running that, that's the easy part.

• 9:32Chris Benson?

  Yeah, it's funny. You would think of AI as, I think, people outside our industry look at this and think we're doing this, you know, dark magic of AI and producing the model. But every time we talk to somebody, it's always trying to get set up to do that. It's getting to that, getting to the starting line of doing the actual modeling itself that people are struggling with. So tell us a bit about Snorkel AI. You know, how did that blossom out of this experience that as a co-founder you were having and as well as what, you know, what the others were driven to do as well? And can you tell us a little bit about your co-founders and just kind of how the whole thing got started?

• 10:10Speaker 00

  Yeah, so we feel very lucky at Snorkel AI to have have the founding team that we do. It's a little bit larger than you typically have. There are five of us. It's Chris Ray, who I mentioned was my PhD advisor, myself, and then three other previous students. We were all sort of in the same cohort, Alex Ratner, Peroma Varma, and Henry Ehrenberg. And all of us began at about the same time, our grad school experience and picking up different projects. And all of us were just drawn to these ideas and ended up collaborating in almost every combination you can think of between the four of us on different papers through those years. And it was going to be in the beginning like, this is an interesting idea. Let's run a quick experiment, pull up a Jupiter notebook, test some of these ideas. And then it really seemed to work. And so then it became a workshop

• 10:53Speaker 00

  paper, and then a full paper, and eventually a best of paper, and an open source project, and then an open source ecosystem, and other derivative projects, and lots of collaborations. And we helped a few different organizations make industry scale versions of this internally to really prove out the concept, a paper with Google, for example, that we were able to publish. And by the time that we were at the end of our degrees, it was clear that there was just such a dramatic pull for this. I mean, the ideas were very well validated at that point, over probably 35 different peer reviewed publications, but maybe more importantly, a whole bunch of different organizations that independently had seen success with these approaches, almost always from working with us to help lead them through

• 11:35Speaker 00

  the process. And so we just learned so much through that time about what you would really need to take this proof of concept and make it something that could be repeatable and with a relatively low barrier to entry that doesn't require a room full of Stanford PhDs to make successful. And that's part of what motivated the company, is the chance to now make this a fully supported, enterprise ready, and able to be shared with a whole bunch of different industries and company sizes and in different work areas.

• 12:02Chris Benson?

  Before you dive into the specifics of the product and service offerings, could you talk a little bit about what you did learn? Because with that opportunity to be doing the academic work and to progress through that over time, and have that insight before you ever actually start the new company, can you talk about what that learning process was like, and what were some of the things that had a big impact specifically, conceptually? And then from there, I'd like to go one into how that was realized in the company itself.

• 12:35Speaker 00

  Yeah, absolutely. And I completely agree. I think it was a huge, huge advantage for us to have that, I mean, really much larger period of time than you would ever get as a startup to do the learning phase, right? We were able to succeed and fail and try different variations and really push the boundaries and intentionally try to find where does this fail? Because as an academic, that's the hat you wear is like, let's really suss out. Let's do every ablation we can think of. Let's figure out, does this work for text? Does this work for video? Does it work for very dependent and correlated data? Does it work for, you know, the whole variety of the space that you can imagine, we were able to test. And so I meant that by the time that we were building now the, you know, the air quotes, final version, like the enterprise version, we were able to bring all these different learnings to bear as part of that

• 13:18Speaker 00

  design. So if I was trying to structure categorically the lessons that we learned, I think one of the big ones was interfaces. I think as a grad student, supported open source project, you don't have a lot of time to polish up a front end for people, right? So it's in the form of like a Python package. And if there are unit tests, you're lucky because that's, you know, of course, we cared about that, but it's not necessarily in your incentive. Unit tests don't lead to papers, right? It just means you have a system more stable development as you work. So it was fine and it worked well. And we, of course, did support it as much as we could. But I think one thing we did realize is, you know, we were writing a lot of the same code over and over again. There were certain templates for labeling functions. I think we'll talk more about those later. But, you know, third party integrations or like just sort of patterns of sequences of steps that people would try

• 14:03Speaker 00

  that get lost if it be in like the forest of scripts and notebooks. Whereas if you can set up a properly structured interface and GUI as well as other access points, you could really dramatically improve the likelihood of success. So that's one category, the interfaces, I'd say.

• 14:18Chris Benson?

  Okay. What else did you learn along the way? Was interfaces the primary driver there? Or was there any other key lessons there?

• 14:24Speaker 00

  Yeah, no, I mean, it's interfaces was a big one. I'd say if I was grouping it into other areas, I'd say there was also infrastructure. There was like intuitions that we gained and baking those in and then sort of like user profiles or like interaction points. So I can say word about each of those. On the infrastructure side, I think that one's fairly self-explanatory. If you're going to have your, as a company, if you're going to depend on a piece of software, you need it to have certain things like, I mean, basic security and logging, encryption, and compatibility with the data formats that you care about, and dependency management, and parallelization, all these things that, of course, of course, you want in software you're going to depend on, but that, again, just aren't necessarily a part of research code that's meant to be more of a proof of concept.

• 15:09Chris Benson?

  Sure. Kind of making it real comes down to really kind of classical software development things that you need in place to deploy remote software. And I think that comes back to a point that we, that we run into a lot on the show, and that is the fact that you can't really separate the AI from the software the AI is running in. And it sounds like y'all had a realization about that even before you got the organization launched.

• 15:32Speaker 00

  Yeah, no, absolutely. I mean, I'd say another big piece of this is, again, as an academic, you test often these ablations. You'll test a very specific problem and can the model learn what I need it to, but in the wild, you often have actually just a problem you need to solve, and you don't necessarily care how that's solved, you just, you know, you want a high-quality system. And so you typically, you know, you don't just have this one model that's ready to go with the data that you care about, that has an output that is exactly what you care about. There's always, it's a pipeline, right? You've got preprocessing steps, you've got business logic, you're chaining together multiple models or multiple operators, some are heuristic and some are machine learning-based. And so this actually gets at one of the big differences, I'd say, in terms of, you know,

• 16:15Speaker 00

  fundamental value out of the snorkel open source versus snorkel flow, the business product now, is that the latter is much more focused on building AI applications, like an application that solves your problem from end to end, rather than just a point solution for a part of a pipeline that is making a training set or training a single model.

• 16:34Chris Benson?

  Gotcha.

• 16:53Chris Benson?

  So Braden, just a moment ago, you were talking about Snorkel Open Source and Snorkel Flow. And could you now define what each of those are and describe what the differences in the two are?

• 17:06Speaker 00

  Yeah, absolutely. So if you go to snorkel.org, that's the website for the open source project that again began almost four years ago at Stanford and served as sort of our our testing ground and proof of concept area for a lot of these ideas around can we basically change the interface to machine learning to be around programmatically creating and managing and iterating on training sets? And so that's what that is. It's pip installable. You can pull it down now. It's got 4,000 something stars and it's used in a bunch of different projects. Snorkel Flow is the offering now, the primary product of Snorkel AI. And that is just based on and powered by that Snorkel open source technology.

• 17:48Speaker 00

  But then it's just sort of expands to much more. It is now a platform, not a library. It comes with some of those infrastructure improvements that I mentioned before. It also bakes in a whole lot of the intuitions that we gained from the years of using the open source. There are certain ways that you can guide the process in a systematic way to creating these programmatic training sets or improving them systematically, really completing the loop so that at every stage of the way, you have some sort of hint at what should I focus on next to improve the quality of my model or of my application? So that application, Snorkel Flow is, as I mentioned, or sorry, that platform

• 18:29Speaker 00

  is meant to be this much broader solution for supporting end-to-end pipelines, not just the data labeling part, baking in a bunch of these best practices, tips and tricks that we learned over the years of essentially writing the textbook on this new interface to machine learning. And then includes also some of those interfaces like an integrated notebook environment for when you do want to do very low level custom one-off stuff, but also some much higher level interfaces like those templates I mentioned for labeling functions. There are a number of ways where it can be a truly no-code or very low-code environment for subject matter experts who don't necessarily know how to whip out the Python and solve a problem, but do have a lot of knowledge that's relevant to solving a problem.

• 19:11Chris Benson?

  Gotcha. Actually, to dive a little bit deeper into both sides of that, let's start with the open source and build on that. What would be a typical use case where somebody would go to snorkel.org and do the pip install, read the docs? And what are you offering with that and through those libraries, what's available? And then in a minute, I'll obviously ask you the other side about about taking it to that next level. But if you could kind of give us a sense of what the open source side experience is like, what the benefit of the libraries are, that'd be fantastic.

• 19:42Speaker 00

  Yeah. So if you go actually to snorkel.org, there's a section that is tutorials. And we walk through a number of different, fairly simple, right, but meant to be sort of instructive tutorials for different ways you could use the library. So often, one of the most intuitive places to start with that is on text-based problems. There also are a couple of demonstrations there for how to apply it to images. And then we've got research papers as well we can point people to for working with time series or video or things like that. But one very simple example, one that we actually rely on in our primary tutorial, just because it's very interpretable and almost everyone has the domain expertise necessary for it, is training a document classifier. And in this case, we could say the document will

• 20:25Speaker 00

  be emails, and you want to classify these as spam or not spam. One way you could do this in a sort of traditional machine learning setting is get a whole bunch of emails that are sort of raw and unlabeled, look at them one by one and label them as this one's spam, this one's not spam, that one's spam. And eventually, you'll have the thousands or tens of thousands or hundreds of thousands of emails that you need to train some very powerful deep learning model to do a great job, right? But when you do this process, if you've ever tried to label a dataset, you do find that very quickly, there start to be certain things that you rely on to be efficient, or that are like basically the signs to you for why you should label things a certain way. So, you know, an easy example here might be lots of spam emails trying to sell you

• 21:09Speaker 00

  prescription drugs, right? So you may see, you know, the word Vicodin in an email, and that's pretty clear to you, this is not a valid business email, this is spam, and you can market as such. And you might eventually label over 100 emails that have the word Vicodin, and all of them are spam for approximately that same reason, among other things, there's other content in the email, but that's what tipped you off. And so if you could instead just, you know, one time, say, "And if you see the word Vicodin in the email, good chance that this is a little, you know, more likely to be spam, rather than, we'll call it ham, or not spam."

• 21:41Chris Benson?

  Right.

• 21:42Speaker 00

  You could write that, apply that to hundreds of thousands of unlabeled data points, and get, in one fell swoop, hundreds of labeled examples. And those labels may not be perfect, there may actually be a couple examples in there, you know, some small portion where it actually was valid, someone was asking, you know, "Did you see where my Vicodin was put?" I'm not sure.

• 22:02Chris Benson?

  Yeah.

• 22:02Speaker 00

  I won't guess. But basically, you know, these noisier sources of supervision can be then, you know, much, much more scalable, I mean, much faster to execute, easier to version control and iterate on than individual labels are. And if you can layer a number of these on top of each other, and basically then let them, you know, let their votes be aggregated by an algorithm, you know, one that we developed at Stanford, you now have the ability to get, you know, maybe not 100 perfect labels, but 100,000 pretty good labels, and it takes about the same amount of time. And as we've seen, time and time again, in recent years, you know, the size of the dataset seems to keep winning the day when it comes to getting high performance with these models.

• 22:40Chris Benson?

  Yeah, so essentially, that open source library is helping you scale out your labeling so that you get to the meaningful thing, meaning that you're actually starting to create models faster. So, a way to overcome that.

• 22:52Speaker 00

  That's right. Yeah. You can think of it as, you know, yeah, essentially as a way of building and managing training sets, you know, very, very quickly, often at a, at a much higher sort of rate of, of production, as well as just much larger magnitude.

• 23:07Chris Benson?

  So at what point, if you've been doing this for a while and you've found that utility in the libraries and such, what is a typical scenario that you're finding with customers where they do need to level up? Maybe they've used the open source software for a while. Maybe they had already been doing it even prior to you creating the company, but now it's time. You mentioned platform specifically. What is it that they are now facing that it's a clear step up and they need, they need the enterprise approach at this point?

• 23:33Speaker 00

  Yeah. So I'd say there are a number of, of different reasons for this. And it's a little bit different, you know, which elements of the grab bag are most important for different customers, but I, but I can list a few of those. So, you know, one of the big ones is just the, the guidance. I think with the proof of concept, you know, library, the open source, over the years of using it, we knew sort of what to look for, what, how, how accurate is accurate enough for labeling function? How many do I need? What, uh, you know, how, how should I come up with ideas for, for what a valid labeling function could be? What, how could I integrate external resources that I may have, like a legacy model that I want to improve on? Or maybe ontology that, that belongs to the business that has information in it and how should I integrate that? So there's, there's a lot of, you know, what would otherwise be folk knowledge?

• 24:17Speaker 00

  If you're using the open source that you just only get through experience, that we've been able to really bake in and support in a native, first-class guided way in the platform. And that's a big difference maker for a lot of people.

• 24:28Chris Benson?

  Gotcha. As we're talking here, I'm looking through your website and I went into the platform and I noticed that you, you're kind of segregating out the, the different processes with label and build, integrate and manage, train and deploy, analyze and monitor. Why that particular segregation? What is it that the platform brings to each of those capabilities? How are you guys envisioning this process? And if you have any insight, what is separating that from other options that you may see in the marketplace?

• 24:56Speaker 00

  Yeah. So I'd say that, that, that label and build is probably the piece of that pipeline that overlaps most with the open source in the sense that that's the area where you're going to write labeling functions and then, uh, you know, likely aggregate these right into effectively training labels, confidence weighted labels for these unlabeled examples that you can now train on. That manage inversion piece up next, that speaks to when you have not just a one-off project, when your goal is not just to fill a table in a paper, but really to build something that you have confidence in, that you can come back to, that you can point to in the case of an audit or whatnot. There is extra value in managing all these different artifacts. You've got often many applications that you care about, many teams working on

• 25:41Speaker 00

  it, many different, again, just artifacts that you create, whether that's models, or training sets, or sets of daily functions. So there is an element here that's as well just the data management side of things, and tracking, and inversioning, and supporting all of those types of workflows. On the modeling side, that is entirely unique to the platform with respect to the open source, that we have a bunch of industry-standard modeling libraries integrated with the platform. So if you do want to train a scikit-learn model, sure, or some of the Hugging Face Transformers right there, Flare is another one, XGBoost. So a lot of these libraries, we've unified behind a simple interface, so

• 26:22Speaker 00

  that it can be a push-button experience to try out a number of different things, and hyperparameter tune and whatnot. But with the goal really being of, you'll find most of the time, you'll get the biggest lift by actually improving the training set rather than the model. And so, I guess that actually moves us on to the fourth part, which is analysis. We have a whole separate page with a bunch of different components that effectively take a look at how your model's currently performing, and where it's making mistakes, and why it might be making those mistakes, and then makes concrete recommendations for what to do next. And so, in some cases, it's, yes, actually, your training set looks pretty good. The learned labels that we're coming up with actually line up pretty well with ground truth. And so, if you're making mistakes here, it's probably because it's

• 27:05Speaker 00

  your model now, so you need to try a more powerful model, or hyperparameter tune a little bit differently. And I think that's where a lot of machine learning practitioners naturally go, immediately to the model on hyperparameter tuning, when in reality, almost always, the far larger air bucket is, there are whole swaths of your evaluation set that have no training set examples that look at all like them. There are basically just blind spots that your model has. And now in the platform, you can go ahead and click on that air bucket. Go look at those 20 or 100 or however many examples where none of your labeling functions are applying. So this is not reflected at all in your training set. And write some new supervision that will add effectively examples of that type to your training set so that the next model you train will know something about those types of examples and can improve upon them.

• 27:50Chris Benson?

  Sounds good. I'm also looking at some of the different solutions that you have that are listed from document classification, named entity recognition, information extraction. I'm kind of curious, since as you're looking at this and you guys clearly found a gap in the marketplace from the perspective you were coming from, what makes your approach to each of these problems? Because these are fairly classical problems, sentiment analysis and anomaly detection. What are some of the ways that you think you're adding value that you weren't finding out there? What is that special sauce to some degree that you guys were really looking to introduce into the marketplace with this platform?

• 28:28Speaker 00

  Yeah, I think what really moves the needle is the fact that with this approach and with this platform, machine learning becomes just more practical, more systematic, more iterative. And so all of these different problem types you mentioned, different ones, I think on the website right now, we mostly focus on the text based ones. But again, we've seen these used successfully and will continue to build out the areas for applying this to other modalities as well. But this paradigm is really agnostic to the data modality and most problem types, right? It's at its heart, it is a machine learning problem where you have a training set and you have a model. And when your model is making mistakes, it's often due to what is or isn't reflected clearly enough in your training set.

• 29:11Speaker 00

  And so for any of these problems, there are different types of labeling functions that you write for a classification problem versus an extraction problem or whatnot. But fundamentally, once you scrape off that top layer, it looks very similar. And so this platform really is meant to solve a wide variety of problem types and work in a whole bunch of different industries and verticals and whatnot. Because again, it's sort of under the hood, they're all relying on the same basic fundamental principles about how machine learning works. And then it was with that in mind that we built the platform.

• 29:53Speaker 03

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 30:44Chris Benson?

  So that was a good introduction. I am curious, though, earlier in the conversation, you talked about some of the third-party integrations. And along with that, I'm kind of thinking from a workflow standpoint. So, you know, could you describe a little bit about how you might integrate in with other tools that are widely used within this industry? What kind of integrations do you have? And how that kind of really helps the practitioner get through the process of modeling that they're trying to do?

• 31:10Speaker 00

  Yep. One of the things that we learned from the open source project was the importance of having intuitive, natural modular interfaces to different parts of this pipeline. The labels, certainly, the labeling functions as well, the models, all that. And so we kept that design principle very much in mind as we designed the platform. And we've made sure that every step of the pipeline can be done either in the GUI or via an SDK that we provide. And so that means that you can write labeling functions via these nice GUI builders that we've got, or you can define completely arbitrary black box labeling functions via code in the notebook, push those up, and then they're treated the same way in the platform. Same thing with the training sets. You can create a training set and then go to the models page

• 31:55Speaker 00

  and identify the model that you want, set a few hyperparameters and train it there with a button. Or you can use the SDK to export your training set, train your own model, and then just re-register the predictions, push them back up, just some very lightweight, assign certain UID, certain labels, and then use the analysis page to still guide you. And so it means that we're able to interact with a whole lot of different Customer types and workflows that have different requirements. For some people, no, we really just need to use our proprietary model. We know that nothing works as well as this does. That's totally fine. At that point, you can pull things down from the platform and then push up the results when you're done. For other people, it's actually, we've got a lot of training labels already available from crowd workers, or it's just as a natural part of our product.

• 32:40Speaker 00

  We're always getting feedback that we can use. But we'd really like to be able to be systematic about how we patch up failure modes that we have. And so we want to use the platform, the analysis tooling especially, but maybe also the models. And so for them, they're able to start in that way. So really, it's that any piece of this can be, that's the sort of the test we use for ourself is, can I complete an application in Snorkel Flow without ever opening up that tab of my browser? And the answer is yes, which makes it just sort of ultimately flexible, I guess, platform for integrating with other workflows you may have.

• 33:12Chris Benson?

  Gotcha. So you guys are, even though you're several years, given the work ahead of time, getting into the company, you mentioned that you're about a year and a half into the company's existence, which is pretty early in the lifetime of an organization. Recognizing that it takes time to get things out the door and stuff, what other gaps are you seeing in the industry that is more of that itching that you want to scratch? Whether it be short term or longer term, what are you envisioning Snorkel Flow evolving into? And what kinds of problems that you're not addressing today necessarily, are you thinking about addressing for the future? When you guys are getting together and hanging out and talking about what ifs, what are some of those what ifs that you're willing to share?

• 33:53Speaker 00

  Yeah. So a few different things come to mind. One of them is that, as I mentioned, I guess, a couple of times, there are different modalities to consider. And the way that you write labeling functions over images is fundamentally different than the way that you write labeling functions for text. And so just given where the market pull was initially, we've started focusing on text, but we absolutely plan to bring in some of that other research we've done as time goes on, over the coming months and years. I'd say, in addition, another area that's really interesting to us, where we would have this sort of unique leg up, based on the approach that we're taking, is the monitoring side of things. When you acknowledge that most applications are going to go deploy, it's not, great, I've got my model now, deploy it and set it and forget it.

• 34:36Speaker 00

  Like test distributions change, the world shifts, people talk about different topics, different words get different meanings. Covid was not a part of the discussion a year ago, and now, you know, it's a huge part of the societal fabric of, you know, what gets talked about on social media. So the fact that you do, you know, very frequently need to iterate on your models, improve them, as well as just sort of, you want, you'd like to know, preferably more than just a single, a single number of, you know, the accuracy of my models, like going up or down. It's really interesting to see what types of examples am I starting to get more right or more wrong? What subsets of my data, you know, are diverging basically from, from what they were when I was trained? And so what's really interesting is after you've written these latent functions, they're essentially a whole bunch of different, like,

• 35:20Speaker 00

  hooks into your data set. They, they each observe different slices of your, of your data that have different common properties. And these could effectively become monitoring tools for you, because you can now observe how those latent functions increase or decrease in coverage over time, when applied on sort of the new data that's streaming through your, your, your application. And inform you when, you could basically set up automated alerts showing you now's the time to go and, and update things, or here's some suspicious activity going on, based not just on, did the number go up or down, but like, we're seeing movement in different parts of the space where your model's operating. Take a look. So that, that maybe appeals more to the, the, the technical, you know, nerdy side of things, but I think it's a really interesting problem, one where you've got

• 36:03Speaker 00

  that information, right? You, you have already identified for you these very interesting angles on your problem. And so why not use those to help guide the sort of post-deployment life of a model?

• 36:14Chris Benson?

  I guess at this point, as you were answering that, I want to ask, within the limits, obviously, of what you can share, customers that you have, what are some of the really interesting things that you've seen customers doing with this? That maybe, particularly things that were outside of what you might have expected? The kinds of things, you know, we all have problems. Everybody in this industry has areas of focus that we're addressing. What are some of the things that made you surprised? And, you know, people went, oh, okay, hadn't expected that to, to see that, or just were plain cool? Just something that someone's doing that you're just like, wow, I love having our platform involved in that?

• 36:48Speaker 00

  Yeah, two things that I've found personally very cool. One of them is the privacy preservation aspect of this approach. That was not necessarily a top priority or top of mind when we were developing these techniques at Stanford. It was often on problems where it's just, I'm trying to get a good result. I want a high quality. How can I get high quality? But it's been really cool to see different companies that have the very desirable goal of, we'd like to have our data being seen by fewer humans. We'd like to have fewer people reading your emails, fewer people seeing your medical or financial records. And how can we do that while not sacrificing the quality of our machine learning models? And so it's been really interesting to see them, and working with them, coming up with

• 37:29Speaker 00

  these setups where now they can take a very small approved subset of the data to give them ideas for how to write labeling functions or to label like a test set to give them a sense of overall how is quality. But then the vast majority of their data never gets seen by a human now. They can take these programs they've developed to go label those automatically, use them to train a model, and then get back just sort of the final weights of the model. And it's really neat to see, and I'd love to see that sort of thread continue because, I mean, not just for the privacy preservation standpoint, but also, we keep seeing articles about the PTSD almost that you get as an annotator over these awful domains. I mean, you hear about some for social media. There are some horrendous ones. Actually, even during the Stanford days, we worked with DARPA on a project for

• 38:13Speaker 00

  human trafficking, where in their case, it was more out of, I think, necessity of keeping up with a very rapidly moving environment where it's this adversarial setting, so your training set's always losing its value because things are always changing. And so they needed to be able to create training sets very quickly, and they did with snorkel, which was cool. But also conveniently now, there are that many fewer people who need to spend a day sitting in front of these awful human trafficking ads. So I think the privacy standpoint is very cool. I think another interesting application we've seen was that we had one customer who, and I'll try to appropriately obfuscate here, but they had an application that was affected, we'll say, by COVID. When you suddenly have the stock market plummeting, and there are certain risks associated with

• 38:56Speaker 00

  that for different businesses, and we were in the middle of a POV engagement with them, sort of their test running the product to see how it worked for them. And they came to us and said, okay, here's actually, this was not part of our scoped, you know, work, but this suddenly matters a lot to us and our typical process would take about a month. Do you think you could help us? Could we try and use snorkel to get some result faster? And so, since it was very early on and we hadn't necessarily had a lot of time to train them yet using the platform, he said, sure, we've got some ideas, you know, give us a sec. Threw three of us in a war room for the day, ordered some burritos, you know, and hacked away. And by the end of the day, we were able to extract the terms that they needed with over 99% accuracy on their application. And that was achievable with a model that was trained on, you know, tens of thousands of

• 39:41Speaker 00

  examples, which we didn't need to label. We were able to quickly come up with, what are the generalizable, you know, rules and principles here that we could use to create a training set to train a model that now, you know, can handle edge cases and things much better than these rules and then get the high quality that they needed. So that sort of, you know, live action, the nerd saved the day, right? Kind of moment.

• 40:00Chris Benson?

  No, it's a good story.

• 40:01Speaker 00

  Super cool to see.

• 40:02Chris Benson?

  You've raised several interesting points there, you know, one of which is the fact that in real life, as this technology is more pervasive, that these dynamic, ever-changing data sets are a reality we have to contend with. I mean, are you seeing the industry getting more flexible at large? In addition, you know, obviously you guys are, you know, in terms of thinking about the fact that that's something that has to be accommodated, but I would expect that that is something that has to be addressed more and more. Do you have any insight into kind of, or any thoughts into where we're going in terms of us moving along this curve from these, you know, static labeled data sets that we were talking about historically at the beginning of the conversation to this dynamic, you know, especially since COVID has struck, the ever-changing world on a day-to-day basis.

• 40:47Chris Benson?

  You know, what's that trajectory look like? And how are you guys preparing for that?

• 40:52Speaker 00

  Yeah. So I think we're definitely seeing a increased awareness of some of these issues. I think a lot of companies are still trying to figure out how to address it in the right way. We see companies realizing that, you know, schema lock-in is becoming this problem for us because real problems, you know, change. Our objectives change. We learn more about the problem. What we thought was a Positive or negative classification problem is actually positive, negative or neutral. And then all of our, our old labels are garbage now because I don't know where the neutrals are and the positives and negatives. So people are being burnt by some of these problems. And so I think that's part of the reason why we've had such early success with sort of inbound interest more than we could even handle at first because people are aware now of some of the costs that come with machine learning. The promise of machine learning is very much being broadcast and how it's seen of the

• 41:37Speaker 00

  future and solves a lot of problems. But there do end up being these very practical, I won't say necessarily limitations, but you know, gotchas or you know, costs really, right? That you need to be aware of. So I see this reflected a little bit in the way that companies are starting to prioritize more the ability to see the like, where did my model learn this? Like that auditability kind of of

• 41:59Chris Benson?

  Yes, we're kind of touching on kind of AI ethical issues. You've talked about auditability and privacy and such.

• 42:04Speaker 00

  Yeah.

• 42:05Chris Benson?

  Totally, totally see that as you're kind of maturing your way through the process here.

• 42:09Speaker 00

  Exactly. So yeah, so I mean, they realize that that's important in a way that they maybe didn't before. I think they're also realizing just from an economic standpoint that there's this, that the training data is not a one-time cost. This is a, a capital expenditure. This is an asset that loses value. There is a, a half-life to these things. And so you start seeing these, you know, ongoing, regular cyclical budgets to get the training data even just for a, a single application, not, we need more data to, to train more models for more applications, but to keep this application fresh and alive.

• 42:38Chris Benson?

  That's a great insight right there.

• 42:40Speaker 00

  Yeah, well, it's super interesting and it changes the way that you account for the, the cost of, of different applications you might use, right? Because there's a certain way that you maintain imperative, we'll call it software 1.0. And there's a different way that you maintain this machine learning-based software 2.0, you know, way of solving a problem. It's something people are learning and I think that's part of all interesting part of the conversations that we're having with different customers as they realize how this can maybe change the way that they approach their machine learning stack in general.

• 43:08Chris Benson?

  Okay, so I guess as we wind up here, I'm finally getting to ask what is always my favorite question. Anytime we're getting to talk to someone such as yourself and that is, you know, blank sheet of paper, what are you excited about right now in the space of machine learning and AI? You know, what is the thing that has captured your imagination? Whether it's work-related or whether it's not work-related and just something cool out there, what's got you going? That's the thing I'm really interested in, in tracking either on my own or through the company or whatever. What's cool?

• 43:38Speaker 00

  That's a very good question.

• 43:40Chris Benson?

  There's a lot, I know.

• 43:42Speaker 00

  So many things. I'd say there are a number of areas that are super important, super hard, but super important. And I'm glad to see that they're getting the attention that they deserve, at least that we're trending the right direction. And that stems around, I mean, the privacy, the fairness, the bias, a lot of that, I think it's just super hard. If anyone says that they've got a solution to that problem, I'd be very dubious. But I think we are, you know, marching toward progress there. And that's something that I'm certainly going to watch with great interest and hope that we can be a part of the solution there. That's one piece. What I think maybe a little closer to my personal, you know, research agenda and history, a lot of that centered around how we you get signal from

• 44:26Speaker 00

  a person into a machine, right? And so a lot of my my research through the years has been seeing kind of how high up the stack can we go? Like, there's this figure in my dissertation that compares basically the the computer programming stack to like the machine learning stack. So, you know, computer programming, computers run on these ones and zeros, right? They run on individual bytes and bits, but nobody writes ones and zeros code. You write in higher level interfaces like a C or even like a SQL or something that compile down sometimes multiple times into this low level code that you're then going to actually run on. And I'd say similarly, machine learning runs on individual labeled examples. That's how we train it. That's how we express information to it. But it feels fairly naive, actually,

• 45:08Speaker 00

  to one by one, write these ones and zeros, write these truths and falses on our individual examples. And so I think that there's a lot of really interesting things that can be done around higher level interfaces of expressing expertise that then, you know, in various automated or just sort of assisted ways, can eventually result in the training sets that have the properties you need to actually communicate with your model, you know, use the compiler, right? Essentially, the optimization algorithm that's in place to transfer that information. So that's a fairly high level description. But I think there are interesting things yet to be done there.

• 45:41Chris Benson?

  Well, thank you for sharing that. I appreciate it. Braden, thank you so much for coming on to Practical AI. It was a great conversation and looking forward to our next one. Looking forward to having you back sometime soon.

• 45:51Speaker 00

  Absolutely. Thanks for having me.

• 45:53Chris Benson?

  Thank you.

• 45:57Speaker 03

  This is our final episode of 2020. We're taking a couple of weeks off to relax, re-energize, and gear up for the new year. If you want more practical AI goodness during the break, I personally recommend the Waymo episode with Drago Anglov, the one where Peter Wang drops a bunch of history and knowledge on the guys, and the human-compatible AI episode with Stuart Russell. I'll link those three up in the show notes for you. Practical AI is hosted by Chris Benson and Daniel Whiteknap. It's produced by me, Jared Santo, and our music is brought to you by the Beatfreak Breakmaster Cylinder. We have awesome sponsors who get it. Thanks to Fastly, Linode, and Launch Darkly for their support. That's our show. We'll talk to you again next year.