Killer developer tools for machine learning

Weights & Biases https://www.wandb.com/ is coming up with some awesome developer tools for AI practitioners! In this episode, Lukas Biewald describes how these tools were a direct result of pain points that he uncovered while working as an AI intern at OpenAI. He also shares his vision for the future of machine learning tooling and where he would like to see people level up tool-wise. Featuring: Lukas Biewald – Website https://lukasbiewald.com, GitHub https://github.com/lukas, X https://x.com/l2kChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Weights & Biases https://www.wandb.com/Blueriver case study from W&B https://wandb.ai/blueriver/blog-post/reports/AI-For-AG-Production-ML-for-Agriculture--VmlldzoxNTA1MDg?accessToken=ps435xipm72k06ljgzl0442gmr64tj7ybpc2xublpxgymsrgiy3rug7egcuh6spoW&B gallery https://wandb.ai/galleryUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I feel like with deep learning, your runs often take days, right? So you want to get a sense for if they're working right away. And you want to really have a good record of everything that happened. You know, from like system metrics to like exactly what was the last commit that went into your code to. And you know, everyone has these like notebooks, you know, like in their Emacs or VS Code, or like maybe a Google Doc if you're really advanced, you know, where they're just kind of typing like, okay, here's all the things I did, you know, and here's what happened. Or in file names, extra long file names.

• 0:28Chris Benson?

  Yeah, that's a really good point as well as the fact that as you move through the different phases of your workflow and you know, you're kind of, you know, doing your research in a notebook and then you need to deploy it as a software component and stuff. There is still friction in the process in terms of moving through each of those very distinct phases of trying to get your workflow from the very, very beginning to the very, very end and deployed. And I think people still, everyone kind of customizes that quite a lot. That's a, there's another opportunity out there for someone.

• 0:57Speaker 02

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com/changelog. This episode is brought to you by Digital Ocean. Droplets, managed Kubernetes, managed databases, spaces, object storage, volume block storage, advanced networking like virtual private clouds and cloud firewalls, developer tooling with a robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses.

• 1:36Speaker 02

  Join over 150,000 businesses that develop, manage and scale their applications with Digital Ocean. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 2:08Chris Benson?

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:34Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whiteknecht. I'm a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a principal emerging strategy, technology strategist at Lockheed Martin. Every time, Chris, I've got to get those words in the right order. It's an impressive title, I have to say.

• 2:57Chris Benson?

  It's a terrible title. It's a mouthful. So, yeah, it's funny. I'll go ahead and make fun of him. I'm hoping he doesn't listen to this. My boss will be introducing me in a meeting, and he can't even say it. And I, it just makes, you know, I'm like, no one's like, who's this guy that joined us?

• 3:11Daniel Whitenack?

  Yeah. Yeah. I mean, you could do an acronym that's like PET, PET Strategist. That sounds also very animal friendly, you know, P E T.

• 3:21Chris Benson?

  That's very applicable. Yeah.

• 3:22Daniel Whitenack?

  Principal Emerging Technologies. Yeah.

• 3:24Chris Benson?

  Exactly. I love animals for, for anyone that, yeah. So, anyone who's listened for a while, heard me carry on about it.

• 3:31Daniel Whitenack?

  So, how are things going for you, Chris? You excited about a crazy week in, in the US at least?

• 3:37Chris Benson?

  Yeah. So, as we record this, it's the day before Election Day. So, you know, I guess everyone is holding their breath, no matter what side of the aisle you're on, and we'll see what happens through the rest of the week and, and maybe, maybe beyond, depending on how things go. I'm remembering the, the election of 2000, and you know, it went on for months to get a resolution. So, we'll see. It'll be interesting.

• 3:59Daniel Whitenack?

  We'll see. Yep. Well, we decided that at least, you know, on my end, I wanted a break from anything political and a little bit of a chance to be practical as we are here at Practical AI. So this week, we've asked Lukas Biewald, who is founder and CEO at Weights and Biases, to join us and talk about a lot of the things that they're doing in terms of developer tools for ML and maybe some other stuff. So welcome, Lucas. It's great to have you with us.

• 4:29Speaker 00

  Thanks so much. Great to be here.

• 4:31Daniel Whitenack?

  Yeah. Could we maybe start by just having you give us a little bit of information about about your background and how you got interested in AI and computing and ended up where you're at now?

• 4:43Speaker 00

  Yeah, totally. I mean, I always was interested in AI. I mean, ever since I was a little kid writing software to play video games against me. And then, you know, I wanted to go into academia and study AI and machine learning. And I got as far as a master's degree and did some research. And then I left to do more practical AI. So I love the title of this podcast. This is, I mean, this is like back in like 2003, 2004. And it was super fun because it was like, you know, the really early days of ML systems and deploying, you know, big ML stuff in production for Yahoo back when that was a super relevant

• 5:23Speaker 00

  company and then at some startups. And then, you know, I got really interested in data labeling, which, you know, you'll probably understand just because it felt like, actually in research and in industry, it felt like the quality of the data labeling completely determined, you know, the quality of the ML systems deployed.

• 5:41Chris Benson?

  Oftentimes still does.

• 5:42Speaker 00

  Yeah, totally. And so I started a company originally called CrowdFlower that was, changed its name to Figure Eight. We were really maybe the first data labeling company for ML practitioners.

• 5:55Chris Benson?

  Yep.

• 5:55Speaker 00

  And I ran that for about 10 years. It sold to a company called Appen, which is a big Australian public company that also does data labeling. And then in that process, I saw a kind of deep learning take off and I saw, you know, suddenly all these applications, these very awesome practical applications across tons of industries. Like when I was first coming out of school, it sort of felt like all the jobs available was, you know, ranking ads or ranking, you know, search terms and, or, you know, going to go into Wall Street and being quant. And I mean, those are horrible jobs, but I feel like the, the really practical stuff is more exciting to me. And so, you know, it felt like there's just so many real world applications happening. And kind of

• 6:37Speaker 00

  what I saw was, you know, the tooling just seemed nascent. And it felt like a lot of the developer tools that we all take for granted, don't quite translate into, you know, machine learning applications because inherently it's statistical and I could go on and on why. But, you know, basically they, they don't seem to translate. And you know it because you see ML practitioners not using a lot of software best practices. And so me and my same co-founder from Figure8, and actually a third co-founder, we decided to start a company that was just focused on making great tools for ML practitioners.

• 7:08Daniel Whitenack?

  Yeah. And we should say too, so you, you mentioned the, the name of this podcast, but you, you've also hosted a podcast with a, with a pretty great name, Gradient Descent, spelled D I S S E N T. So, so great work with the, with the naming there.

• 7:28Speaker 00

  Excellent.

• 7:30Daniel Whitenack?

  Yeah. So definitely check out, check out that podcast as well. We'll put the link in the show notes. Has it been interesting in terms of, you know, after kind of being a practitioner, being a founder for a while, has it been interesting having conversations on, on your podcast with, with other practitioners? Have there been sort of viewpoints that have surprised you or insights that you've gained that, that maybe you didn't expect to as, you know, as you started, as you started having some conversations with people, you know, in, in various parts of the industry?

• 8:03Speaker 00

  Yeah. When you run a company, especially as I think as a founder, CEO, most of what you do is you, you talk to customers. It's a huge part of, of what you do. And so the podcast is really kind of just recording the conversations that I was having and giving me an excuse to kind of ask a little bit higher level questions to people. You know, you, we, we kind of get into the details really quick, you know? With the user, but we thought it would be fun to sort of step back a little bit and just have kind of open-ended conversations, which you're kind of doing anyway when we're starting Weights and Biases, because at first we weren't really sure what the first best tool would be to make. And so we had a lot of these and they were so much fun. We kind of thought, you know, maybe we'll record these and other people will find them useful because we were citing them a lot. But I mean, I would say it's been absolutely fascinating

• 8:47Speaker 00

  and it's been really fun to step way back and ask questions because you hear surprising stuff. Like I'll say, you know, Nata Kincirla from Lyft was a long time, a customer of Weights and Biases and he's a great guy. But he was basically like, you know, ML is just the same as anything else. It's just a discipline of engineering, which I thought was a bold, you know, bold statement from a guy running, you know, the level five team there. Or I remember Jeremy Howard was like, basically, Python can't possibly be the future of machine learning, which is another kind of controversial topic. So, you know, it's funny, you ask people and usually they say the same stuff, but then every so often you get these gems and you really think, huh, interesting and dig in.

• 9:26Daniel Whitenack?

  Yeah, yeah. Statements that would provoke a lot of discussion at any conference.

• 9:32Speaker 00

  Yeah, yeah, exactly.

• 9:34Daniel Whitenack?

  Anything like that. It's like people just assume certain things like, oh, you're going to use Python. That's kind of what people do. But people really thinking about the foundations of those things and why we're using those things is really interesting. We've had those experiences here as well, for sure.

• 9:49Speaker 00

  Cool.

• 9:50Chris Benson?

  So I'm actually want to take you back for a moment to your introductory bio a little bit and hit a particular point that you kind of went right over there. And when you were talking about founding Figure8, which used to be, I can't say it now, CrowdFlower, I apologize, which is really how I think of it. Because I've worked for several companies which have been your customers.

• 10:10Speaker 00

  Oh.

• 10:11Chris Benson?

  So there you go. There you go. I'm just kind of curious, at that point in time, relatively early, certainly based on where we are with machine learning today, and Figure8 is a well-known name in this industry, and you have a lot of big customers and stuff. But going back to when you founded it, at that moment in time, what was it that made you say, this is the thing I'm going to go do? What was the problem that you saw? How did you want to solve it? You know, I'm kind of, if you can take yourself back to that year and tell us a little bit about what that moment was like.

• 10:44Speaker 00

  Yeah, totally. Well, I think it started in my master's program and grad school. And I was doing research on a thing called word sense disambiguation, where you try to figure out if like, you know, plant means like the power plant or like the living plant, right? Kind of classic problem in NLP. And I spent all this time with a data set that's called, WordNet is the ontology, but there's a data set of documents labeled with the meaning into WordNet. And it turns out word sense disambiguation is kind of a tricky, it's a tricky thing to know really what the senses of a word should be. And I think my personal opinion is WordNet makes way too many senses. So like, they'll have like 15 meanings of plant. Like, there's sort of two main ones, like I said, you know, like plant, the living thing and the power plant.

• 11:29Speaker 00

  But they'll also have like, you know, plant a block into someone. Like in football, it's like its own meaning, which is like questionable. And just all these kind of like weird meanings.

• 11:37Daniel Whitenack?

  Yeah, like domain-specific or like very niche.

• 11:39Speaker 00

  Metaphorical. I mean, it's almost like, it sounds so obvious, the senses of a word when you talk about those two. But when you think about like planting a block or planting your foot into the ground, it actually gets a lot trickier of like what meaning is different from what. And you kind of only learn that once you've spent a long time with that ontology. Anyway, what happened was I was finding all these patterns in the data set. And I was really excited to like publish a paper. And then I realized that actually what I was doing is not finding interesting patterns. I was basically figuring out who the annotators were. So like, you know, I was doing this sort of like cross-document classification. I was realizing if you could look at the classifications of one part of the document, it can help you the classifications of the other part of the document.

• 12:24Speaker 00

  And I thought what I was doing was topic detection. But then actually what I realized that I was doing is I was basically doing annotator detection.

• 12:31Daniel Whitenack?

  Right.

• 12:32Speaker 00

  And it was so like heartbreaking because I had spent like months, like, you know, doing this research and it seemed like really promising. And then suddenly I realized that I was just finding this like artifact of the labeling. And it like the label shouldn't really matter, you know, for like, you know, like documents. But of course it does. And then... So that was really like top of mind. And then when I was working at Yahoo, I was actually switching their search engine from kind of a rule-based system to an ML system. And I was doing it in all these different languages. And it was funny because like each country that I'd go into, sometimes they'd be super happy with me and sometimes they would be like really pissed off. And it had nothing to do with me because I was doing the same algorithm in each country. But it was totally based on how seriously they took the labeling process.

• 13:16Speaker 00

  Right? So like some countries, I mean, I guess now I could probably name names like, you know, he took like China was totally not taking the labeling process seriously. They're just trying to do it as fast as possible. They had all these like mislabelings. And so it was really hard to get a system out where like Japan was was taking it very seriously because they had like, they had a kind of a, for historical reasons, they just had a bunch of labelers that really cared. And they were labeling it much more carefully. And so it was much easier to launch something there. And I was sitting there like, hey, I'm, I'm this, you know, I'm the ML guy. Like, you know, I'm doing the same thing in each country, but you seem to have very different like takes on it. And so it just made it so obvious that the labelers are kind of what matters. And so I sort of felt like if you want to make ML real, that's something I've always really cared about.

• 13:59Speaker 00

  Like, it's so exciting when you get something really deployed for a real application. If you want to do that, most of what you should spend your time on probably is the labeling process. I think at the time, what was happening a lot was ML teams were kind of relying on some other function in the company that was already kind of doing a lot of data collection to do the labeling. And so the ML teams would be kind of divorced from the labeling process. And so I think what, you know, Crowfler at the time then became Figure8, what it really did was it gave the ML practitioners control of the labeling process, right? So it wasn't like an outsourcing firm that is really like finance using. And then you're sort of like, you know, kind of like sneaking your project like into their budget. It's like now you actually are running a labeling process and you're really looking at the data

• 14:43Speaker 00

  and you're like writing the instructions. And I really deeply believe that that's essential in almost every case for making a machine learning thing work in production.

• 14:53Daniel Whitenack?

  Yeah, it it almost seems like, you know, when you were doing your research work, you really found these issues with the foundational, the underlying data itself and the annotation process. And then, like, assuming you have some data to work with, it seems like the next sort of level up are these developer tools that were either nonexistent, or really rough, or not being used, or something like that. And that seems like at the level of of weights and biases. I'm curious if that's how your thought process like went, because it seems like, you know, it seems like a natural progression to me, but I'm not sure if if, you know, after you had thought about the data, the data and annotation problems for a while, you just, you know, your sort of next pain point or your next

• 15:38Daniel Whitenack?

  obvious layer in that stack was that layer up of developer tools?

• 15:42Speaker 00

  Well, actually, I'll be a little more specific, since we're on a long podcast. I never really kind of skip over these details. But honestly, what happened really was was I got kind of worried when I was running Figure Eight that I was getting out of date. Like, I remember at my first job, there was a big fight between rule-based systems and machine learning systems. And I remember kind of looking at the, you know, mid 30s guys or 40 something guys who had, you know, kind of invested a lot in rule-based systems. And they were kind of like, oh, this machine learning stuff is stupid. Like, you know, it's like hard to scale, requires a lot of training data, like hard to deploy. And I remember just looking at them thinking, you guys are like idiots. Like, you've just like committed yourself to this one thing, and you can't see that this new thing's better,

• 16:24Speaker 00

  right? And I I kind of found myself, I hadn't done a lot of, you know, training models in a long time, you know, I was running a company. And I was kind of saying the same stuff about deep learning. Like, I was like, you know, like, is this really better? Like, it seems really hard to deploy. And I was kind of looking in the mirror like, man, I am like out of date. Like, I don't know, I don't really know how like what TensorFlow does. Like, I pretend I do, but I haven't like messed with it. I was feeling really bad about that. And then I I got myself in a like an internship at OpenAI where I worked for like, you know, a really, really smart 24 year old for a little while. And I just was like, just make me do all your dirty work. I just want to kind of get back into

• 17:04Speaker 00

  doing something real. And it was kind of a shocking experience, I think, because I think the tooling had been bad for ML. But it's like strikingly, shockingly bad for, for deep learning, right? I think it just sort of exacerbates, you know, the problems even more. And so, you know, quickly like my entrepreneur brain is just like, I have to, I have to build tools to, to fix this up. And so I didn't last very long as an intern because I just really wanted to fix these underlying problems for, for everyone else. But I'd love to kind of go back. I mean, training models and doing research is so fun. I guess I just, I don't stick with it very well.

• 17:38Chris Benson?

  Before we go one from that point though, I just wanted to draw, I mean, it's very insightful of you to recognize that, that thing that you had noticed before about, you know, the, the older, you know, contingent at the work and then recognizing that you were in danger of doing that yourself. And I know I'm definitely older than Daniel and have certainly found myself in the same position when I was younger saying, oh, look at those older people that are, and then when I got older, I did that and I recognized this stuff. And I think that's a great lesson for people out there is don't ever do that. Don't ever, no matter what age you're at, don't ever lock down on your technology. Don't ever be in love with it because it's going to change. It's going to change and there's going to be a new best thing out there.

• 18:18Daniel Whitenack?

  Yeah, I think that there's, there's definitely a balance. I mean, maybe I'm saying this because like I'm too, too stubborn, but I think there's definitely a balance because there's like the side of things where like, if you're not passionate enough about something and, and stick with something long enough, it takes a good amount of time to like, really like make a difference in a certain area. But then also like you have to be willing to sort of jump ship onto other things occasionally. So yeah, I definitely think that there's a cycle with that and.

• 18:49Chris Benson?

  It's hard to do.

• 18:50Daniel Whitenack?

  It is hard to do.

• 18:51Chris Benson?

  It's hard to find that sweet spot of committing enough to becoming very good at what you're trying to get done at the moment and knowing when to shift.

• 18:59Daniel Whitenack?

  Yeah.

• 19:00Chris Benson?

  It's not an automatic thing. So I just wanted to draw that out. It was a great point you made there.

• 19:03Daniel Whitenack?

  Yeah, I'm curious if you could go into like, um, we do like to get into the weeds sometimes here, but like in that state, when you were doing that internship with OpenAI. And when you're talking about the tooling, quote unquote, like, what was your workflow like during that time in terms of training models and that sort of thing? Like, what were the tools at your disposal for you to use?

• 19:26Speaker 00

  Totally, yeah. I mean, it was funny, actually, because I also was like, you know, kind of working for a 24 year old, 25 year old. I felt really embarrassed that I was like using Emacs, which like, not many people think that's kind of cool, but like, I was like, I'm gonna use VS Code, like, I'm just gonna use the latest stuff. I'm not gonna, you know, I can't be using these crap old tools.

• 19:43Chris Benson?

  We're not offending any of the Emacs users out there.

• 19:45Daniel Whitenack?

  Right, right.

• 19:47Speaker 00

  I gave up my .emac files. Pretty crappy.

• 19:49Daniel Whitenack?

  Now it's super, you know, it's hardcore, you know?

• 19:52Speaker 00

  I know, now people think it's hardcore, but, but come on, like, VS Code is good. I don't know.

• 19:56Daniel Whitenack?

  Yeah, VS Code is good, yeah.

• 19:57Chris Benson?

  A lot of people have moved off of them and Emacs finally, you know, after resisting for years to VS Code. I mean, it's made a big difference in the industry.

• 20:05Speaker 00

  I strongly recommend it.

• 20:06Chris Benson?

  I, that's what I use. I moved away from the other stuff into that, so.

• 20:10Speaker 00

  There's actually, there's another one where you, you know, you build up these like, you know, your, your .emac file. You don't want to give it up, and my hand's still like...

• 20:16Daniel Whitenack?

  You've invested so much.

• 20:17Speaker 00

  Yeah, yeah, exactly.

• 20:19Daniel Whitenack?

  Yeah.

• 20:19Speaker 00

  But, yeah, so OpenAI was unique in that they had a ton of compute resources, but at the time, it was actually, basic stuff like just trying to train across multiple GPUs was an enormous pain. I mean, even setting things up is, was painful, I think still is way more painful than it needs to be. Like, just like, you know, you buy a GPU at the store and then you want to train a model on it. I mean, expect to wait a day, you know, while you, you go through the latest iteration and then like, you know, you have to find something that's, that came out in the last like three or four months, or else it's, you know, the instructions are gonna be wrong. And this is totally Nvidia's fault.

• 20:57Chris Benson?

  Look at it squarely there. But yeah, I mean, the, the startup workflow definitely has room for improvement, absolutely. I've found that as well over time.

• 21:07Daniel Whitenack?

  And you'd love for those artifacts that like, or implementations from like, a year and a half ago, it seems like, this is exactly what I want to do, but... Like I haven't got to start over because it doesn't, you know, it's some compatibility issue or something.

• 21:22Speaker 00

  Yeah, like I thought, you know, I thought I would never see like a linker error in my life, you know, after like 2006, you know, but like, you know, now you see him when you're just trying to, you know, get your situation set up. And, you know, I think like collabs and stuff have really helped with this and made the onboarding smoother because even this is, you know, three, three years ago, maybe.

• 21:40Chris Benson?

  Yeah.

• 21:40Speaker 00

  Two, three years ago. So look at me, I'm out of date again. But this wasn't even the, the problem that I was trying to solve with advices initially. And actually, the thing that, the thing that was really striking to me was just that you, um, it's really hard to go back and, um, look at a run that you did in the past. And I mean, this is something that's always been true, like any kind of long running, um, thing. But I feel like with deep learning, your runs often take like days, right? So you want to get a sense for if they're working right away and you want to really have a good record of everything that happened, you know, from like system metrics to like exactly what was the last commit that went into your code to, uh, and you know, everyone has these like notebooks, you know, like in their Emacs or VS Code or like maybe a Google Doc if

• 22:22Speaker 00

  you're really advanced, you know, where they're just kind of typing like, okay, here's all the things I did, you know, and here's what happened.

• 22:28Daniel Whitenack?

  Or in file names, extra long file names.

• 22:31Chris Benson?

  Yeah.

• 22:32Speaker 00

  Yeah.

• 22:33Chris Benson?

  You know, that's a really good point as well as the fact that as you move from, through the different phases of your workflow and, and, you know, you're, you're kind of, you know, doing your research in a notebook and then you need to deploy it as a software component and stuff. There is still friction in the process in terms of moving through each of those very distinct phases of trying to get your workflow from the very, very beginning to the very, very end and deployed. Um, and I think people still, everyone kind of customizes that quite a lot. That's a, there's another opportunity out there for someone.

• 23:03Speaker 00

  I think there are a lot of opportunities right now in, in the like tooling space for, for ML. I think it's like, you know, companies aren't really seriously doing it. And I think every step is basically broken, right? Like, you know, from like, you know, you visualizing your data to tracking your data, to tracking your experiments to like. You know, deciding like gating deployments with like continuous integration to production monitoring. I actually think all of these things are huge pain points. And I think there's going to be a lot of really interesting companies that spring up solving them in different ways.

• 23:31Chris Benson?

  I think so.

• 23:32Daniel Whitenack?

  I'm curious, you talked a little bit about like, you know, wanting to go into this internship to figure out, or once you started, you know, figuring out that there was this problem, like, you had to start somewhere with like the tooling, like for a specific thing. Where, where did you end up landing in terms of like the particular rough edges that you wanted to smooth off in the tooling? Like, where did you focus when you first started thinking about weights and biases?

• 23:58Speaker 00

  It's funny, so the thing that we first started focusing on is now the thing that weights and biases is known for. I sort of expected to do, you know, a couple of different iterations and kind of like hit walls. But I think this is an example of like, you know, the second time you do a company, when you've kind of been in the space for 10 years, you have better instincts about what the needs are. So the, the thing that, that I started with, which is really what weights and biases is known for, is essentially tracking, tracking your training. And it was basically like, you know, TensorBoard was doing this, but it really only worked with TensorFlow at the time. And it was hard to look at multiple runs, which is really what you want. And so, you know, they, they sort of have a way of doing, putting hyperparameters into the name of the run, which I think is kind of awful.

• 24:39Speaker 00

  I really don't recommend that. And then it's clearly not designed to show you like runs in the context of other runs, right? Like, you know, the fact that like your model has like this accuracy after like eight epics. It's kind of not that interesting. Like, what's more interesting is like, how did the accuracy change from the previous thing you did? And then also, you know, being able to dive in and look at exactly what was that previous thing you did? Like, that, that was sort of the core of the beginning of weights and biases. And still, I think the thing that people appreciate most about us is like, Saving all the metrics of a run, you know, the performance metrics, the system metrics and the sort of like accuracy loss and those kinds of performance metrics and being able to quickly compare them with lots of other runs.

• 25:19Chris Benson?

  Could you describe that a little bit about how, you know, that how that contextual comparison is realized and how that's different from previous tools people might have used?

• 25:29Speaker 00

  Totally. And I think TensorBoard is the best comparison, but you know, it's like, I mean, let's pick a concrete example, like say you're trying to train a self-driving car. That's right. So say you're doing perception of self-driving car. So you know, you're looking at these images and you're doing semantic segmentation where you want to like label each pixel with what's in that pixel. Is it a pedestrian or is it like a road or like what's going on? Right? So, you know, the first cut of what you can do is as the model trains, you can look at the loss function, right? And you know, whatever that loss function is, you can see if it's, you know, going up or down and hope it's going down because it's what you're optimizing. And you might want to compare that to like other runs you did, maybe with different hyperparameters or different state of the code, but say like you have one with a higher learning rate and a lower learning rate, you know, you want to see which is better, right? Now, what turns out, the second you start to do

• 26:12Speaker 00

  this is you realize that better or worse is more complicated than you think, right? So like,

• 26:16Chris Benson?

  Yeah, how do you define that?

• 26:19Speaker 00

  Exactly. So, you know, first you're looking at the loss and you're like, oh, great. This run, this run has like better loss, right? But then, you know, it actually might be that when you look at accuracy, which is a more actionable thing, it completely flips, right? So like the, the lower learning rate has actually higher accuracy at the end, right? But then you might say, well, what is accuracy really like? You know, like maybe it's just predicting that everywhere is a road, you know, and just ignoring the pedestrians because there's not many pixels that correspond to a pedestrian. So now you say, you know what? I don't want to have anything with like under 99.9% pixel accuracy on pedestrians, right? And so now, you know, maybe you didn't check that the first couple of times you, you trained your model. So you want to go back and look at what those were. And so like, you know, the requirements as you, as you do this in the real world,

• 27:01Speaker 00

  they keep changing as the businesses change and you realize, you know, that, that things are different. And so fundamentally, what, you know, the, what the, the kind of core weights and biases tool does is it, it tracks all this stuff for you and it shows it to you in graphs. But then why you might really want this is like, you know, six months from now, you've deployed this model into a car and the car crashes, right? Now you really want to know, okay, what was that? What was that model trained on? Like, what was happening in the training that made you think you could deploy it? You know, like, where could things have gone better? Right? And like, if you don't do this in like a systematic way, like with weights and biases, you can imagine it's gonna be really hard. Like, if you're just tracking like notes, you know, in your Emacs file, it's probably not gonna include all the detail that you wish you had. So what we try to do is capture lots of stuff passively.

• 27:46Speaker 00

  Right? And we think that like when you capture like everything passively and you don't put a burden on the person training, you get a lot more useful information. Because usually you want this information like after the fact.

• 27:59Daniel Whitenack?

  And when you say passively, you're meaning like you don't want a user to decide like, you know, weights and biases save this parameter and save this parameter, but not like these other things. You want to kind of enable or disable rather than like having a user like click a bunch of boxes for all the things that they want to, that they want to save. Is that what you mean?

• 28:22Speaker 00

  Yeah, I'll give you an example. I mean, and so just fundamentally, the way this works is you import our library and then our library basically collects lots of, you know, system metrics and other metrics as it runs, kind of similar to TensorBoard, but we collect a lot more stuff. I'll give you an example, the sort of passiveness of it. Like, you know, one thing you should probably do before a big training run is like commit your code into Git, right? And so like, you know, we could just like connect each training run you do to the commit shot. And we do do that if you want us to. But one thing we found, right? Because we're talking to people all the time is like, you know, most of the time, people don't actually commit their code before it runs, right? So, you know, we capture not just the commit shot, but also a diff against the latest commit shot. So that way, like every time, you know, every time you do a training run,

• 29:05Speaker 00

  we keep track for you the exact state of the code, the way it was when it trained, right? And that way, you know, like a month later, you know, you want to go back and like look at something and be absolutely sure what the state of the code was. You can do that, but you never need to think about it, right? Like, once you turn that on one time. Now you'll be sure that every single time your model trains, there's a snapshot of exactly the state of the code. And you might say, well, that's like a waste of space. But like, come on, like, if you look at the size of training data sets, you know, they're going to be like petabytes, right? So, you know, you could do this kind of all day long, like capture every metric you might care about, all the state of the code, everything going on, like, you know, exactly what kind of hardware it was on. You can capture all this without really making a dent into a modern machine's storage.

• 29:46Chris Benson?

  Would it be fair, as you say that, you know, I know that certainly Daniel and I come from a developer background in addition to a data background. And, you know, something that we've talked about lots over time is kind of the maturity of DevOps relative to the maturity of DataOps. And you're clearly, you know, as you just, as you were just explaining that, you know, you're clearly kind of addressing some of that. Is that maybe the core participants in those two arenas are coming from fundamentally different places, do you think? Or those of us who have been developers prior to doing machine learning, and we're kind of carrying some of that in and we try to put that in place. I mean, is this just different constituents that this arises from? And maybe a data scientist wasn't worried about DataOps because it wasn't a mature enough field prior to this.

• 30:31Chris Benson?

  But now it's, it's important. What's, what's your take on that?

• 30:34Speaker 00

  Yeah, I mean, look, I think like, I'll say a couple of things on that. So first of all, I think data science trains you to be bad at DevOps. Right? It's, it's actually a different thing than engineering. And most of the code that you write as a, as a data scientist or an ML practitioner, most of it's throwaway. And I could really relate to this, right? Because I came from this world and I was kind of surprised when I got, I remember I got to Yahoo and I checked in some code that actually crashed the search engine. And they were like, man, you are a moron.

• 31:03Daniel Whitenack?

  It's a claim to fame, though.

• 31:05Speaker 00

  That is, that's a good one.

• 31:06Daniel Whitenack?

  You can work that into conversation now.

• 31:08Speaker 00

  Yeah, I mean, they should have put some more controls around me, honestly. But, but I just really like didn't, you don't get trained in like testing. And not only that, but what you do get trained is that like, most of your stuff is throwaway. So you have this very, very 80/20 rule of I just want to write a lot of fast code, kind of figure out what's going on and then like toss it over the fence to someone to kind of harden it, right? I think that that Breaks now that AI is getting used in so many like mission critical applications, even like applications where you know, someone could get seriously hurt or like really bad things could happen. And so I think the other thing that happens is that DevOps doesn't really understand machine learning and data science that well. And they're not really prepared for the statistical nature of it. Right? So for example, I mean, just, you know, like continuous

• 31:53Speaker 00

  integration tools, they crash if any test fails, right? And that totally makes sense, you know, from like a normal engineering perspective.

• 32:00Speaker 04

  Yeah.

• 32:00Speaker 00

  But it's just not realistic to have a suite of continuous integration, like real world continuous integration test for a machine learning thing, right? The machine learning stuff that we put into production, we know that it has like a failure rate that's non-zero. And so instead, we have to like, deal with that versus trying to drive the failure rate down to exactly zero. And so I think there's a real culture gap and there's very, very few people trained in both DevOps and machine learning. And those people make tons of money as I've seen. So it's a, it's cool when someone has both skills, but it's natural that people don't have both skills. And I think the skills in a way teach you very different styles of development.

• 32:38Chris Benson?

  As a two second follow up to that, as we get to where we're deploying out there and you're addressing it with your, with the workflow that you guys have been creating, how do you address the use cases, the variability in the use cases in terms of if it is something that is more just kind of typical consumer than totally see your point, but there are use cases where you have these models that are going into, that are starting to go into life and death, you know, issues, you know, where people's safety is becoming dependent upon it and they are trained 99% is not going to be acceptable. Is there a point there where you evaluate or cross over or acknowledge that in some way? Or how, whether now or in the future, how would you address that kind of variability in you're starting to get models in so many different real life

• 33:23Chris Benson?

  situations that you have to account for that issue?

• 33:26Speaker 00

  Well, I mean, look, weighted biases mission is to support the ML practitioners to see what's going on. Right. So, you know, we're not trying to tell you if 95% accuracy is good enough. What we're trying to be sure is that you know what you have and you know if it's 95% or 99%. And then you can also dig in and say, okay, this is 99% on roads, but only 8% on pedestrians. And that's, you know, Got it. I should flag it. I also think like, I think it would be nice. It's hard to acknowledge, but I think the industry does have to acknowledge that 100% accuracy in most applications is not attainable, even in life or death ones. Right. So I think it's really hard to come out and say like, look, like, you know, this self-driving car is going to get it wrong, you know, 0.001% of the time or whatever.

• 34:10Speaker 00

  But, you know, people talk about miles between intervention and it's not infinity miles between intervention. And humans also, by the way, are not 100% accurate. So Good point. You know, that's a little bit outside of the scope of what we try to do. But I think fundamentally, as we do more and more complicated things, it's better to acknowledge that errors are inevitable and have plans to deal with them than try to claim that there are no errors. Gotcha.

• 34:34Daniel Whitenack?

  I'm curious about a sort of different side of variability, not so much on the performance of the actual model side, but as a package that someone imports into all sorts of different workflows. There's such variety now in terms of like the tooling that people are using for their ML workflows. And I see like on your website, you're talking about, you know, kind of integrating weights and biases into all sorts of workflows, TensorFlow, PyTorch, Keras, scikit-learn, you know, hugging face, transformers, all sorts of things, you know, could you speak a little bit to that challenge in terms of, you know, I'm sure as a company and as a, you know, sort of product roadmap, it is extremely difficult to sort of understand and navigate like.

• 35:18Daniel Whitenack?

  How tightly you're integrated with various things versus other things and how to reasonably support the variety of tooling that's out there in AI right now? How do you, in general, do you think about that and approach that?

• 35:32Speaker 00

  Well, it's a conversation with our customers. So, you know, when you sign up for Wits and Biases, we really only ask you two or three questions, right? And, you know, one question is, how often do you train models? And if you answer that, that you train models frequently, then we take very seriously answer the next question, which is what frameworks and other tools do you use? And so, you know, when we see something like JAX being used by lots of people that train their models frequently, we say, okay, we should prioritize a good JAX integration. And you're right, it's tricky. I mean, I think it's nice that you appreciate the work that we do. I mean, a lot of people look at what we do and it's just like, we generate these graphs really quickly. And making these graphs, I hope that it's kind of a simple experience for the user, but it's a tough challenge for engineers to make sure that those all kind of reliably

• 36:16Speaker 00

  generate in a way that's useful to our users, regardless of what framework. I mean, forget what framework, like Python version, also like PyTorch and TensorFlow, both incredibly cavalier about making breaking changes to their libraries. So it's definitely a challenge.

• 36:31Daniel Whitenack?

  Yeah.

• 37:40Daniel Whitenack?

  So Lucas, I would be curious to hear a little bit. So we talked a lot about experiment tracking, which I know is like you say, where you kind of started with this journey and maybe what weights and biases is most known for. But as I'm exploring the website and the feature set that you're supporting, there are some other things. In particular, I see like sweeps, which I think is related to hyperparameter tuning, and then artifacts, which is related to data set versioning. I'm really curious about the latter one. Actually, I'm really passionate about data versioning, actually. And so I was curious, you know, when this came about, it seems to me like maybe a couple of years ago, if I was talking about like data versioning, like people just like had no idea what I

• 38:25Daniel Whitenack?

  was talking about. And now there is this sort of acknowledgement that there's something here. And maybe that's, you know, fed into the support of that and weights and biases. So yeah, could you tell us a little bit about your support for that and also your thought process around data versioning and how you think about it?

• 38:43Speaker 00

  Sure. I mean, I would say all the things that we add are 100% customer driven. So, you know, if you're a practitioner and you want us to make tools for you, come into our Slack channel and talk about what you're doing and tell us what you need. And we would love to make things to support that. So that's how all this stuff came about. I would say, you know, hyperparameter, we call it sweeps, hyperparameter search or hyperparameter sweeps, we built because everyone kept telling us, yeah, we know this is best practice. We know there's a lot of tools available. But when you ask them, okay, which of these tools are you using? How are you doing it? They'd always be like, I'm kind of doing a random search with my own homegrown stuff. And when I hear that as an entrepreneur making tools, it's like, okay, these tools aren't working for you. Like, why aren't they working for you? And I think it's because the reality of hyperparameter search is

• 39:28Speaker 00

  messy than the theory of it, right? Like, the math behind doing Bayesian optimization or things like that. It's not so complicated. What is complicated is the real world where you start to do it in a distributed way and six of your runs break and then you decide to change the code midway through your search, but you don't want to throw away your old data. And so we've really tried to focus on that and kind of making it simple to do and simple to see what's going on. And in fact, you can use other optimization libraries that would even tell you maybe better than ours, right? So you can use Ray Tune, which I think is a fantastic library to do advanced stuff that you can't do with us. So that's how we think about that. And I would actually say data versioning is almost the same thought process where people know that they should be versioning their data, right?

• 40:11Speaker 00

  They know that it's dangerous to not version their data and you know that you can't have reproducibility unless you version your data. But I think that all the existing solutions are kind of causing problems. So I think, you know, Git has like the large file store system, which seems like it should be made for this kind of thing. But when we talked to practitioners, most of them weren't actually using that. And so we kind of dug into why and it's like, well, a lot of data sets these days are in object stores. So supporting an object store is incredibly important for versioning data, I think. And then, you know, another detail that shouldn't be overlooked is most ML practitioners are not super fluent with Git, right? They find Git to be kind of a scary interface. DevOps people find it trivial.

• 40:54Speaker 00

  In fact, my co-founder Sean comes from, you know, less of a ML background and it's a more of a kind of DevOps background. And he just constantly makes fun of me for how bad I am at Git and how, in his view, I just kind of type in random commands into Git until I like break everything and then like ask him to fix the stuff. But I think that was a little bit eye opening for him when he was designing the versioning because he's like, I have to really dumb this down to make something that like Lucas can understand and use. Yeah, I think that's a real thing, you know, I mean, and so.

• 41:24Chris Benson?

  You're representing your customers in that way too, though. I mean, there's something to be said for that.

• 41:28Daniel Whitenack?

  Yeah, no, I mean, I remember this a couple of years ago, our conference in New York, which they're going to be having another one here soon, which Chris and I are going to lead a panel or moderate a panel on. I remember in that conference, they had like a Git workshop sort of deal. And it was like one of the like most popular, like, like people were just like so into it because there is this sort of like, yeah, I can create a repo and like maybe make an initial commit, but then I like all this stuff happens and I have no idea and get scared.

• 41:59Chris Benson?

  It gets scary.

• 42:00Speaker 00

  When stuff gets weird, it's just like, ugh.

• 42:03Chris Benson?

  No, really. I mean, if you haven't been doing professional software development for a while, Git is scary. It really is. I mean, I've seen a lot of people getting into the field and they struggle with it and stuff. So I totally, I'm very empathetic to people who are coming from a data science background and they're trying to figure out how to do this, either Git or something similar to that. It's a new concept. So there's a cultural barrier that has to be surmounted there.

• 42:26Speaker 00

  Right, right.

• 42:27Daniel Whitenack?

  Yeah, definitely. So when you are kind of seeing, and I'm sure I was looking through your website, there's lots of good stories and that sort of thing on there. But in terms of like, success stories with weights and biases and like the workflow that people used, you know, could you give us like, tease us with sort of example? I know I'm putting you on the spot, but in terms of, you know, maybe a team that was doing something, you know, rolling, trying to roll their own before, and then was able to switch over to some tooling, maybe a combination of what you offer and other things to accomplish something. Do you have one of those stories you could share?

• 43:08Speaker 00

  Yeah, totally. I'll tell you my, one of my favorite ones, because it's so practical and just so great. Although it's been told, the application has been told a couple of times. You may have heard it before, but I feel really proud of it. So this is a team at John Deere called Blue River. And they basically make a system that looks down at lettuce and tractors. And if it sees weeds, it sprays the weeds. I believe it's with fertilizer, so it burns the weeds and then it becomes fertilizer for the lettuce, right? So instead of blanketing the fields with pesticides, they aim a sprayer at the weeds.

• 43:44Daniel Whitenack?

  Yeah, I've seen those videos. It's amazing. Yeah.

• 43:46Speaker 00

  And they did a whole case study with this. So I know I can go into detail and you could maybe post the case study where they actually show their weights and biases runs and how they think about it. But you know, when we went in there, They were, you know, having the problem of they were training tons and tons of models and trying to keep track of which models they were making and what they were learning and then also kind of what was getting put into production, right? Because it's kind of high, I mean, it's kind of, it's very high stakes. If you're a farmer, you, you, you can't have your crops get killed by your AI

• 44:13Daniel Whitenack?

  Yeah.

• 44:13Speaker 00

  eating machine, right?

• 44:14Daniel Whitenack?

  And a single, single percentage, uh, over, you know, hundreds of thousands of acres is not trivial.

• 44:20Speaker 00

  Yeah, yeah. So, I mean, they're a super smart team, super technical. Actually, I remember when we went in there, we had this funny feature where we would try to pull up a web browser to do authentication. And they actually had links installed on their system. Do you remember this? It's like a Unix terminal-based like web browser that it then like popped up and I thought, oh, man, there's like a bug. Yeah. Anyway, so it was kind of a rough, um, initial meeting with them, but they, uh, you know, they got it working. And so then the whole team can, they can look at what they're doing. And I should say, maybe the most important thing for them is that like each unit of work, kind of each thing that they try, they put it into a report, which they attach to a pull request.

• 45:01Speaker 00

  And so they can go and they can look at kind of, and it's not like necessarily one run, right? It might be like, okay, you know, I tried this new thing and I tried like 10 different sets of hyperparameters. Like I tried a new data set. Like maybe there's a new camera that picked up like UV light. And then I tried like these 10 hyperparameters with it. And here's what I concluded, like the camera was good, we should explore it further. And so they'll actually kind of write that all into a report and then attach it. And they actually put an example report that they gave to us use for a case study. So it's really cool to look at. You can see they're trying different types of gradient descent on their data set. So, you know, that was like the initial case was them using it um, to track their experiments like that. But then they started to use it to look at their data sets too, because they have so many different data sets and it's constantly being collected in these fields. So they use their, say, our versioning system.

• 45:45Speaker 00

  And they attach it to our experiment tracking system to really know kind of end to end reproducibility of the stuff that they're deploying into tractors.

• 45:53Chris Benson?

  Pretty cool. I guess as we start winding up, if you could wax poetic a little bit for us for a moment about kind of what you are aspiring to in terms of future of tooling, what are some of the things in your head that you have this vision for, you would like to move into, you're not there yet, but you realize what a big impact it would make? I'd love to see kind of where the future of tooling is going because I know that will directly affect both Daniel and my lives. And so, any insight there?

• 46:21Speaker 00

  Totally. I mean, what I'd love to do is make simple tools that really help the day-to-day lives of people trying to do practical AI. So I would say the thing that we haven't done that I'm kind of dying to do, but it's a huge undertaking, is stuff around production monitoring. You know, we do have some stuff today, where we have a lot of customers use us implicitly for continuous integration, but we don't support that case explicitly yet. And I think that that's a big deal to, because you know, AI safety is so important. And I really believe that all the ML practitioners are good people that care more than society gives them credit for AI safety. I actually just think it's really hard. And so I think, I think these ethical issues are real, but they're, they're hard. You know, and I think actually tooling can help here by showing people what's going

• 47:06Speaker 00

  on. And, and you know, it's like developers, when you, when you deploy a bad, a bad piece of code, right? It's like, you know, you should improve the systems. I think a lot of the tooling is missing for ML practitioners. And so it's dangerous, right? So continuous integration is a big one. You know, production modeling, detecting things like model drift or data drift and things like that, you know, I think makes me nervous, should make anyone nervous. I think, you know, Andre Karpathy had this really evocative talk he gave at one of Figure It's conferences around sort of like a new IDE for ML that I keep kind of kicking around in my head where it's like, what would that really mean, right? To really... Like be able to tweak all the things that go into the model, like not just the hyperparameters in the code, but maybe even like the data that flows into your model.

• 47:50Speaker 00

  So those are kind of ideas I have, but you know, these ideas always take forever to actually get into the world. So I don't know. I don't know what we'll get to.

• 47:57Chris Benson?

  That's what we're looking for. Not today, but you know, down the road, you know, what, what are people aspiring for today that they might work on in the future, you know, without commitment there? It's always interesting to hear that.

• 48:08Daniel Whitenack?

  Yeah, we'll definitely link to the website of Weights and Biases and some articles and other things. And of course, your awesome podcast, which, which people can find and, and listen to. I definitely encourage people to do that. It sounds like Weights and Biases has a, has a Slack community that we can link to where you can connect with them. If you're having trouble finding out, you can always reach out to us as well on our Slack community and find that at changelog.com/community or on LinkedIn or Twitter or anywhere. But really appreciate the insight today, Lucas. It's been a real joy to talk to you and looking forward to all the great things that are, will be coming from Weights and Biases. So thank you so much.

• 48:51Speaker 00

  Thanks so much. That was fun.

• 48:56Speaker 05

  If you enjoy Practical AI, we would enjoy a five-star review on Apple Podcasts, a blog post in response to something said on the show, and/or a recommendation to a friend or colleague. Those word-of-mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whiteknecht. It is produced by Jared Santo with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Shout out to Fastly, Linode, and Realbar. That's all we have for you today. We'll talk to you again next week.

• 49:57Speaker 00

  Sorry guys, I think you're frozen. Can you, can you hear me still?

• 50:00Chris Benson?

  Yeah, we can hear you fine.

• 50:01Speaker 00

  You're just totally still. That's amazing.

• 50:03Daniel Whitenack?

  That's the thing about these Zoom meetings.

• 50:05Chris Benson?

  I'm like that, I'm just listening.

• 50:08Daniel Whitenack?

  We're just on the edge of our seats, you know?

• 50:10Speaker 00

  Oh, good, okay. You had me in a r- I continue my riveting story, I'll try to wrap it up.

• 50:16Chris Benson?

  I'm gonna like TikTok a little bit, just to make sure that you see, I was like really into your story, so I was sitting still and listening. The audience can't, yeah, we're watching each other on Zoom, we can see each other's facial expressions. So it's not just audio for us as we're having it, and yeah, I think with Daniel and I, we're really, really still.

• 50:33Speaker 00

  Nice.