Eradicating Machine Learning Pain Points with Weights & Biases CEO Lukas Biewald

How are ML developer tools helping to advance our capabilities? Lukas Biewald, CEO of Weights & Biases, joins Sarah Guo and Elad Gil this week on No Priors. Lukas explores the impact of ML in various industries like gaming, AgTech, and fintech through his insightful perspective. He discusses the impact of LLMs, puts them in context of the evolution of ML engineering over the past decade and a half, and tells the backstory of Weights & Biases' success. He gives advice for aspiring AI company founders, placing emphasis on customer feedback and using insecurity as a vehicle for better customer discovery. Prior to founding Weights & Biases, Lukas attacked the problem of data collection for model training as the Founder of Figure Eight, which he sold in 2019. He holds an MS in Computer Science and a BS in Mathematics from Stanford University. Show Links: Lukas Biewald - CEO & Co-founder - Weights & Biases | LinkedIn Weights & Biases Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @l2k Show Notes: [0:00:00] - Lucas Wald's Journey in AI [0:08:16] - Startup Evolution and Machine Learning [0:18:54] - Open Source Models Implications and Adoption [0:29:54] - ML Impact in Various Industries [0:40:27] - Advice for AI Company Founders

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  We've talked to many practitioners who are pushing the state of the art. This week on the podcast, we're exploring the dominant ML developer tool, Weights and Biases. Alad and I are sitting down with CEO and co-founder Lucas Biewald. He has a knack for creating companies that support pain points in ML development. His first company, Figure8, addressed the problem of data collection for model training. And his second company, Weights and Biases, has created an experimentation platform that supports AI practitioners at companies including Nvidia, OpenAI, Microsoft, and many more. Lucas, thanks for doing this. Welcome to No Priors.

• 0:36Speaker 01

  Thank you. Great to be here.

• 0:38Sarah Guo?

  Lucas, you studied at Stanford, where I assume you discovered your interest in machine learning, and under one of our previous No Priors guests, Daphne Kohler. Can you talk about when you started working in AI and learning from Daphne?

• 0:51Speaker 01

  Yeah, totally. As a kid, I was obsessed with playing games and I got really into Go. And I was super into the idea of, or thinking about how would computers win at these games? And so I actually sent Daphne an email, maybe as a freshman, being like, hey, can I, can I work with you? Like, I'm really interested in games. I want to learn how to like beat Go. And Daphne wrote me actually a pretty polite email being like, that's not what I do. Go away. A few years later, I took her course and I was actually, I studied math at Stanford. And I have to say, Daphne cared about a thousand times more about teaching than even the best professor in the math department. And so it was really just eye opening. Like, I just loved how much she actually cared about teaching. And it got me really excited about the AI that was working there.

• 1:35Speaker 01

  And I went on to be a research assistant for her. And the funny thing at that time was like, nothing really worked. Like, it was just before kind of, you know, Google was thought to be really like PageRank at the time was the thing that was making them work. And I think later, you know, it became clear that machine learning was a big, a big part of that. But really, when I was doing ML, it was like searching for applications that were working. And Daphne was actually really obsessed at the time with a thing called Biases' nets, which you don't hear about too much anymore because I don't think they ever really, you know, worked for many applications. I hope I'm not offending anyone, but that's why my understanding. I actually think, you know, the thing that I really took away from Daphne that that really lasted with me was, um, I mean, she is one of the smartest people I've ever encountered.

• 2:19Speaker 01

  And she had this incredible clarity of thought and an intolerance for sloppy thinking that that's just like really served me well. I think that's sort of separate from machine learning. You'd see like other professors would come and give like guest talks and, you know, they would say something's kind of lazy. And like, we'd all just be sitting there just like waiting for Daphne to like eviscerate them. And I think her personality has mellowed a little bit over time, but I kind of miss, I just miss that sort of like aggressive, clear thinking. And I really admire it.

• 2:51Sarah Guo?

  I don't think we got a taste of that, but we did talk about whether or not probabilistic graphs are are coming back a little bit. How did you, how'd you go from, you know, Stanford to founding Figure8?

• 3:03Speaker 01

  Yeah, you know, it's funny, I actually really struggled doing research with with Daphne. Basically, the things that I tried just barely, barely worked. Like, you know, I published a couple of papers that I feel kind of ashamed of where it was sort of like, go from like 68% accuracy to 70% accuracy in a task. Nobody cares about by throwing like a thousand X to compute. And by the way, like kind of guessing the most likely answer is probably like 64% accuracy. So, um, you know, it just, it felt honestly kind of pointless and sad. Like, I love the idea of like computers learning to do things, but it's hard to sort of sustain the enthusiasm for that when everything you try just completely, you know, doesn't work.

• 3:45Speaker 01

  And even the things that do work, you kind of wonder if you're like p-value hacking, like, okay, I tried a thousand things, you know? So I guess something's going to be like a little bit more accurate than than a baseline.

• 3:54Sarah Guo?

  What tasks were you working on? Did you, did you end up working on Go or games or anything?

• 3:59Speaker 01

  No, Daphne was, Daphne is not interested in games, let me tell you. And it's actually another, I kind of admire that that perspective too, as much as I love games.

• 4:06Sarah Guo?

  I'm a Go nerd, so I'm curious.

• 4:08Speaker 01

  Oh, you are? Oh, me too. Yeah, I love Go. Yeah, Daphne was very not interested. She really was practical. And so I worked on a task that you really don't do now called word sense disambiguation, where you're trying to find out like, okay, I have the word plant. Actually, if you look in most corpuses, because they're government generated often at the time, plant typically will mean like the power plant sense of plant. Or cabinet often means the sort of president's cabinet sense of cabinet. And so you're kind of trying to figure out like, what is the meaning here of these words? And then applied it to translation. It's a cool task. I mean, and actually, it turns out, I think that these, again, nobody kill me, but my general sense is that these sort of like linguistic oriented

• 4:52Speaker 01

  strategies really don't work that well. It's kind of like by feeding more data in and sort of like working on outcomes, you can figure these things out much better. So a little bit of a dead end. And actually, you know, I was so frustrated by that that I just really wanted to work on something that people cared about. I actually turned down an offer from Google because they didn't tell me what I would be working on to go to Yahoo, because they they were like, okay, you can work on, you know, search ranking in different languages. And but that actually turned out to be incredibly fun, right? Because it was super applied. It's actually a task that works really well. And Yahoo is kind of in the infancy of switching from hand tuned weights to machine learned weights. So they really had no one, not many people actually like working on deploying this stuff.

• 5:35Speaker 01

  So I was like writing code to translate machine learning algorithms into C code. And then check it like we would check it into our little code base and run this kind of like semi hand generated C code in production. So that was that was super fun. But you know, the thing I learned there, actually, which I think I'm not the only one that learned this, but I just felt it. I would go from like country to country, trying to switch from hand tuned weights to an ML model. And like, I was sort of the messenger here. So like, sometimes it would work and sometimes it wouldn't. And so like, people would either really happy with me when it did work, or they'd be really pissed at me when it didn't work. But I kind of realized, actually, the model that I'm building is like the same for each country. It's the, the training data though is different. So some countries would take the training data collection process really seriously and they'd get a great model.

• 6:20Speaker 01

  And some would just like really half ass it or like, you know, have these crazy like issues in the data collection. And then the model wouldn't work. And so I just really kind of viscerally felt how much the, the training data process mattered. And I kind of felt like, you know, why don't they let me get involved in the training data process? Like, that would be a better use of my time than building these models. And so I wanted to make a company where the people doing the ML could actually have control over the training data collection process and really get like visibility into it. Because, you know, at the time, I think the thinking was like, oh, this is sort of like a manual task. That's like more of like an operations team should deal with this.

• 7:02Speaker 01

  And, and they would like, they would do this thing where you'd like, make this giant requirements document. It was so like waterfall, like it would be like,

• 7:09Sarah Guo?

  Yeah, it wasn't iterative.

• 7:10Speaker 01

  Oh, it wasn't iterative. And it'd be like, you'd make like a 50 page document. And like, you know that the people doing the labeling are not like reading that document. But you kind of need that to like cover your ass if they did like labeled something, you know, not the way you want. And it would have been so much better to be like, look, we're trying to write search results. Like, put yourself in the mindset of like someone, you know, who's like looking at this, like, is it good or bad? Versus trying to lay out in like excruciating detail what makes something relevant or, or not relevant.

• 7:36Speaker 02

  I think also at this time, like when, when you first started, I think originally it was called Dolores Labs and then Crowdflower and then eventually Figure8. Like, I think I met you in your Dolores Labs days or something.

• 7:45Speaker 01

  I know, I remember, yeah.

• 7:46Speaker 02

  Yeah. And at the time, there weren't really, um, solutions for data labeling externally, right? Some people are using mechanical Turk from Amazon to sort of run jobs on untrained workers. There wasn't like scale, there wasn't, you know, there was none of these services.

• 7:59Speaker 01

  Yeah.

• 7:59Speaker 02

  And so you got really early to this idea of starting like a data labeling company and that that was actually very useful for machine learning. And so it'd be great to hear like, you know, what were the early days of, of that like? And what was the industry like? And how did you get all that running?

• 8:12Speaker 01

  Yeah, I mean, it's funny, right? Because back then I was coached actually quite a lot by Travis Kalanick, who's famous now for doing Uber and other things. But he was like, don't tell anyone that it's like AI, like VCs like don't want to hear AI, which actually good advice at the time. And it's good advice in the early days of the company.

• 8:30Speaker 02

  And Sorry to interrupt. I think one interesting side note on that, just from a Silicon Valley history perspective is Travis used to have these effectively like hackathons or meetups at his house called the HackPad. And, you know, I think he used to go to those, you know, a bunch of friends of mine used to. And so a lot of startups actually had some impact or influence from Travis in those days, like due to his fact of like, you know, being another founder in the scene and kind of getting everybody together. And so it's kind of an interesting moment in time or in history. And to your point back then, like AI wasn't really as popular as it became later. So it's kind of an interesting like side note.

• 9:04Speaker 01

  Well, I mean, not only was AI not popular, but like startups weren't popular, right? Like my family didn't, you know, understand about startups. And I had graduated Stanford, you'd think I'd have all these great like connections, but it didn't feel like that. Like I had no one who knew how to like raise money from VCs. I didn't know any, you know, VCs or I didn't really know any like entrepreneurs, honestly. And we had this website for Dolores Labs in the early days, just trying to get customers. And it put my, my personal phone number. Actually, I remember I was like the first user of Twilio because I needed to make a phone tree. And so I used Twilio software and then like all three of the founders came to my house to like, help me like make that phone tree like work better. It's kind of amazing. It was like, you know, like, you know, one of those like, you know, 20 something like, you know, grungy apartments in the mission.

• 9:49Speaker 01

  And then, and then Travis called in, but you know, it's funny because the phone tree, we were just trying to pretend like we were a big company. And Travis called in because of the phone numbers on the website, not because he wanted to buy anything, but he just like, thought it was like awesome. And so I'm just like, you know, I pick up my phone and then there's just like this guy in the other and just be like, oh, man, like this is so cool. You know, I'm like, okay, like who are you? You know, it's like, do you want to like get coffee? And, and that actually turned out to be incredibly like helpful. But then I think like, the thing that was so different back then is that the people doing ML, there just weren't that many. Like, so there were people like heavily investing in ML, but there, but it wasn't that many. And so what happened was, you know, we got like eBay as a customer, which has really mattered at the time.

• 10:34Speaker 01

  And we got like, you know, Google as a customer and Bloomberg. And then there just like wasn't anywhere else to go. So like, you know, my board was always like recommending like re-crossing the chasm. And, and we tried like a million different ways to like, you know, grow the company. And, you know, I don't know, I hope this doesn't sound defensive. I mean, maybe I was just a bad CEO, but we had like years of like struggle. Because there was no chasm to cross, right? There was like nowhere else to go. So we tried all these different things to like, you know, build more complete solutions for our customers. And it just didn't work. And then kind of all of a sudden, um, you know, autonomous vehicles got popular. And that really actually suddenly caused our revenue to, um, you know, start to, to grow really fast again.

• 11:17Speaker 01

  But it was like an eight-year lull of like, you know, really no growth, right? So it's hard because we started off fast, got everyone really excited, you know, kind of got like womped for just like years and years and years. Actually, we had all these competitors, they all went away. So at some point, we had like no competitors left, right? Because like everyone had, uh, had gone out of business. And then it was a funny experience because like scale came along and totally ate our lunch on, in the self-driving market, which is a market like I knew and loved. And so, you know, I, I was so excited to sell the company after, you know, so many years of struggle. You know, but then like right after that, we see like scale, just like skyrocketing and revenues like, oh man, like I wish we had just like, you know, maybe held on a little bit longer, but then, you know, it gave me the, the space to start weights and biases. So, you know, who knows?

• 11:58Speaker 01

  I, I want to be like Daphne Koller and evaluate my decisions like accurately and, and critically. But it also does seem like, you know, I've had some good luck along the way.

• 12:08Speaker 02

  Yeah, no, the market shifted so dramatically. And I think to your point, self-driving was the first time that you suddenly had a bunch of systems at scale that people needed data labeling for. And then of course, now we have this LLM wave, but it's all very, very recent. And I think a lot of people basically view ML as this sort of continuity and everything has always been kind of rising in a sort of almost linear way. And in reality, it's this very bumpy set of discontinuities in terms of the set of technologies and markets that people are adopting it in. And so it's not continuous, it's a discontinuous thing and nobody thinks about it that way. When you started weights and biases, you said something along the lines of you can't paint well with a crappy paintbrush. You can't write code well in a crappy IDE. And you can't build and deploy great learning models with the tools we have now. I can't think of a more, any important, more important goal than changing that.

• 12:53Speaker 02

  And that's, I think, like when you announced that you were starting Wastin' Biases. And so I was just curious, like, what lapses in capability really got you going on one B? And can you also just, you know, many of our listeners know what it does, but for those who don't, could you explain what the product does and how it works?

• 13:08Speaker 01

  Sure. Yeah, so it's kind of constantly evolving, right? Because we're saying it's like a set of tools for for people doing machine learning. We're best known for our first thing that does experiment tracking, which keeps track of like how your models like perform over time as they learn and train. But we also have a lot of stuff around like kind of data versioning, data lineage, you know, production monitoring, model registry, kind of the sort of end to end stuff that you'd need to do machine learning reliably. And I think the thing that happened to me was I had been running CrowdFlower for years, and I always loved machine learning. But I was like really starting to get out of date, like deep learning came along. And at first I was kind of skeptical of it because people are always saying, oh, I have a better model that's like magically better. And they're like, wrong, wrong, wrong, wrong, wrong. It's just like really like data. And then, but then

• 13:53Speaker 01

  they were right, right? So there actually was a sort of a better modeling approach that worked. And I kind of realized, you know, when I was in my early 20s, I was really judgmental of, you know, the people in their late 30s that hadn't like adapted to machine learning at the time. Because like rule based systems were kind of all the rage when a different generation was was growing up. And I was like, wow, you know, I am actually getting out of date myself. Like, I'm saying these kind of wrong things that were true 10 years ago and are not true now. And I honestly felt like really bad about myself. And so I did a couple of projects to try to, you know, get up to speed. I started teaching free machine learning classes and deep learning classes to kind of force myself to to learn the material. And I actually like interned briefly at OpenAI,

• 14:35Speaker 01

  where I was just like, look, I will just do whatever, you know, work you want. Just I want to be like, I need, I know that I need like an accountability partner essentially to force me to learn stuff, even though I love to learn stuff. It's like my favorite thing, but I always need accountability partners for anything I do. So I sort of used the students as an accountability partner and OpenAI. And then what was happening was I was showing my old co-founder, Chris, like all the the cool stuff. And he's like a really good engineer. And I'm like, actually, really like bad engineer, like I'm like, really lazy and like, trying to write the like, you know, I'm just like, like people, my co-founders make fun of me all the time for like, you don't really know how Git works. And I just openly, I have no idea how Git works. I just sort of mash the Git keyboard until like, I kind of like, you know, get in a bad state. And then I like call Chris and beg him to like

• 15:18Sarah Guo?

  The CEO rebased.

• 15:21Speaker 01

  Yeah, I just, I don't know. I don't know. I don't understand it. And it's like, my co-founders just find it baffling that I wouldn't understand it. But I think it's like, for them, you know, it's like, they're like, wow, this guy like needs some basic tools, you know, like, because, you know, they're like, okay, like reproducibility, like, why don't you just use Docker? I think that's sort of the ops mindset. But I'm like, man, I don't understand Docker. Guys feel like I installed on my like laptop and then it's always like taking up memory and stuff. I like, I don't really know what it's doing. And I'm like, kind of scared of it. And like, I don't know. So it's like, I just feel like it's adding weird complexity to understand. And so I think the tools kind of exist in a way, but they just weren't made in a way that like, ML people could really use them. Because like, you know, if you're like me, you kind of come from a mathy

• 16:05Speaker 01

  background, or like, a research background, you kind of didn't really learn to do like, industrial style coding. And so, you know, I think companies have this idea that like, the researchers are just gonna like, throw the thing over the fence, and then it's gonna be in production. But it doesn't really work, actually. Like, I think that's a bad pattern that people sort of like, imagine they're gonna do, and they don't ever really do that. They end up like, with researchers, always the research code bleeds into production in every company. And so I think a better way is to give, you know, researchers and like, ML people, tools to just make their stuff more reliable. And it has to be simpler, maybe, or it's just a slightly different audience. Like, you can't just give someone like Docker, you can't just

• 16:48Speaker 01

  like, you could, I mean, a lot of people are like, hey, why don't you use like, the Git large file system stuff to version your data? And like, there actually are some reasons, like, it doesn't work well with like, object storage. So there's some like, ergonomics reasons. But it's also just like, man, Git is like, complicated. I'm like, willing to use it for code. But if you start making me like, version my data with Git, like, I just want to like, cry. You know what I mean? So, like, give me something like, simple, you know what I mean, where I don't have to like, think about it. Or I'm just gonna start like, renaming my data sets like, latest, latest, really, latest, that's really for sure, June 27. So, I just need my stuff to be simple. That's kind of the mindset, you know, behind the companies, like, let's like, make these, like, kind of simple, clear things that actually help people.

• 17:31Sarah Guo?

  We were talking about how much you wanted to, like you were thinking through how much LLMs were going to change, like experimentation and ML tooling when we last saw each other in person, not at the zoo, but before that.

• 17:44Speaker 01

  Yeah.

• 17:44Sarah Guo?

  And you, you guys launched this prompt suite in April. Like, can you talk us through the sort of, you know, thought process of, hey, like, you know, I really admire this as a leader and as a technical person, you're like trying to stay really plastic about what is actually changing in machine learning. How'd you think through this change?

• 18:01Speaker 01

  Well, it's really hard, right? I mean, so what happened was we have a great business that, you know, makes like an ML, a set of ML tools for training models. And we actually helped most of the LLMs out there were built using weights and biases. And then we started to see, like, wait a second, some of these ML tasks, you could just ask the LLM, right? So instead of doing like a sentiment analysis model, you could just be like, hey, like, is this document positive or negative sentiment? Like for structuring documents, you could just be like, hey, find all the names like in this document and it actually works super well. And a little piece of me is a little bit sad about that because we have this like great, simple, relaxing business that grows revenue every, every month that I always dreamed of, right?

• 18:43Speaker 01

  So, you know, part of me is like, shit, this is actually our kind of first real existential threat, I think, you know? And, and, um, and, you know, I went to my like leadership team and I went to my board and I was like, I think there's like a real existential threat here. And I think they were like, hey, you know, we don't like see it in the data. Like, are you sure? Like, maybe you're being paranoid. And I guess I do feel sure, and I don't want to say I'm like the only one or like paint myself as the hero, like, you know, my co-founder is also seeing this and you have people talking about it. But it's sort of like, you know, this threat is like now, right? And we have to actually like get the whole company to do this thing because it doesn't show up in any of our like metrics yet. But I just really believe that, you know, our customers are rational and they're going to do a thing that like makes sense for them.

• 19:28Speaker 01

  And so I see a lot of my colleagues being like, oh, there's going to be like lots of different models. It's like, nice if it were true, but like what I see everyone doing right now on July 27th is using GPT. Like I can see like 95% of the people out there, you know, using GPT for these ML tasks. And so it's like, look, we gotta support that. And so we really rallied the whole company behind it and we pushed out prompts. We'd also, this is really my, my co-founders, my co-founder Sean had really put a lot of effort into making our stuff really flexible. Because he was like, you know what, Lucas, like there's gonna be like changes, you know, coming. We don't know exactly what they are, but like, you know, kind of from the beginning, we really tried to build very flexible infrastructure.

• 20:13Speaker 01

  So this was kind of a moment where we could really sort of like flex that and get out a, um, you know, a product for, for monitoring and stuff. And, you know, now it's like, you know, kind of, it's our number one priority is getting out more tools for this new, this new workflow.

• 20:26Speaker 02

  Out of curiosity, because you know, there's a lot of debate right now in terms of proprietary models versus open source models. And I think there's a really great quote, I think it's from Harrison from Langchain, which is, you know, no GPU until product market fit. Right? You should first like figure out if the thing works at all, or if there's a customer need, and that means using GPT. And then once you prove it out, you know, you may use GPT four or something for very advanced use cases. And then you kind of fall back to 3.5, or you start training your own model for things where you just want cheap, sort of high throughput things happening. And it increasingly feels to me like people, the most sophisticated people who are the farthest sort of cutting edge on this stuff are kind of doing both. Right? They, they use GPT to prototype and then in some cases they're, they're training their own instance of Llama 2 or whatever they're using.

• 21:12Speaker 02

  Do you think that's where the world is heading? Or do you really think things kind of collapse onto some of these proprietary models like over time? Like it's six months from now, it's a year from now, it's two years from now. I'm just sort of curious about how you think about adoption of open source.

• 21:23Speaker 01

  You know, it's funny, I feel like lately, what I've been telling people is like, I'm just trying to see the world clearly as it is today. I can't predict the future and I can barely keep track of, you know, what people are doing today when I consider it like my, my full-time job. So I'm like scared to prognosticate like what, you know, might be coming, but I think you're right. That that's what's happening now. I think like there are like a bunch of things that could change, right? Like, I think like, you know, GPT is way far out ahead and it's hard to fine tune it, not even possible with with GPT four. And I think that that is like a little, that's not like a technical limitation, I guess, sort of like a business model.

• 22:02Speaker 01

  You know, limitation. So that might change. I think that there's a lot of hidden costs to running your own model. I think people are really enamored with the idea of running their own model. And I've kind of seen this before where I think at the end, people do rational things, but it kind of takes them a while. So I'd rather sort of support what looks like the rational workflow. I mean, I think the insane thing must be crazier to be an investor in this world is like very, very few people have L L M's in production. Like, there's probably more companies that have raised money as like L L M tools than companies that have L L M's in production, which is like insane.

• 22:42Speaker 01

  It's just like an insanely saturated tools market with very few people getting things out. But it's because.

• 22:50Sarah Guo?

  When you, Lucas, when you say, when you say L L M's in production, you mean my own that I have fine tuned, that I serve myself?

• 22:59Speaker 01

  No, sorry. I mean, like, like GPT, like using GPT in production.

• 23:02Sarah Guo?

  Oh, really? Okay.

• 23:04Speaker 01

  Look, I mean, you, you may be like closer to this to me, but I. It's a small handful. Yeah. I'm like desperately trying to find them because like these are our customers, like we, you know, our stuff is just like, our ethos is like, we want to help people do things in production. So it's like, if you're not in production, we're not relevant to you. So I like, I mean, back in January, February this year, we were looking for design partners that had stuff in production and boy, was it hard to find, right? Like, you know, now there are more, but even when you, you know, you find people that are sort of like claiming to have these things in production. It's sort of like, well, it's like, you know, it's coming, like, you know, we have like all these like sort of like prototypes, you know, running.

• 23:40Speaker 01

  And so I think it'll change. I think it's changing quickly, but I think it's a, it's a funny moment where, I mean, I think if you actually looked at the TAM today of like tooling for like L L M's, like, I don't know. I bet you it's, um, it's small. And I think also, I think VCs maybe sometimes have this, this funny window where you see like all the companies that are using L L M's. But the enterprise adoption has been slower. I mean, despite the fact they talk about it like constantly, like constantly, like everyone's talking about it. But in enterprises like. Boy, I don't know if I've like used a product of like any enterprise that actually like was backed by a LLM. And there's a bunch of things that make it hard. It's like, you know, it's kind of unfair because this stuff has only been out for like six months or so.

• 24:24Speaker 01

  But it is like, I think the adoption maybe maybe take a little longer in the short term than people think.

• 24:29Speaker 02

  I think that's a really key point because ultimately, you know, ChatGPT came out eight months ago. And that was kind of the starting gun for all this stuff, in my opinion. And then GPT-4 came out in March or something, right? Which is three, four months ago. And if you look at enterprise planning cycles for large enterprises, it takes them six months to plan something. Right? And so people often ping me and ask about adoption of these sorts of things. And it's like, well, Notion is seeing, you know, has adopted it in interesting ways already. Zapier has adopted it in interesting ways. But it's basically these technical founder-led companies that jumped on it really early, relative to everybody else. And the big enterprises are going to take another year or two because it's they're just in their planning cycle still around this stuff. They just started really thinking about it and how to incorporate it and what to use it for. And then they're going to have to prototype and experiment for a while and then they'll push it into production.

• 25:14Speaker 02

  And so that's why I was kind of asking a little bit about the future. I just feel like it's so early.

• 25:18Speaker 01

  Yeah.

• 25:19Speaker 02

  And we all talk about it again as if it's this continuous industry cycle, but it's really not. It's a disruptive new technology. And so, you know, I think a lot of it's still to come in really interesting ways.

• 25:28Speaker 01

  Oh, totally. And there's tons of product issues too, right? Like, you know, like Notion and Zapier both have these really compelling demos and they're both products that I use. But then I actually don't use the LLM like piece of them myself. And I wonder, I have no insider knowledge of the level of adoption, but I think they're, I think they haven't gotten it like perfectly right yet, despite like a lot of thinking and really smart people working on it.

• 25:51Speaker 02

  Sure. For the core one B product, you know, you folks are being used for a wide variety of areas around autonomous vehicles, financial services, scientific research, media and entertainment. Is there any industry in particular that you think you're either surprised by adoption of the product? Or you're really excited to see sort of how people are using it?

• 26:08Speaker 01

  Yeah, I mean, the one that stands out for me, because this is the one that's really different than, you know, my figure eight days is pharma. So I actually think this is kind of flying under the radar a little bit, but every pharma company is making major investments in in ML and not just on the sort of like, I mean, they do have these operations to sort of like sell more, you know, drugs to doctors that uses sort of like light ML. But I think the thing that's really exciting is like the actual testing of drugs, you know, before they they have to test them in the physical world. And that's like, obviously working, you know, super well. And I think I see this before, too, with like autonomous vehicles and stuff. It's like, there's a big lag there, right?

• 26:49Speaker 01

  Before you get something through like all the clinical trials. So no drug developed by ML has gone through clinical trials. But if you look at the behavior of all of the big pharma companies, I can tell that it's working because they're hiring hundreds of people, right? Like, you know, like, companies will hire like a few people for like an experiment. But they're all gearing up to like operationalize this stuff. And that just gets me really excited. I mean, they could all be wrong, I suppose. And I don't really have any insider knowledge, except for the seats that get bought on, you know, weighted biases. But when I see that, I get pumped because I just like, you know, the drugs that they're working on, you know, the diseases that they're curing, it's like the ones that like, you know, like our relatives have, right? Like, you know, Alzheimer's

• 27:31Speaker 01

  and Parkinson's and these are kind of horrible things. And I think there's just like huge promise in being able to do physics, like inside a computer versus in the world.

• 27:40Sarah Guo?

  Yeah, I think there's a, I think this is a really important point, too. It's actually commonly said, like, no, no machine learning developed drug has actually come to market today. But it's a backwards looking metric in a very slow industry, right? Like, the clinical trial cycle is very long. And so I'm actually like, quite, quite optimistic on this.

• 28:01Speaker 01

  Yeah, I don't think that's a, that stands out in pharma, because it's very under discussed, but there are certain venture funds that have done incredibly well financially in pharma, where there's one in particular I can think of that never shipped a drug.

• 28:13Speaker 02

  Until the COVID era and they were in business for 20 years.

• 28:15Speaker 01

  Wow.

• 28:16Speaker 02

  And they made all this money and they founded all these companies and none of their biotechs ever launched anything in the market.

• 28:22Speaker 01

  Wow.

• 28:22Speaker 02

  So I think that's a, that's a broader sort of issue with pharma and we can talk about that, I think, some other time, but it's, it's kind of interesting how, how little biotech has actually delivered and there's been amazing deliveries, right? In terms of different drugs and things, but it's, it's actually more common than just the ML side, I think.

• 28:37Speaker 01

  Yeah.

• 28:38Sarah Guo?

  Lucas, you, okay, so pharma is something you're excited about and you think has promise and, and growth in at least seats of one B. Figure eight, like you talked about, you know, Yahoo, eBay, like it's a very small set of people. Who else do you see in the weights and biases, like customer base now? Like, how has that changed since? It's, it's actually incredible to me that you've been, you know, working on this from the entrepreneurial side since 2007, because it's like, you know, pre, pre even deep learning revolution, right? And so I imagine, you know, you've got a much broader user set now.

• 29:13Speaker 01

  Oh yeah, it's so cool. I mean, the coolest thing about running weights and biases is the customer set is everyone. I, I really think every Fortune 500 company is doing something with ML that they like actually really care about. And it's always surprising, right? Like we work with, you know, most of the big game companies. Like I'm not a big gamer, so like I, you know, like I'm vaguely aware of like Riot Games and like Unity and stuff, but you know, but they do all this cool stuff with ML to like, you know, make the games more fun, to make like, you know, models in the games. And this is like big investments they really, really care about because, you know, again, like we're sort of the last step in your journey is to want good tooling for your ML team. You kind of need something to work, so you hire an ML team, you get into production, then you like run into problems, then you come to weights and biases.

• 29:55Speaker 01

  So like, we see stuff, you know, after it works and, and like, you know, like ag tech, like we work, you know, big agricultural companies, I'd like never heard of some of them when they showed up. And then there's like these huge, you know, businesses that are actually using ML to find ways to do like cleaner farming. Like a lot of the reasons, you know, you, you spray a whole field with, with pesticides is just because it's like so expensive to do something smarter. And so, you know, I think, I think that like. Crop yields and the, you know, the, the cleanness of the, the farming practices about to like dramatically improve. Like, you know, we worked with John Deere for years back from a figure eight days to, you know, weights and biases and they're, they've deployed sprayers that only target the weeds in, in fields. It's deployed. It's like, you know, I remember like for years seeing pictures on the wall and them showing me like prototypes. And then one day they're like, yeah, you can like buy

• 30:40Speaker 01

  this, you know? And it's, it's cool. Cause like this intelligent stuff. It's like software, right? So it's like, it's not like a machine. You just like press copy and then you have, you know, more of it. And so, so yeah, I mean, we see that. We see like a lot of,

• 30:54Speaker 01

  you know, I mean fintech, probably obvious to you guys, but like, they're kind of, I think always out in the forefront, you know, of this stuff for lots. I mean, like there's like consumer oriented stuff that you'd recognize, like, you know, making chatbots not annoying, right? And then there's like, you know, kind of more, you know, financial forecasting and, and things like that. But yeah, I mean, it's funny. We, we don't do any vertical based marketing because there's not one vertical that's like dominant enough to, to warrant it. And our customers bounce around between verticals so much that I think the common thread here is people doing like ML and data science versus any particular application, which I just do is super cool. That means it's sort of like table stakes, you know, for everyone.

• 31:33Sarah Guo?

  You, you know, made jokes, I think jokes about like not being a terribly good engineer. And now the weights and biases messaging is very much about developer first, right? Can you talk a little bit about how you think about like, you know, and it actually, it is like, as far as I understand, it's like one of the most broadly adopted tools by developers working in ML. How do you think about like developer adoption versus like researcher adoption? And what did you do that worked?

• 32:03Speaker 01

  Yeah. I mean, it's like developers and researchers that kind of blend together. But I think that, I think that what happened in the sort of ML app space. Is that you got a lot of, well, the early companies had to sell to executives, which I totally understand. Like, that's what CrowdFlare had to do. And the, the problem there is you kind of get stuck in these like multimillion dollar deals. And like, you just can't get out of that. Like, you can't switch to like a PLG motion. And so the early companies, I think, are kind of stuck, right? With like, these products that like CIOs love and the, you know, engineers hate. And that's just like, I just didn't want to do that with weights and biases. No matter how big the market is, or how juicy that is. And the good news is, it's like, not a good market. Like, a developer-oriented sale is better.

• 32:45Speaker 01

  When you, when you look at like developers versus ML researchers, that line is really blurred in the time that we've been doing it. And, and I think that like, there's sort of like subtle differences. But, you know, when Nvidia came along and these chips worked for deep learning, it just like broke the entire stack. Like, it was like the first time that in, in my career where I'm like running into like, like linker errors. I'm like, what the fuck is a linker? Like, I vaguely like remember this, you know, from, you know, like a CS class I took, you know, like. And, um, and so it's like, I think that ML researchers really had to be kind of become software developers. And then at the same time, you know, the AI class is the most popular class. So like, all these software developers and smart ones kind of become ML researchers.

• 33:29Speaker 01

  So I think that line is weirdly blurred. But then I, I think there's a funny thing that also has been happening where, like, every DevOps person on the planet rebranded themselves as like an MLOps person all of a sudden. And so you get all these companies that come out of, like, every MLOps team then realizes they could raise like a shitload of funding, you know? And so like, you got like every, every major company, their MLOps team like went off and like raised money to like make a new product in the market, which I think from an investor, that's logical, right? It's probably they have a good thing. But they're just like not good at connecting with actual developers, right? Because they're actually like, DevOps is, is like a little bit of a different discipline, where you're sort of obsessed with reliability. Kubernetes seems like simple to you. And that's just not.

• 34:14Speaker 01

  Like the experience of like an ordinary developer, like, you know, like, like my co-founders or me. And so I think that the joy of weights and biases is we're kind of making software for like ourselves. And I think it turned out that like, maybe in the median of my three co-founders was actually the, the target audience for us here. I think I skew more towards, you know, an ML researcher barely. But, you know, if I had to like, pick one end of that spectrum and, you know, my co-founder Chris probably skews more towards software developer and Sean's probably somewhere in between.

• 34:45Speaker 02

  One of the things that's common to people or to developers is that they love to write their own tools. And they tend to really enjoy using open source over closed source solutions. How did you think about the open versus closed source approach? And how did you think about, you know, making something that's valuable enough and good enough to overcome that natural inclination to just do it yourself?

• 35:03Speaker 01

  Well, it's funny, like, I think the tools thing, I've always felt like, I've always felt like kind of proud of making tools for developers. Like, that's always felt like really good because I think developers sort of know what quality is like. I mean, it's like, I kind of like making a tool for someone that could make the tool themselves because it kind of raises the bar. And this definitely, my grandfather was like a pattern maker, which is like a sort of, you know, like the person makes a pattern for other machinists. And he had the same attitude of like, look, I'm making this stuff for like other engineers and like, there's like an honor in that. So I definitely feel that pressure and love it. The open source versus closed source thing was really just like, we didn't know how to make an open source business.

• 35:46Speaker 01

  So, so like, we kind of started off closed source because we just, we actually wanted to have like a working business. And it's had like a pro, there's been a major pro, which is that all our competitors are open source. And what that means is that they don't get to see how users actually use their software. And so I think our software is a lot more ergonomic because we have like metrics on what people actually click on. If people aren't clicking on a button, we remove it. If people like, you know, pick an option all the time, then we know to like, make that the standard option. As we've grown and you kind of can't just like rely on anecdotal user feedback, that I think has made our product like a lot better. Like people find it like nicer to use.

• 36:30Speaker 01

  At the same time, I understand why people want to go to open source stuff, but. Honestly, I feel like it's a little bit of a DevOps mindset also. Like, I mean, DevOps people, like, they're obsessed with, like, you know, open source. And usually, like, the ML ops people we talk to in companies really want, like, an open source piece, which is why our client is open source, the thing that actually runs in your server is is open source. But like, I don't know, like, ML researchers aren't so precious, in my experience, generally. They kind of want to get a job done. And I think they're kind of happy to, like, that we have, like, a stable, like, business that generates money in, like, a normal way and isn't going anywhere. Or at least, that's what I tell myself.

• 37:08Sarah Guo?

  I think this is like the the part about, like, the need for, like, ongoing telemetry and application feedback. Like, there are a, you know, zero to marginal number of open source applications that have actually succeeded. I think part of it is like the sort of, you know, hierarchy of honor of, like, the deeper in the stack you go, like, do people really want to work on, like, web UI in the open source or just like random business logic on a relational database? Like, it's not as sexy and exciting to, like, go put your, like, GitHub badge on. But I think the piece that you described is actually really important, where, you know, you work on complex workflows. And if it's something that, like, somebody can just run in infrastructure and, like,

• 37:51Sarah Guo?

  you know, you you get data back on, like, config files or YAML or whatever, like, that might that might work in terms of, like, one person's architectural point of view or some framework. But I really don't think it works at the application layer for for these two reasons, right? Like, one, total lack of feedback, and two, sort of the lack of interest in the, I don't know, technical brownie points you get for it.

• 38:11Speaker 01

  Yeah.

• 38:11Sarah Guo?

  Do you still pay attention, I'm sure you do, actually, to, like, annotation? Like, what do you what do you think happens to the data annotation space and, like, you know, the land of LMs and RHF and such?

• 38:24Speaker 01

  You know, I'll be like, honest, actually, this will be like, totally honest. I find it like incredibly stressful because I still feel bad that we lost the scale. Like, I still like, it's just like lingered with me and I I admire scale. Actually, I know hard that that business is, so I have just like deep admiration for their, like, execution. But as a competitive guy, I kind of can't get over it. So I've like always inundated with questions from VCs, like, whatever any annotation company's raising, I know about it because everyone like calls me. But I I honestly try.

• 38:54Speaker 01

  I know I should be closer to it, but I try to stay away from it just because it caused me so much anxiety to look at what's going on that I just can't deal with it.

• 39:02Speaker 02

  What were some of the things you did differently with the second company? I feel like, you know, I've started two companies and with the second one, there's all sorts of lessons I applied immediately. Were there two or three key takeaways that when you started with some biases, made the second time around easier or was it harder? How did you think about, you know, key, key, key learnings or how to apply new things?

• 39:21Speaker 01

  Yeah, I mean, I think like one thing was like extreme clarity about who we were serving. So I'm surprised I don't hear this more because like the, the ways that biases started with a, with a customer profile. And I think it's actually a nice way to start a company because, you know, especially as like a founder, you have to spend so much time with your customers, you have to seek them out. Like picking a customer that you love, I think is a really good thing for your like mental health, you know? And so that was like a big thing. And then I think like, I think I've just been a more confident person in myself. Like anytime I start thinking like, okay, like long-term versus short-term, it's just like, you always want to think long-term, like everybody wants you to think short-term. Like everyone's going to push you to think short-term.

• 40:06Speaker 01

  They wouldn't say it like that, but it's like, you know, it's like people can see like ARR growth. They can see like user growth. They, it's harder to see like product quality, right? And so I think like, I think I'm a competitive guy who likes, you know, metrics and likes accountability. But I actually think that can get counterproductive for me where, you know, you start like sacrificing short-term things to grow these external facing metrics. And I just really try to fight that myself. I think everybody like chases, every entrepreneur chases like short-term, like ARR numbers like in quarter. But then it like hurts your growth rate the next quarter. It's like, it would actually be better always to like push out deals. But like, nobody thinks like that, right? You can't think like that, but it's, I don't think it's totally rational.

• 40:49Speaker 02

  Is there any advice that you'd give to founders who are running their first AI company or just getting up and running?

• 40:55Speaker 01

  Yeah, you know, the advice I always give is like, it's like the generic advice that everyone says it's like even truer than you think. It's even truer than like I know, even though I like deeply believe it. So it's like caring about like if you're making something people want, like everybody knows it. But like, no one cares about enough, right? Like people just, they get distracted. They do other weird stuff. Even I do it. I understand. But like, you should care more than you think, no matter how much you think. I've never met anyone that cared too much about that. And then spending time with customers, it's like, it's so critical. Everyone says they do it. But I don't really believe it. Like, I feel like I'm obsessed with this. I mean, like getting like, when you're an early company, getting like three customer calls in a week.

• 41:35Speaker 01

  That's like tough, man. I mean, you gotta like scrape and claw and like beg to get those meetings and you know, like two of them are gonna like cancel. So I don't know. People tell me, oh, I met with like 30 customers this week or something. It's like, really, did you? Like, I don't know. I, I, I tried really hard to get customers attention. So I don't know. I have this feeling that nobody does enough of that, but I don't really know. I think people are lying to each other about how much like actual kind of customer meetings they're doing. And then it's like, you know, when you get to a customer, it's so precious. It's just like, man, like show up prepared and like ask the tough questions. Like, I think like, I feel like one thing about me is like, I always like default to like wanting people to like me and it's a terrible trait in a, in a CEO.

• 42:16Speaker 01

  You know, it's like, I feel like I have all these like coping mechanisms for myself to like, not just like kind of flip into that mode. But I think it's good for customer discovery because I'm always like so afraid that they secretly like hate my product, you know, that I, I get like really insecure. And I'm just like, okay, like, you gotta tell me like more, you know, like, like, are you sure this is really like working for you? Actually, it does actually help in that one important like entrepreneurial process to lean into your insecurities with your, with your early customers.

• 42:43Sarah Guo?

  Lucas, this has been great. Is there anything you want to talk about that we didn't cover?

• 42:46Speaker 01

  No, this has been fun. I mean, I just, I think the message that I'm trying to tell the world is that we're really trying to make tools for this new LLM workflow that people are calling LLM ops. And so my, my advertisement for Wits and Biases is like, hey, if you knew us and liked us for our ML op stuff, try our LLM op stuff called prompts. I think it's, I think it's not amazing yet, but I think it's kind of ahead of the market and it's about to get a lot better because we are like investing every, every resource that we have into making as good as possible. And we're really listening to feedback and iterating. So if people want to, you know, email me directly and tell me some issue they had with prompts, I really want to hear it.

• 43:23Sarah Guo?

  Is it, is it [email protected]?

• 43:25Speaker 01

  Yeah, Lucas with a K, yeah, at 1b.com.

• 43:28Sarah Guo?

  Okay, you're gonna get a flood. Well, I'm optimistic. You're such a pioneer here. Thanks so much for doing this, Lucas. That's great.

• 43:35Speaker 01

  Thanks, Ross.

• 43:35Speaker 02

  Yeah, thanks for joining.