Ep 7: Co-Creator of Databricks Dolly Mike Conover on Open-Source LLMs

Patrick and Jacob sit down with Mike Conover, Staff Software Engineer at Databricks and Co-Creator of Databricks Dolly, the world’s first truly open instruction-tuned LLM, to discuss the magic behind Dolly, Alpaca and other instruction-tuned LLMs, the unreasonable effectiveness of fine-tuning, how they got all Databricks employees to help them curate the Dolly dataset (hint: google forms), and more.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to Unsupervised Learning. I'm Patrick Chase, partner at Redpoint Ventures. I'm joined by my colleague Jacob Effron. Today, we are so excited to have Mike Conover on the podcast. Mike is a staff software engineer at Databricks focusing on large language models. He is the co-creator of Databricks Dolly, the world's first truly open instruction tuned LLM. Dolly launched a couple of months ago to some amazing traction. Mike, thank you so much for joining us.

• 0:29Speaker 01

  Oh, it's truly my pleasure.

• 0:31Speaker 00

  Uh, maybe to start, it'd be awesome to just hear a little bit more about about your background. Uh, I think you have a really unique experience in machine learning, doing both big companies and startups, um, applications and infrastructure, and also kind of traditional machine learning and LLMs. Um, and we were fortunate enough to to work together at LinkedIn. Um, I had a lot of fun there and I I saw some amazing LinkedIn folks on the Dolly paper as well, um, with like Sam Shaw and Matt Hayes.

• 0:57Speaker 01

  Working together on Feed Relevance at LinkedIn with you. I have a flashbulb memory of sitting in my kitchen in San Francisco, uh, late at night, and you and I just going back and forth and that feeling of like, man, this is working. Like what we were building is this is going to be a thing and it it was. So yeah, Patrick, those are some of the best best times in my professional career was working together.

• 1:17Speaker 00

  I appreciate the kind words, though, and I have the same recollection, but I was sitting in my bedroom, uh, coding away.

• 1:23Speaker 01

  Yeah, yeah. It was, it's great. And, you know, I think gave me a taste for, uh, what it looks like to work with strong teams on interesting problems and sort of like, those experiences shaped a lot of the the subsequent ways that I've chosen to invest my time. You know, a lot of the, and I'm, you know, I'm feeling a little philosophical this afternoon. So a lot of my, the ways that I see the world through the lens of technology involve the digital trace data that people create implicitly and how that can be leveraged to help increase human effectiveness. So my my PhD research was on the structure of, uh, communication networks and how information moves through social networks, particularly in the areas of propaganda and misinformation.

• 2:07Speaker 01

  So this was DARPA-funded research, uh, dating to You know, 2008, 2009, we had access to the Twitter data set and that large scale graph structure and language embedded in a social network reminds me very much of swarms. When you think of how large flocks of birds move together and that's sort of like emergent behavior that they seem to exhibit effortlessly, you know, now in many, many years of working with digital trace data from humans, I'm convinced that people exhibit the same behaviors. We're just generally not able to see it because we are in the swarm. But when you step back and you look at planet scale web crawls and synthesize these using tools that allow us to see these relationships more clearly, the hidden structures that

• 2:52Speaker 01

  circumscribe our lives become very evident. And that led me to LinkedIn where you and I worked together and looking at human mobility, led Feed Relevance at LinkedIn, one of the first pass rankers for Feed Relevance at LinkedIn, and joined, did a bunch of good research there, but really got that taste for like, oh, you can apply these tools to useful problems that affect millions of people. Peeled out of LinkedIn, joined Sam Shaw and Pete Scamrock's startup Skipflag with my colleague Matt Hayes, who's my, you know, partner in crime on Dolly. So we've been working together for quite a while now. Skipflag was the smart enterprise knowledge graph that builds itself from your workplace communication before transformers.

• 3:35Speaker 01

  So been thinking about this stuff for a long time. Did a stint at Workday, loved my time there, saw the enterprise, led a pretty large and highly, highly effective and operational machine learning team. And just kind of got that taste for amazingly like enterprise software. And then now Databricks where, you know, we, we have this, if you think about kind of tying it back, and the reason I started with the swarm theory analogy, if you think about how social insects imbue their environment with chemical signals to coordinate their work, like you think of a honeycomb matrix as having chemical signals that say this should be an incubation cell, this is for storage, and you sort of that collective decision-making process.

• 4:18Speaker 01

  Databases are what humans use to do that same thing. Each cell in a honeycomb is like a record in a database. And it's not an exact metaphor, but it's like also not, not what's happening. And so this idea that language models, as you see sort of the ballistic trajectory to ever-expanding context windows that can perceive simultaneously a non-trivial fraction of all the records in a database and attend in a context-dependent way to all of the information that your company knows about what it has seen and done and then reason about it, that's an exciting business to build.

• 4:57Speaker 00

  I love the analogy to a record as well because I, the first thing I thought of when you were talking about humans moving as swarms is almost the research and what one group is doing affecting another group where it's kind of like, you know, Stanford came out with alpaca and then you guys have DALL-E, which is kind of like in some ways building on that.

• 5:17Speaker 01

  Totally.

• 5:18Speaker 00

  You know, an endless list-

• 5:19Speaker 01

  Yeah, many, many ways.

• 5:20Speaker 00

  Of llama metaphors or whatever it is.

• 5:23Speaker 01

  Ungulates.

• 5:24Speaker 00

  Yeah. Yeah, you always had a much better vocabulary than me. The, but yeah, I mean, I guess it'd be, it'd be awesome to hear a little bit more about kind of DALL-E and what led to that project. You know, you guys launched it a couple of months ago. I saw your nearly 10,000 GitHub stars now and it's been trending on Hacker News all the time and on Twitter all the time. So, you know, maybe it'd be great to just hear a little bit more about the project and how that came to be.

• 5:54Speaker 01

  Yeah, of course. So I think we recognized, particularly with the alpaca finding, that there was a suggestion that it may take fewer resources than you would have imagined to breathe life into this general-purpose instruction-following behavior. And I think it was unclear. So we had, we've had large language model work in flight for quite some time. And so a lot of the public DALL-E stack is work that we had just, you know, for projects I can't really speak to yet, stay tuned, just had on hand. And so we saw the alpaca finding and it was like, well, first, we couldn't get the llama weights. And like, you know, I get it.

• 6:39Speaker 01

  We too got a lot of inbound emails. I don't take it personally, but it was just like kind of a practical matter that we did not have the llama checkpoints. And so it the OPAC data set was out there and it was, you know, our leadership team is very, very attuned to what is happening academically and also in the marketplace. And I think suggested that there may be something here for the enterprise. And so kind of a necessity, I had worked with GPTJ and it was like, I wonder what will happen if we instruction tune this old two year, this two year old model. And that evening, it was writing love letters to Edgar Allan Poe. And I

• 7:24Speaker 01

  don't know if you've ever worked with GPTJ on like not dev or the hugging face space. It will not do that unless you teach it how. And so I think at that point, yeah, it just, it begs a lot of questions about like, because I mean, Patrick, like imagine that I came to you in July of last year. I swing by your house with my laptop and like, look at what I've built. If this has been possible for years, GPTJ was released in 2021, and it took, it took Databricks 10 days to release to build the instruction following data set that we released as open source that gives these old models this capability. Anybody could have done this for at any point in the past two years. That's kind of weird.

• 8:03Speaker 00

  Wild.

• 8:04Speaker 01

  Yeah.

• 8:05Speaker 00

  Do you think it was the finding at Stanford that, that, you know, made this all, they were like, okay, this is possible. Now I should go do a commercial version or like, I guess, why you guys? And why Databricks if, if anyone?

• 8:17Speaker 01

  So a couple, a couple of things. So I think there's an element of it, which is like the four minute mile. You know, I know you're a performance athlete and like now it's a two hour marathon. It's like, nobody could run a four or four minute mile until everybody could. It was like these sort of dawning recognition that this is physiologically possible. There's some neurochemical override that people hit walls. I think there's part of it is like, wow, this like chat GPT is the existence proof of this technology. And it begs the question, how can I build it? So people start asking. The other thing is that, you know, I think, I think there's a good hearts law effect in play, which is that you, you know, any metric that you define as a metric becomes useless in the limit.

• 9:02Speaker 01

  The if you and this is on our hugging face page, you know, we led with this. Dolly is not a state of the art model. I have gone to great lengths to express that it is it. There are many, many things that it cannot do. It is remarkable that it does what it does at all. But the benchmark metrics, when you look at like Stanford Helm or Luther's, I actually don't know how to pronounce that team's name, if it's a Luther I or a Luther.

• 9:28Speaker 00

  I actually don't either. I was I was hoping that you would pronounce it first and then I could learn how it was pronounced because I I knew that you guys that you started with their weights. And so I was like, I hope I hope Mike says this before I do. But yeah.

• 9:39Speaker 01

  It's better than Luther. Better than Facebook's vector search index, which nobody like literally nobody knows how to phase face what F A I S S. Anyway, so I think, you know, if you look at the found the base model, G P D J and Pythia 12 billion, the standard benchmarks relative to the instruction tune model, they're just really not distinguishable. But the behavior just could not be more different. And so I think, you know, researchers kind of focus, you know, got this kind of tunnel vision and focused in on, you know, can you pick the right answer out of a multiple choice list? Those benchmarks are things that were easy to automate the creation of, you know, Boolean question answering.

• 10:23Speaker 01

  And they I think they instrumented more of an understanding aspect. And so you kind of chased these metrics. The other thing is that even the Databricks Dolly 15 K data set probably represents, let's say between 500 and 1,000 hours of writing time.

• 10:40Speaker 00

  Wow.

• 10:41Speaker 01

  I think there was a local minimum that the scientific community occupied, which is that like the unsexy work of writing poems for computer scientists who are like, I just want to make G P U's go burr. Like, it's just like, I'm not gonna like, spend 500 hours writing summaries of Wikipedia articles to see. If maybe that's useful training data. And so I think that's part of the recognition as well that, oh, this can be done.

• 11:10Speaker 00

  I mean, you got the entire Databricks org to do that in pretty short order, right? It seems like there was some latent demand for that.

• 11:16Speaker 01

  Yeah, true. And I mean, you know, I've said this before, but like, it's a true credit to our colleagues that everybody took it real seriously. And I think you now you see that a lot of the foundation models that are being released have, you know, the Mosaic ML just released their MTB 7 billion checkpoint. And one of the things that they released was MTB 7 billion instruct. And it's fine tuned on Anthropic and Databricks Dolly. And it's like that, you know, is a durable, valuable AI artifact. And I think that is super cool. And people wanted to be a part of that. And yeah, our team really rose to the moment.

• 11:57Speaker 00

  Amazing. And just like to piggyback off of that. So what exactly went? I feel like the data curation is one of the most fascinating things of like, what went into that? And kind of how did you orchestrate that task across Databricks? So like, hey, like, you know?

• 12:14Speaker 01

  So Google Forms.

• 12:17Speaker 00

  Good to know.

• 12:18Speaker 01

  Actually, yeah. You know, perfect's the enemy of good here. And it turns, so I think it's super interesting and not well understood, but also very important to the enterprise and startups generally. How the composition of instruction tuning data will shape the resulting model behaviors? Like, we actually don't know really how to measure lyricism. I mean, I don't know if you guys saw the tweet from the founder of Replit, but he talks, they just released a code generation model. He talks about doing a vibe check. And there is like, that's some real talk right there. It's like you can human eval all you want. Like, at the end of the day, you're like, what's it like?

• 13:03Speaker 01

  I think that.

• 13:06Speaker 01

  It's funny, but it's also, you know, if you're building tractors, doing drug discovery, I really do not want to spend a million bucks on data and hope that the resulting model exhibits. I mean, you don't have to, you don't have to spend a million dollars. It's much, much less than that. But like, I think there's some new science that needs to be developed to understand how the qualitative character, the define tuning data, shapes the resulting model behaviors. We followed the instruct GPT paper. So they define several different kind of categories of behaviors. And if you think about these like, like eigenvectors, I think what happens is that the different, so like summarization, closed form question answering. So that would be like, given a Wikipedia passage, answer a question about it. Open question answering, you know,

• 13:50Speaker 01

  how old was Abraham Lincoln when he died? Brainstorming, they rephrase this. There are a couple other behaviors that that paper outlines. I think what, and you, I don't know if you guys are familiar with the FLAN dataset. So we, we hosted Shane for a talk, one of the authors on that paper for a talk. And like, it's 1800 or so different task demonstrations. And I think of those kind of as regularizers, where the model must, it cannot adapt to one thing. It must interpolate between all of these different behaviors, letter writing and summarization and bulleted lists and like, write it as a short form, write it as a long form. And that what

• 14:33Speaker 01

  happens is that in order to effectively compress all of the required behaviors, these models learn an internal representation of the world.

• 14:46Speaker 00

  That's fascinating. I never thought of them, the analogy to regularization. And yeah, I guess what you're saying is if you, by having a diverse set of tasks and data curated for those tasks, then The model learns more about the world, I guess, than it were if it were to just do one kind of task.

• 15:07Jacob Effron?

  I feel like you've kind of catalyzed now this, you know, to your point, like you could have always used this instruction fine tuning, you know, with some of these models that have been around. The ChatGPT, OpenAI paper had been around for a bit. You were really, I feel like, the first ones to prove, like, hey, you can, you can do this, you know, with your own kind of labeled data. It feels like there's, you've unleashed a wave of other folks that are probably going to be trying to do the same. And so, you know, a few questions around that. The first would be, you know, I'm just curious what you guys learned through the process of gathering this data. I mean, obviously, I assume there are other folks that will read the InstructGPT paper and then try and replicate some of this stuff, you know? Any wisdom to share for, for the next set of folks that are trying to build, you know, whether it's domain specific or whatever it is on top of this?

• 15:48Speaker 01

  Yeah. So I first, I want to recognize that, I mean, the Alpaca team definitely had the critical insight. And I think there are, just, you know, there, with many eyes, all bugs are shallow. There are a lot of people doing interesting work in this space. You know, GPT-4 all comes to mind as well. But I, we have started to, there's some function that is not well understood, or at least the people that understand it are not talking about it, in between how that composition of the data shapes the resulting model behaviors. One interesting observation I'll make is that the Alpaca data, the DALL-E model trained

• 16:32Speaker 01

  on the Alpaca data set, DALL-E V1, can write letters in a well-formatted template. It understands salutations. The DALL-E V2 does not. It can write letter-like prose, but it doesn't understand what the beginning of a letter looks like. And so I think that is because... Our annotation rubric was, like I said, Google Forms, so relatively Spartan. You know, there's kind of like how in part you generally pay annotators to do this work. And so you can be very specific. So I think that there is demonstrating the capabilities that you want.

• 17:12Speaker 01

  And you know, you think about like a business, like an enterprise, doing this using an internal team or like, hey, I have my customer support tickets and I want to ask questions or I have some sort of corpus of pre-existing data that I want to augment with, you know, reasoning demonstrations. I think asking the same questions slightly different ways. So there's like a pretty cool paper recently called Wizard ML that describes how to sort of expand prompts and like make them more complex, make it simpler, diversify in domain.

• 17:47Speaker 01

  So I think asking the same question many different ways, asking in many different, and these are like the kinds of prompts you would pose to a human annotator, many different formats. Like if you want a model that can write bulleted lists or produce JSON, demonstrate that capability to the model. And I think there's a lot of work on like using, there's this, I forget if it's called UL2, like a sort of a spiritual successor to T5. Using, you know, a large open source model to generate samples and then having human annotators review those samples.

• 18:30Speaker 01

  There's a, I think a paper from OpenAI on like self critique, where you can use a model to like critique a human written summary. And then the human is able to like rewrite the summary more effectively. So I think there's this like flywheel that you can get spinning, which is like my human annotation process as augmented by models that supervise and generate. So that you're just sort of like nudging the flywheel rather than requiring humans to think up absolutely everything that your model knows.

• 19:02Speaker 00

  I guess something I was curious about was. The you guys, Alpaca, like, I feel like the community has shown that fine tuning these models is remarkably effective, right? Like, the you get the open source, checkpointed model, you curate some data, which obviously is really hard to curate the data. But the model, you know, I think Alpaca costs $600 to train. I forget how much DALL-E

• 19:27Speaker 01

  Thirty bucks.

• 19:28Speaker 00

  Thirty dollars.

• 19:29Speaker 01

  Thirty dollars, ladies and gentlemen. You can do it in an afternoon on Databricks.

• 19:32Speaker 00

  And so I guess, like, do you think that is going to be the predominant paradigm? You know, it seems like Mosaic and others are still training from scratch. They obviously have a different kind of business. But if I'm like, an enterprise or startup, you know, I would imagine I would start with open source, fine tune on maybe the data that I have, or just start with it open source and see how it works.

• 19:55Speaker 01

  I mean, I think that's right. I think that's right. The foundation models are highly, I mean, as a in large measure, as a function of being trained on the same corpus, right? Yeah, very similar architectures. Well, and you could say, well, you know, maybe maybe all the work in private hands is going to be happening on architecture innovation. But I think like, these pre-trains are highly correlated in their generative outputs. And in some measure, they like do two things. One is like they compress language capabilities, and they also compress world knowledge in a world model. Now, I believe, as somebody

• 20:38Speaker 01

  who works at a database company, a lake house company, that you're that retrieval augmented methods are important, and that like databases exist. You don't have to compress everything there is to know about the world into several gigabytes of state. Yeah, and that, you know, you can use tools. So I think the pre-train.

• 21:07Speaker 01

  Absent some new like cost, a unit economics, I think, you know, Mosaic is doing a good job of driving down the training, you know, it's

• 21:14Speaker 00

  Totally.

• 21:15Speaker 01

  It's also the, just the availability of GPUs. Like, I don't know if you've tried to get an A100 lately, but it is, uh, you know, if you need a thousand of them.

• 21:25Speaker 00

  Yeah.

• 21:25Speaker 01

  You may have wanted to ask some time ago. Um, but you know, it's like, you can do serious work on just a few on V100s, you know, you can serve on A10s. So, you know, I think that, um, pre-trains are probably like largely commodities, fine tunes, there's going to be a lot of that, but it's still a point, you're not, I run models, like my, our team, uh, has online retraining pipelines where we're refitting models dozens of times a day.

• 21:55Speaker 00

  Oh, wow.

• 21:55Speaker 01

  Not for language model, not for language models, but for machine learning models that do important things. The other part of my job is infrastructure optimization, that do important things at Databricks. Language changes very slowly.

• 22:07Speaker 00

  Yeah.

• 22:07Speaker 01

  Right? Like, there's not new language to learn, you know, and there's like news, but again, you can like retrieve news. And so the fine tunes also seem like a point in time, but many more of them than the pre-trains. And then serving, I think is where you're going to see a lot of action. And I think you're going to have a lot of businesses, um, standing up services in their sort of secure environments with their fine tuned models, um, that are just doing real work for the business.

• 22:39Speaker 00

  Yeah, that makes a lot of sense. It's fascinating. The Mate also talked a lot about the, uh, kind of bringing together traditional information retrieval systems with

• 22:49Speaker 01

  Yes.

• 22:50Speaker 00

  You know, that have knowledge and then kind of, um, a language model which has the ability to generate text or make sense of that knowledge. Uh, and I think that makes a lot of sense. Um,

• 23:01Speaker 01

  Completely, right?

• 23:02Speaker 00

  Yeah.

• 23:02Speaker 01

  And then you, you can like learn new facts, right? You can have a stream of, you know, a model operating over a stream. And I think you see some of that with these. Alter huge models that just like the, listen, I can't, I can't, you know, have not operated one of those, but like, it certainly seems like some of these smaller models have valuable use cases.

• 23:24Jacob Effron?

  Totally.

• 23:24Speaker 01

  I think, just to like riff on this a little bit, I think the hallucination is a real problem.

• 23:31Jacob Effron?

  Yeah.

• 23:31Speaker 01

  And I think like, that is one of the, as a family of technologies, like, we need to have a good answer to the question of like, how do you make sure that these things are verifiable? And I think that's like a really interesting opportunity for companies operating in the enterprise space is, you know, if you're like doing creative writing, it's, it's fine, but if you're building jet turbines.

• 23:55Jacob Effron?

  Yeah.

• 23:56Speaker 01

  It, it's important that it's correct. And so I, I'm looking forward to and following closely the literature on correctness and verifiability.

• 24:05Jacob Effron?

  So kind of related to some of the stuff we're talking about, I'm curious, you know, as you think about, you know, the future of these instruction fine tuning data sets, do you imagine this is something that makes sense for folks to build themselves from scratch? Obviously, it seems like you at Databricks did this, this pretty well. Are we going to end up in a world where there are some massive open source data set and you kind of just add a few of your own examples on top? Like, where do you think this space is going?

• 24:28Speaker 01

  That's a, that's a great question and super relevant to our customers as well. I think of it kind of like sandpaper of increasingly fine grain. So like you, the pre-train is just your belt sand or just your, you're even just, you're out in the woods, just chopping down trees, right? I think that these instruction tuning data sets, I mean, even like, if you think about fine tuning, but not instruction tuning, like Bloomberg had that Bloomberg GPT paper, where they have a lot of very, very powerful, non-public descriptions of the world that are, contain things like domain specific jargon, just there's a

• 25:11Speaker 01

  Medical and legal have this too, where there's like a way of talking about things that is not at all like Common Crawl or Wikipedia or News. And so I think there's like that, that's the next level of granularity, which is like, I'm going to show it a million pages of legal documents. And then I think the, the sort of public instruction following data sets, and I think you're going to see the gamification of this. I just saw there's a, something like a $250,000 prize for the best, you know, Chai in partnership with Together Compute. And I think the Gnomic team, actually the guys behind GPT for all, are having this challenge to build the

• 25:52Speaker 01

  best chat, or open chat bot on top of the Together 3 billion checkpoint. I think you're going to see a lot of gamification. Open Assistant is another example of this, where you're going to have these open data sets that have, you know, you can treat it like conversation trees, where you can have the same conversation with multiple people and see different responses. I think you're going to get to a pretty large corpus. Anthropic has its helpful harmless data set, which is like a supervised fine tuning for non-toxic behavior. You're going to have this most like kind of a cornucopia of open data sets, but I think nothing beats like, man, this is just laser focused on my domain. And I think that's that super fine grain, like real polish, kind of make the thing exactly

• 26:35Speaker 01

  what you want it, make sure that it exhibits the specific behaviors that are important to your business. I think this is why companies like Scale and Labelbox have a bright future ahead of them, because there's just, you can have your team do it, you can hire it out, but at the end of the day, like you just want really, really exact demonstrations of the behaviors that are relevant to your business. And also, there's a lot of, like, customer support tickets are a good example. If you have an, like, an inbound and an outbound, and you actually have instruction pairs, sort of, or like, you know, fine tuning pairs already at hand. And so I think there's going to be a lot of that, too.

• 27:13Speaker 00

  Yeah, it's fascinating. The Because I was thinking about the data labeling as well. And obviously, it was super important in kind of machine learning 1.0. And I feel like, you know, the stuff that we did at LinkedIn or those types of models, like more traditional machine learning, it's like better data wins every time, right? And it seems like the, but then I was, I was kind of, you answered this question, which is like, it seems like the data is still quite important and curating the data is still quite important because it seemed like, you know, maybe with so much data or everyone's, you know, throwing everything they have at these, they all kind of do the same thing.

• 27:53Speaker 00

  And, you know, maybe there's less importance to having the kind of last part labeled.

• 27:59Speaker 01

  Yeah, I think like, you know, think about like question answering over like tractor repair manuals.

• 28:05Speaker 00

  Yeah.

• 28:06Speaker 01

  You really, you really just need examples of tractor repair manuals to solve that problem effectively. I also think, and this is what, this is kind of one of the things that I like about working in the enterprise space, is that a lot of the consumer web moves that you can make in terms of like impression and action tracking and like click stream data, apply to the enterprise. It's just not a super well-worn playbook, but I think there's a lot of value on the back of taking these, these opportunities. There is going to be the AdWords moment for RLHF or whatever actual technical solution becomes the way to imbue these models with the revealed preferences of real humans,

• 28:51Speaker 01

  not just the like greedy search, which is predict the next token. Somebody is going to figure out Bard shows you three samples. Absolutely, they are collecting that data on like which sample you choose to continue the conversation with.

• 29:04Speaker 00

  Yeah.

• 29:05Speaker 01

  What is the correct mechanism for eliciting those preferences from as a product loop where you don't know that you're training, I don't know that I'm training Netflix about what movies I like to watch, but certainly works. I think this is a huge product opportunity. And I think like the UI stuff is really exciting too, because you see.

• 29:26Speaker 01

  I think people have gotten kind of like target fixation around chat as a modality. And also even just L L M's and generative AI, right? Like, I don't know that these are going to be. I mean, I don't know. It's like, I maybe the case. I'm certainly it's above my pay grade to judge. I don't know that these subsume all other machine learning models. I think like, just take the vector search index scenario, like search re-ranking. It making sure that the model is maybe, maybe context windows like render this moot in the next six months. But there's going to be a period here where you will need to like prefer certain results.

• 30:07Speaker 01

  I think like, even if I just have like a list of inbound tickets. Ordering those in terms of urgency, what is the likelihood that this gets escalated? Requires deep language understanding capabilities, but is not a generative problem. Annotating them with tags like that's classical, super classic supervised machine learning. I think like the businesses that succeed in this space will use all of the tools at our disposal. Rather than like trying to make everything fit into this one box, which is generative language model technology.

• 30:41Jacob Effron?

  Very cool. Yeah, makes a lot of sense.

• 30:44Speaker 01

  Are you, what are you guys seeing on the UI front? I if I can reverse interview a little bit, I'm I'm super curious to hear, you know, what's exciting to you?

• 30:51Jacob Effron?

  Well, I mean, I think those data feedback loops, right? Like you've seen it in the image generation companies, right? Like they all give you like, you know, if you mid journey, they're, you know, they give you four, right? And choose one. And I think there's, there's a tremendous opportunity to, you know, layer that kind of stuff back into the product. Or even just to you, you kind of, you have explicit like user choice, but you also have like implicit like, you know, when does the user spend more time on the product or what do they choose to download? Or when are they actually exporting your output and putting it into their workflow tool? And so I think there's lots of, it's really early, like you said, in doing that. And I think it's, it's still an outstanding question, like how much more of that data do you need? I mean, it seems like, you know, in part of the work, the Alpaca team and you guys have done.

• 31:33Jacob Effron?

  It's like, maybe it's not as crazy of a hurdle as everyone thought it was to, to, to get all this data. And I think similarly, and it will be different across different, you know, problem areas, right? But I think there's, there's. You know, it's still TBD, I guess, how much of that, how much that data makes it tremendously better and how much is kind of just, yeah, a few examples, you're kind of good to go. I don't know if you got anything, Pat?

• 31:53Speaker 00

  No, I think, I think you captured it really well. I think there are a lot of people that are early in the journey across a bunch of domains, right? And they're like, hey, we're, we're raising now for like, you know, generative AI on top of legal data or for design or for XYZ. And I think a big part of those, the stories for those companies is like, we're going to curate this data over time. And maybe in the long run, that will give us some sort of defensibility. The, the way that I think about it, which, you know, TBD on how this all pans out, but I think that the kind of distribution and usage and what the users are doing will be more of a moat than the actual data.

• 32:36Speaker 01

  Kind of riffing on this, you talked about like the amount of training data that's required to elicit these behaviors. And I think you think back to, you know, the, I think it was Peter Norberg, somebody, I'm getting my citations wrong, but the, it's not uncanny, unreasonable effectiveness of data was the title of the paper back in 2008. And it's just like, wow, as you scale to millions and billions of records, the like beta coefficients of your logistic regression just gets super dialed in. And that, that was kind of the paradigm for, you know, let's say 10, 15 years, or at least in the consumer web space.

• 33:17Speaker 01

  I think what's really strange to me as somebody who like has a, you know, I'm, I'm a hacker, I'm a builder, I'm a user of tools, but I, you know, have taken many, many hours of statistics, coursework. If I was to go to my professors in 2005 and say, listen, so I've got a 12 billion parameter model and what I'm going to do, you're going to love this, I'm going to show it 50,000 new records. And it's, it's going to be great. I don't, I don't think anybody would have believed me. There's like no reason that these techniques should work, or at least no reason that, you know, sort of in the limited way that I understand things.

• 33:58Speaker 01

  It's surprising and it's surprising that such a whereas historically it's been like you need a billion impression action, you know, pairs. Now it's like, oh yeah, you know, 5,000 records is actually enough to get started. That's pretty, that is pretty powerful. I'll also make a bid, uh, just from a product standpoint on structured interfaces. I think like a lot of things are getting, you know, bundled up in like pure text, but I actually think things like knowledge cards and sort of like, hey, we have sort of slot filling templates that are like UI affordances and I want to like maybe generate with online inference decorations into that interface rather than like the essay. Like I don't always need to read an essay, guys. Like,

• 34:42Speaker 01

  you know, give, yeah, give like thumbnails are helpful. Like, you know, some, you know, uh, but and I, there's like a mirror to cartography, right? Like when you have,

• 34:54Speaker 01

  it requires less cognitively to process an interface you have seen before and you sort of abstract away the like decorative elements and you can attend only to the things that matter. And I think essays in particular are very hard to do that with because there's no salience signal in text natively.

• 35:11Speaker 00

  Yeah, that's fascinating. We, we talked to a couple of people, um, at companies like Canva and Descript and that were deploying these types of models. And, um, one of the things that they have really been working on is giving users control, right? But also not too much control, um, because you kind of want to put guardrails on what they can do, but also allow users to be in the loop of, of what's going on and in the creation process, which is kind of what you were talking about with like buttons versus just the essay. The thing that I was wondering though is like, I think, I mean, some of the models are kind of slow and I wonder if they would, if they would work fast enough to have the buttons kind of ready to go versus one thing that's nice about the language generation is you're like, oh, I see

• 35:54Speaker 00

  that it's doing something and the words are coming in and it's kind of like, it makes me happy as a user.

• 35:59Speaker 01

  I feel like that's a gimmick though. I mean, like even, yeah, it, it's like a design affordance that masks the fact that complete inference to end of sequence token can take 40 seconds, which is, yeah, uh, that, you know, I think, uh, somebody at Notion said like, uh, You know, speed is the best feature. You know, it's like, just make it fast, make it fast.

• 36:22Speaker 00

  Yeah.

• 36:23Speaker 01

  I also, I also kind of think that there's like a, like, what is Photoshop for text like? Where I can like brush? And I say like, you know, given this passage, like, just make it a little more strident. Or a little more like accommodating and sort of like, you saw some of the stuff with GANs, where you would have like knobs where you could control like the degree of facial hair that somebody had. And there were sort of like tunable vector bases within the space of kind of the generative seed. For text, it's kind of like, interesting to think about like, hey, I've not used Grammarly. Maybe Grammarly does all of this already. But like, what does Photoshop for text look like? I think is kind of an interesting, open question.

• 37:08Speaker 00

  Yeah, I remember the, with GANs, they had the style transfer from Van Gogh or things like that. Where like, hey, like, make this drawing look like Van Gogh made it. And you could imagine something similar for text where it's like, hey, make this email look like it was written by William Shakespeare or something.

• 37:25Speaker 01

  Which, and there's a huge market for that.

• 37:26Speaker 00

  Want an email written by someone else, but you know.

• 37:29Speaker 01

  Yeah. I mean, so, you know, kind of like riffing on this, like, because I, do you think the style transfer thing is really interesting? I also think there's, there's a lot of interesting aspects to this problem in the enterprise, I guess I would say.

• 37:42Speaker 00

  Yeah. Super cool. The, maybe last question, then we can move to a quick fire round, if that works. The, I was curious kind of to get your take on the latest in machine learning infrastructure. You guys, you're obviously at the forefront of this. And you said that you thought there'd be a lot of innovation on serving. And I saw the article with that Replit put out, that they were kind of using Databricks plus the, what you guys had shipped to Hugging Face, I think, and Mosaic.

• 38:13Speaker 01

  Oh, yeah.

• 38:15Speaker 00

  And I guess like, how do you think about traditional kind of machine learning infrastructure like Spark and how that fits in with, you know, the infrastructure that people will be using for model training and serving going forward?

• 38:28Speaker 01

  Yeah, so I mean, one of the things that you see, I'll speak from my sort of like, there are many ways to approach this problem. I'll speak from my local environment because it's the one that I know. I think like, one of the things I like about the Databricks tooling environment, and this was not obvious to me when I started working in this space, is it's just you can push a button and get like, all the tools that you need for this specific task, like this, you know, supervised fine tuning of language models. You've got the notebook, you've got the blob store, you've got ML flow, you've got GPUs on demand, you've got a web terminal that you can click into, you know, you can SSH into the machines. It's just like, pretty fully featured out of the box. I think additionally for the

• 39:13Speaker 01

  training aspect, you know, we've got big plans with ML flow. Like, you, you know, I think there is a major, major unmet need for L M Ops. When you think about like, hey, it's hard to, like, I want a service that will just give me fast computes of metrics and run a battery of prompts against my model over time and make it really easy to interact with and understand how the qualitative character of the model evolves over this course of the training run. Because you'll see this, I mean, there are public examples of loss curves on the OPAC data set where validation loss takes a U. And the later checkpoints actually are preferable to the absolute minimum of validation loss. So

• 39:57Speaker 01

  it's really like, a lot of the statistical tools you would use to make these selections are not necessarily appropriate. And I think we are building an incredible platform for training these models. And then you get into the serving space, I think that I know for a fact, that it's a treat to work with models that are served on Databricks, because there are a lot of hard optimizations that require a pretty deep understanding of how these models that won't fit on a single U can be partitioned efficiently and among GPUs that...

• 40:28Speaker 01

  I don't want to have to build it. I just want to push a button and get a service that has all of that baked in. That's, you know, it's a force multiplier, also because you have to interact with these models. I think the like, you know, to kind of talk about that vibe check, like it's really like many, many turns with the model where you see how it behaves in response to many different stimuli requires low latency interaction. If it's 40 seconds per inference, it's just not. And that's like, you know, a lot of these big models with the native hugging face inference just are slow. It's just slow.

• 41:06Speaker 00

  Yeah.

• 41:06Speaker 01

  And so, you know, I think that we were starting from a really strong position. I don't, again, like not to, I'm doing this not to be like a Databricks fanboy, though I am, but more because it's like, it's the, it's the dev tooling chain that I know best right now. And I like to talk about things that I know about rather than to speculate. But I, you know, I think what Mosaic is doing with their training ply, I mean, they're saying you can train stable diffusion from a random seed for 50 grand. You know, like there, I think it's going to get for as a buyer, like it's going to be real low margin. It's just like, how much does this cost?

• 41:45Speaker 00

  Yeah.

• 41:45Speaker 01

  You know, if you're, if you're, if all, if all that you have to offer is like, we have fleets of GPU machines. And so that's where the like tooling, and to say nothing of the LLM augmented developer productivity, which is also, yeah, something we may have something to say about.

• 42:03Speaker 00

  Yeah, that space is, is really fascinating. One other thing I was interested in was, you know, you guys did so much fascinating machine learning on text and MLP at, at Skip Flag. Like, how do you think Skip Flag would be different with the tools that you have today?

• 42:21Speaker 01

  It's an interesting question. I think that that was an incredibly prescient business. I think the, the perception of the need was just way ahead of its time. I think that. And we were dealing with a lot of the same challenges like factualness, right? If you're totally making decisions based on what a model is telling you, or even like extractive summarization has this problem also, which is that like a sentence can be true in its global context and false and it's like, you know, pull quote context.

• 42:55Speaker 00

  Totally.

• 42:56Speaker 01

  I think like those are both methodological and product questions. Like, how do you present?

• 43:04Speaker 01

  How do you choose one to present appropriate context to verify and create verifiable citations? Or sort of like, you know, this is why I believe the following statement. So I also, I also believe that, you know, my background is in how information moves through graphs of people. The enterprise social graph context is highly underutilized as an artifact. You think about all of the interactions on Slack, right?

• 43:32Speaker 00

  Totally.

• 43:33Speaker 01

  Who knows about what in my business is like for any company with more than, let's say, 700 employees? That is a major part of your day is like, who can I, who can I ask about this? And so what's also interesting here is you need to, and this is, this is why like unclear to me that companies like, you know, I don't, I think it's foolhardy to count Google out. You know, they have so much structured data about the relationships between all of these entities and how to, how much to trust signal from them. I think obviously the transformer is the main answer to your question.

• 44:15Speaker 01

  Like you would use transformer based architectures rather than sort of this hodgepodge of deep learning grad student projects that we wound up building skip flag with. But it, I think the essential urge is, you know, and the market need is the same.

• 44:34Speaker 00

  Yeah, totally. The, yeah, it is fascinating because I feel like LinkedIn did a really great job of the intercompany data, right? Like between two different companies, but I think the kind of within a company data,

• 44:52Speaker 00

  you know, still a lot of opportunity there, right? To know kind of who is working on what and who you should, who has the knowledge that you're interested in, right?

• 45:00Speaker 01

  Yes. And you just think about like, just like as a closing thought.

• 45:06Speaker 01

  How many human lifetimes are wasted in pursuit of that? Like, like tracking down the right stakeholder? Or like, who knows what? Or repeating the answer to the same question?

• 45:19Speaker 00

  Yeah, totally.

• 45:21Speaker 01

  It's like an interesting business, but it's also just like these. There's this question of like, how these models will shape society. And I think humans are incredibly adaptable. And I think we also often misunderstand what is uniquely human. If you think about the boilerplate that you have to write in your life, much of that is not actually that interesting. And it's probably like, you think about spreadsheets. It's just kind of rote calculation and spreadsheets. You know, there were highly paid, you know, people with graduate degrees who were like, but I, I do the math. I fill in, I fill in the paper form to run projections. And it's like, you know, all of those people still are gainfully employed. And many of them are still gainfully employed.

• 46:04Speaker 01

  I think that we will focus in on the thing that makes us human. And that's a world that I want to inhabit.

• 46:12Speaker 00

  That dovetails nicely into kind of few closing questions. What are your thoughts on AGI? Is it possible over what timeline?

• 46:23Speaker 01

  Great. This is, this is the lightning round. This is the lightning round. You want a short answer to this question? I think the, anybody who says that they know the answer to that question is,

• 46:36Speaker 01

  that's something to sell you. You know, I think these models can be incredibly powerful and transformative for society without being self-aware. I think that there is, you know, if you want to know what I think about that, I refer the listener to go to Lesher Bach from Douglas Hofstadter and Recurrent Processes and the emergence of self-awareness.

• 46:59Speaker 00

  Awesome. Love it. What's one thing you wish you knew when you started Dolly that you know now?

• 47:05Speaker 01

  You know, so I think one of the things, I think it's been inspiring our executive team. They have the vision. They have, this is a group of people who has done impossible things and had it work. And, you know, somebody says, listen, we should take a run at this. What if, what if you can do it? Yeah. Oh, man, that sound, that's a tall glass of iced tea you got there. Like, I'm not sure. And then it works. It's an incre- it's a really living when those are the kinds of shots that you're taking. And that, I think, has been, you know, something I wish I had had with me

• 47:46Speaker 01

  more clearly throughout the course of my life. I've seen flavors of it, but this is definitely a pretty powerful demonstration.

• 47:55Speaker 00

  Awesome. And lastly, I guess, where can people go to kind of learn more about the work that you're doing at Databricks, or Dolly, or follow you? If there's, if there's any, what's the best way?

• 48:05Speaker 01

  Yeah, sure. So we've got, yeah, we've got a lot of great blog posts on it. We'll probably put a paper up on archive shortly here. You know, I'm Bagabon Jack on Twitter, and the Databricks blog has, you know, we've got a lot of great stuff in the pipe. We recently shipped a FromSpark method for hugging face. Like, we are, like, Dolly is just one of many, many investments that the company is making in this family of technologies. And so, you know, stay tuned.

• 48:34Speaker 00

  Awesome. Well, Mike, thank you so, so much for, for joining us today. This was really fun.

• 48:38Speaker 01

  Yeah, it was great.