Ep 2: Databricks CTO Matei Zaharia on scaling and orchestrating large language models

Patrick and Jacob sit down with Matei Zaharia, Co-Founder and CTO at Databricks and Professor at Stanford. They discuss how companies are training and serving models in production with Databricks, where LLMs fall short for search and how to improve them, the state of the art AI research at Stanford, and how the size and cost of models is likely to change with technological advances in the coming years.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Speaker 00

  Welcome to Unsupervised Learning Red Points AI Podcast. I'm Patrick Chase. I'm joined by Jacob Effron. We are so excited to have Mateo Zaharia on the show. Mateo is the CTO and co-founder of Databricks. In this episode, we dive into the founding story behind Databricks, some of the latest research at Stanford, including the demonstrate, search and predict framework that his group is building and open sourcing, how Databricks is using large language models, the flaws that large language models have, especially when it comes to search and how to fix them, open source versus closed source, and some fun stories like Mateo getting a $50 Amazon gift card for weeks of work

• 0:44Speaker 00

  in the early days of Databricks. I think it's a super awesome show. Thanks for joining us.

• 0:54Speaker 00

  Today, we are so excited to welcome Mateo Zaharia to the show. If there was a Mount Rushmore for AI technologists, Mateo would definitely be on it. He is the creator of Apache Spark, one of the most widely used AI infrastructure projects, and is the founder and CTO of Databricks. Databricks famously built a data platform around Spark that has reached incredible scale. They passed $1 billion in ARR in August of last year, raised $3.5 billion in funding from A16Z, NEA, Co2, and more, and their last round of funding put the company at a $38 billion valuation. If that wasn't enough, Mateo is also a professor at Stanford focusing on computer systems and machine learning and one of the leaders of the Stanford Don Project,

• 1:39Speaker 00

  which is a five-year research project to democratize AI. Mateo, thank you so, so much for joining us. We're really excited for the show.

• 1:46Speaker 01

  Thanks a lot for having me.

• 1:47Speaker 00

  Maybe to start, it'd be awesome to talk about your journey building Databricks. You know, it started as your research project with Apache Spark, and then now it's used by more than 7,000 organizations worldwide, including 40% of the Fortune 500, which is just really incredible scope that you guys have reached. How did you come to start Databricks?

• 2:05Speaker 01

  Yeah, for sure. So it basically started out of some of my research as a PhD student, as you mentioned. So I was doing my PhD at UC Berkeley, and I was really interested in, you know, large-scale data-intensive computing, which was just starting to become a thing back then with web companies like Google and later Facebook, you know, Yahoo back then, all these companies that were indexing the whole web and like building search engines and other products. And, you know, I was excited about bringing this kind of technology to, you know, to all users, like storing, collecting data, storing it isn't that expensive. So why can't everyone do, you know, really large-scale computations?

• 2:49Speaker 01

  And indeed, there was this trend that in most areas of industry, most areas of science, you could start collecting lots of types of data. And then I was also excited about letting people on more sophisticated algorithms on them, because that's what I saw at the early Mapreduce users. You don't want to just do analytics, especially when you have so much data. It's perfect for machine learning. It's perfect for unsupervised learning, you know, when deep learning became popular and so on. So, yeah, so we started first this open source project that very quickly grew. It was clear that there's demand for something. And so, and we decided that, you know, we should start a company to like really, you know, have an impact in this space, because you can only get so far by just having an

• 3:30Speaker 01

  open source project with no one backing it. And the other bet we made that was a bit controversial at the time, that turned out to be right, is we decided to go cloud only. So like you can only use Databricks in the cloud on one of the three major cloud providers as a SaaS service on top of that. And this was 2013, so it wasn't as obvious that, you know, banks and, you know, other really big enterprises would move to the cloud. But it turned out to be the right bet, and it like really focused the company on delivering, you know, excellent experience there with things you can only do in the cloud. And it helped us, you know, land as one of the big options if you, if you want to do these things in there.

• 4:14Speaker 00

  Yeah, amazing. It's super cool just to see how how far it's come. I remember when you were just getting started, I actually was on the machine learning team at LinkedIn. And we were some of the early adopters of Apache Spark and I think you and Reynolds and you guys came in did a talk. And it's just, it's just amazing the technology that you've built and then how how far it's come. And I remember a big part of it was kind of for these sophisticated algorithms, you're moving to training things, you know, to doing things in memory. So iterative computation would be a lot faster than for Hadoop. Was machine learning a target, kind of a target use case for Spark from day zero? Or was that something that, you know, over time, it started to really emerge as like this is great for training models and and that sort of thing?

• 4:56Speaker 01

  Yeah, no, it actually was one of the main use cases from the beginning. And actually, the first applications we built on it on Berkeley were machine learning ones. Because I was sitting in this lab that had, it had, you know, a lot of the systems researchers at Berkeley, like Dave Patterson, for example, but they had set up this lab with Mike Jordan, who's a huge, you know, machine learning researcher, doing, you know, both theoretical machine learning, applied ML in different areas and so on. And so they wanted to look at the intersection of computer systems and machine learning. Can we use ML to improve computer systems, but also can we use like these large scale emerging systems to to do machine learning better?

• 5:37Speaker 01

  So I had seen people using, you know, MapReduce and other things in industry. But at Berkeley, I wanted to find someone local that would actually try out, you know, my crazy new ideas for a programming model and like a system that, you know, still has a few bugs and stuff. And so the people around were the machine learning ones and they were, they were very excited to try it. So for example, one of my classmates in in the year I started was Lester Mackey. He went on to basically to get a faculty job at Stanford and then he went to Microsoft Research. So he, you know, he became like a full time, you know, AI researcher. And at the time, he was part of a team entering the Netflix Prize competition to build a recommendation engine.

• 6:20Speaker 01

  And he told us, you know, we really need scale. We need to be able to run. We have all these ideas, all these training ones we have to do. We only have this many days before the competition, so we need scale. So he actually tried it out and was able to accelerate some things. And, you know, his team did really well in the in the contest. They placed second at the end. And there were also other groups. So yeah, that's something we were excited about.

• 6:45Jacob Effron?

  Yeah, I mean, it's a good start to a tradition of, I guess, now the company's supporting many companies, building kind of cutting-edge ML models on top. And, you know, I guess I'm curious, you sit at, you know, Databricks, you sit at this kind of really interesting opportunity where you you get to observe, I guess, a ton of folks that are building interesting, you know, applications on top. And so I'm wondering, I guess, what you've kind of noticed trend-wise and how folks are deploying some of these ML models in production. And then obviously, you yourself have been very involved with MLflow and building some of the kind of supporting tooling here. And so would love just, one, kind of the trends you've seen in that. And then, two, I imagine with ChatGPT and the kind of rise and all this tension around LLMs that the last few months might have been pretty interesting as well. So just anything you're kind of seeing on the ground and how folks are using these.

• 7:25Speaker 01

  Yeah, definitely. So I'll call out like two things. So first of all, I think just ML in the enterprise, maybe maybe not necessarily deep learning. Increasingly, there's a lot of deep learning, but just, hey, ML and, you know, like day-to-day, like business processes is established now. So it used to be, when we started out, you know, there were many people who said, oh, would enterprises actually want to do machine learning? Do they even have the expertise? Like, is there anyone there who could help them? You know, I believe it could help them, but is there anyone there? And so on. Or will it be this thing for like only these weird web companies? But now all, you know, major companies have hired data science and machine learning teams,

• 8:06Speaker 01

  and they're building products. Their first one or two models are very likely in production. For some of them, it's already hundreds of models, and they have the team, they have the connection to the business data, and they're looking to build more. And these things provide tremendous value in a lot of areas. Like, for example, one customer that's talked a bunch about things they do with Databricks is Shell, right? They produce, you know, all these chemicals, they have all these like, you know, these these plants that can process them. Storing chemicals is expensive, you know, if you produce it and no one's going to use it, it's bad. All kinds of things can break down. If that happens, it's really bad for you. And so anything you can do that's that helps you estimate demand, optimize, you know,

• 8:49Speaker 01

  In the plant, monitor, detect anomalies, has many millions of dollars worth of impact. And it's so easy to just collect the fine-grained time series data from these things. So why not, why not do it, right? And so they're at the point where they're basically, you know, doing a lot of use cases through fully automated like auto ML. Where like the ML team only has to look at it when something's broken. And that's just one example, but it's like anything, like companies that produce milk, you know, companies that build houses, like whatever it is. Right. They're using it. So that's like one change, you know, you can now, if you do something in the space, you can find lots of enterprises who have ML teams and are doing things. And of course, the next wave of that that they're all asking is, okay, we did the first few use cases.

• 9:34Speaker 01

  It was a lot of work, but now we have ideas for like a hundred more use cases. So how do we make it way easier to create new ones and bring them to production? And I think whoever, you know, cracks that code is going to, you know, determine sort of the next platform. And that's, that's the kind of thing we're, of course, trying to do through open source projects like ML Flow and through the SaaS services we have. So that's one thing. And then the other thing, yeah, you mentioned chat G P D, L L M. So deep learning is definitely on people's minds. And I think, again, within enterprises, I think people are especially excited about natural language just because there is so much data with that. And there are entire products you can make using it.

• 10:14Speaker 01

  You know, I, I see like virtually all software companies now, for example, thinking of including it in their UI in some form. But I also think other things like computer vision will, will also be increasingly important. We're starting to see some things like that. And the really interesting thing with chat G P D is I think it just opened up, like it got more people to try these foundation models. Like basically G P D three had been there for years, could do these things in stark G P T, which is easier for people to prompt. Also had been there for a while, but there was no like nice web interface where you could just try it out. So I think now. A lot more developers are thinking about it and trying to build these applications, which is really cool.

• 10:57Jacob Effron?

  Right. Does the rise of like, you know, more developers trying to build on top of like large, you know, language models, does that change at all kind of how you think about, you know, the roadmap at MLflow or kind of the tools you guys build at Databricks? Or is it kind of, you know, yeah, how are you kind of thinking about, you know, of that first world that you talked about, the kind of specific models for a company like Shell versus, you know, the, the kind of, let's build a support product on top of a large language model, you know, how are you guys thinking about supporting enterprises across this?

• 11:23Speaker 01

  Yeah, it's a good question. So I think there are, so I think these, the LLMs out there, sometimes called foundation models, I'm part of, I'm actually part of the team at Stanford that like tried to, you know, come up with a, with a name.

• 11:37Jacob Effron?

  It's a very impactful term you guys coined. I feel like everyone uses it now.

• 11:40Speaker 01

  I mean, you know, there are pros and cons of this term. Some people don't like it because it sounds like you have to build on one of these. That wasn't really the intent, but the idea of this like pre-trained model that you can then adapt to use cases is, is very powerful. At the same time though, I'm seeing that you need, for, for every company and every product, like their application of it is different and to get the best quality, you will want to customize it. And so people are trying to figure out what to do. I mean, some people are getting predictions from these models and then actually like training their own model, you know, by fine tuning these or even by just training their own from scratch, like just using them to label stuff. Some people are looking at really clever ways to incorporate it in their product.

• 12:23Speaker 01

  For example, there's this nice post on GitHub Copilot, like how it decides what context to send with your prompt, so that it has the highest chance of knowing about your code and completing it. You know, some people are designing like user interfaces that actually can collect, you know, reasonable feedback and make things happen. So I think there's still a lot of engineering, but these models let you do some simple, like some basic capabilities of like understanding language and tense and fuzzy sort of semantics that you couldn't do before and they're also super easy to try out, which is great.

• 13:00Jacob Effron?

  Yeah, no, it's interesting. I mean, I think, you know, some of like the ML ops companies that are emerging around the foundation models themselves, you know, it feels like it's things like around prompt engineering or around providing more context in prompts or chaining models together. And one thing I'm curious about, and I'm sure you guys think about this, is, you know, it's hard to know how much of that is point in time for like GPT-3 today, like that's required, versus how much of this is always going to be a fixture of like working with these models? And so, you know, you guys must think about this in terms of what you build. Like, how do you think about, I guess, how much of these problems today are point in time for where foundation models are right now versus like permanent fixtures of kind of an ML op stack that you might need for them long term?

• 13:39Speaker 01

  Yeah, great question. Yeah, I think, yeah, it's important to separate things that are very algorithm or technique specific from general things you need. And for Databricks as a company, at least, we've focused on figuring out, you know, what are the platform pieces that are really general and will be useful later? So like, what do I mean by that? So for example, one thing is, whatever you do with your model, you're going to deploy it and you're going to want to collect back data on how it's doing, or maybe A/B test different versions and easily analyze that data. Could be lots of data because you want to log, you know, everything you can to improve it. So making that workflow super easy. So like, anytime you deploy the model, you just get a table that is, you know, here's all the stuff that it's

• 14:20Speaker 01

  served. Or you'll want to have different versions of it and compare them, or you'll want to have reproducibility of seeing what went into it. I think that will be important whether you're running, you know, like scikit-learn or PyTorch, or you're calling an API for the model with a prompt, you still need all these things. And they're not easy to do. It's not easy to figure out the right workflows. So there's a lot to do there. Of course, you can also do something more specialized for one type of model or one application, and you can have really great products there. I think, again, some of the most successful ones are ones that are pretty broad, like, for example, Things that let you look at computer vision more in general and like maybe, you know,

• 15:02Speaker 01

  design like easily build an object detector for a new kind of object or something like that. Those are, you know, likely to be widely used, but it remains to be seen. I should also say on this prompting aspect specifically, I mean, I think you, maybe you folks would have prompted me to talk about it later too, but yeah. Some of my research at Stanford is on this now, is on how to build complex applications using language models. So we have this programming model called demonstrate, search and predict, or DSP. We have some open source code and we're also looking at these things as a building block. And actually, the approach we've taken that's maybe a bit different from some companies is we wanna, you know, use language model for things it's uniquely good at, like approximate

• 15:47Speaker 01

  matching or things like that. But as much as possible, we wanna retain like programmatic control of what the system does. So we have something where like you basically write a pipeline of steps in Python and then some of the steps call into an L M. So you can also basically fine tune it and so on. But we're trying to limit its scope to task that will be basically really easy for it so that the overall accuracy as you combine many of these is still good. Because otherwise, if you bet on it doing one thing, you know, like, like say come up with a plan to answer this question and then like follow each step and it comes up with the wrong plan, then you know, you're toast. It's never gonna succeed. But so there'll be different philosophies.

• 16:29Speaker 01

  I'm sure there'll be people doing the opposite too, but we're looking at it from an engineering point of view of like, we want something that's, you know, 99% reliable, like, you know, what's the failure probability of each stage that we want to get it to work?

• 16:41Speaker 00

  Yeah, it's super cool and DSP is definitely one of the projects that that was on our radar and it seems like one that that people are really excited about. So we were gonna prompt you on it later.

• 16:52Speaker 01

  No pun intended.

• 16:53Speaker 00

  No pun intended. Yeah, it seems like a really interesting area of research in terms of how people are interfacing with L L M's and what work is done by the L L M versus in in code. It seems like with DSP, it's actually pulling work out of the L L M and into code. Is that the right way to think about it when people are kind of writing these Python scripts to orchestrate L L M calls?

• 17:14Speaker 01

  Yeah, more generally, as I said, we wanna have You know, some things you can call, well, like some steps where if you're pretty sure you want to do one thing or another, like you then use the model, you know, in a limited capacity, like say generate synonyms for this word or whatever, then I'll search for them or something like that, as opposed to like asking it a very complicated question and hoping that it figures everything out. So it's a sort of a challenge. But there are, you know, it remains to be seen. I mean, the other factor this is getting at that's maybe kind of interesting is right now, these large language models bake together and combine both what I'll call like semantic understanding or reasoning and facts, like knowledge about

• 17:58Speaker 01

  the world. So for example, like who's the president of the US? Well, if you ask ChatGPT, it, you know, I mean, today, maybe it gives the right answer, but like in four years, it'll be wrong, right? Actually, I don't remember what it is.

• 18:11Speaker 00

  Unclear.

• 18:11Speaker 01

  Maybe they got it for a long time. Maybe it's now good. But those things like you'd like to decouple them. So we also want to look for a way to separate these. And it's actually kind of hard if you train a model on tons of data to give it the semantic understanding. It's really hard to like then make it forget those facts. Like, you know, now, like just imagine a world where like suddenly, you know, this thing changed. Now can you answer my question? So.

• 18:39Speaker 00

  I think that's super interesting because it is, it is a big problem that then this knowledge would be encoded in the model weights and you'd have to, you know, not easy to update, right? Like, do you think in the future there will be more of a traditional search index that is encoding kind of knowledge and then something that is, you know, more generative, generating the response? Or like, how do you see kind of traditional information retrieval working with LLMs in the future?

• 19:04Speaker 01

  Yeah, I think that's an attractive architecture and, you know, we need like strong reasons not to do it. But I think at least in what I'd call knowledge intensive applications where there's some either like public knowledge or even private knowledge, like, you know, support tickets in my company or stuff like that, it makes sense and you want to be able to plug those in. And there's a question of how to best build these, but you can get, you know, very impressive, like, you know, demos and quality, even, even in like fairly easy ways of combining current tools.

• 19:35Speaker 00

  Very cool. The last thing I want to touch on this point is there also seems to be some interesting things where, you know, you have your DSP program, but then you also have a language model telling you how to orchestrate a DSP type program. Like, do you see, do you see those being written by a language model in the future? Or do you think that's kind of the right level of abstraction where it's like human takes over writing this program and then, you know, interfacing with the LLMs that way?

• 20:00Speaker 01

  You could have things produced by a language model. And I think one of the, the best kind of new use cases for these is, they are good at generating like long, you know, kind of complicated answers or examples or whatever for something. That's like, for example, when people ask like, hey, is ChatGPT like a Google replacement? I would say it's not so much a Google replacement in its current form as a Stack Overflow or like Quora or whatever type of replacement. Of course, and even the ChatGPT augmented by a search engine, I think the most interesting bits are the ones where you're asking it to generate a lot of stuff. So it is very powerful for teaching people programming.

• 20:42Speaker 01

  Actually, when we think about like in databases, you can imagine we're also, you know, building, you know, ways to use these language models like in our product. And one of the things we're most excited about is it will make the product easier to learn for people, right?

• 20:56Speaker 00

  Yeah.

• 20:56Speaker 01

  We are, we're all about like expanding access to data to like more people, you know, non-experts, like let them do cool things. And, you know, hosting things as a cloud service is one approach. But this kind of like a natural language that gives you something that's like maybe usable or you can edit from is, is another one. So I think there'll be an interesting back and forth. And the other thing I'll say about that is if you think about like, oh, okay, I'm gonna tell a language model to generate some code for me and then I hope the code is correct, right? Like it's Python or something, I'm just gonna run it. If you don't know how to program Python, like it's it's kind of hard to make that work. You know, it's it's good if you're if you're learning. But you could also try getting it to generate something else.

• 21:39Speaker 01

  Like, what if it generated, you know, a drag and drop like UI in a in a tool like Figma or something like that, right? Then you see it, you can edit it. You know, you don't need to be an expert to tell whether it's it's correct or wrong. What if it generated like one of these visual like data flow lines, right? Like generate a pipeline to like parse and transform these things. And then you can just like click on it and view it. So it doesn't have to generate like an inscrutable, you know, programming language to be useful.

• 22:08Jacob Effron?

  Totally. No, it's really interesting. You mentioned, you know, potentially using these models like within the Databricks product to make it more accessible to folks. Like, I I guess what are some examples of of how you're thinking about that? Or how you might actually end up incorporating these types of models?

• 22:19Speaker 01

  Yeah, there there are a lot of these. I mean, and and, you know, many, many companies are doing this. But certainly, like any place you could, like, for example, any place you can type a query, right, in in like Python or SQL, like, maybe you could type it in in natural language. That's an example. But there are also others, like what if I'm doing a search either over my data sets or over the documentation, right? Like, you know, why why not let me do that to a better model? Another simple one that many companies are doing is just basically auto fill or recommendation. So you know how when you, like, for example, when you share a document in in Google Drive and you you add one person, it it suggests, hey, maybe you also want to share with these

• 23:03Speaker 01

  other people. That's like really easy to do. So like some of the software companies we work with, even before ChatGPT, you know, they were doing machine learning on the platform. They wanted to do a lot more. And we asked them why, and they say, well, look, I have this application. It's got like a thousand text fields on it, and there are millions of users using this every day. I want every single text field in there to have autocomplete, like, based on that customer's deployment. So how how can I get that, right? It'll it'll make everyone like my application better and and all that. So. Yeah, so I think there are quite a few and I'm definitely hoping we get to a place where these like kind of quality of life things are very easy to add for any developer.

• 23:45Speaker 01

  Like, you know, imagine I'm adding a text field in like HTML, I could just put like, you know, learn equals like, you know, based on these other fields, basically. And then the ones where like you really want to spend time like hands-on, build like the best quality thing you can, you know, invest lots of engineering, you have nice tools where ML engineers can just focus on that. So, you know, like the, when you're building like your self-driving car, whatever, you're not going to do that with just like one-click AutoML. You want the best tools to keep cracking away at it and like make sure it's really like good 99.99% of the time, so.

• 24:20Speaker 00

  Awesome. Would love to shift gears and talk a little bit about kind of the latest in the LLM world and training those models. It seems like the transformer architecture is kind of becoming the de facto architecture that's used, you know, across a bunch of industries. And so then it's interesting because it seems like people are using the same architecture, but then it's about how you can actually scale these. And like you're a systems person and so you're like the expert on scale. I guess, how do you think about the importance of kind of modeling expertise versus systems expertise and how you scale, you know, the training and the serving and that sort of thing? And like, how do you think, you know, the importance of those might change over time?

• 25:00Speaker 01

  Yeah, it's a great question. I mean, they're both important. The really interesting thing about scale, and I think the reason a lot of folks went for it, is you don't have to maybe think that hard, I guess, to get gains, right? Like, you know, to put it, I mean, the systems people, building the system have to think. But if you, if you built like some model architecture, some training process like SGD or whatever that, you know, that seems to do well and you can like throw in 10x more machines at it or like 10x more data and it does even better, that's like fantastic. So I think a lot of people are going down this road and saying, hey, I can get, you know, X million dollars in investment, like. I am doing a lot of things that involve thinking hard and like, you know, like doing,

• 25:44Speaker 01

  you know, trying to understand from the fundamentals, but why don't I also do, like, why don't we just buy more GPUs and like, put in more data and like, try to do this thing? So I think a lot of folks have gone down that way. Now, of course, today with language models, it's at the point where very few companies can even like reproduce the largest scale ones and for researchers, that's certainly a problem. It actually means, I think, that the research will, or the innovation will kind of slow down because fewer people can, can be chipping away at it and like, you know, trying out new ideas. So maybe there'll be some new thing that's even, you know, that's easier for people to play with that leads to like a better sort of, you know, descendant ideas and better long-term research.

• 26:29Speaker 01

  But it also means that there are enough people working on making it cheaper, making it easy for anyone to do stuff at scale that, you know, there'll be activity on that. And if this is the method that works well, you know, for many applications, like it's still very inexpensive, like, you know, even if serving a prediction costs you like a dollar or something of GPU time, in some applications, it's like totally worth it. So people are doing it. What I found though is for, at least for very targeted applications where you want very high quality, like, you know, like say answering a support ticket or something, you don't want to be wrong, that becomes a whole, like, that becomes kind of a traditional ML engineering process where you have to think, do I have the right validation data? Do I have the right inputs?

• 27:13Speaker 01

  How do I check this version against this other version and so on? So the foundation models are only one piece of that and having the right tool and having the right like process, the ops process and the right tools to like see what's going on is also important.

• 27:29Speaker 00

  Yeah, it makes a lot of sense. On, on the cost side, it's fascinating what you were saying around the training versus, versus serving and it seems like the training costs have gone down a lot over, even just the last couple of years. I think there was one estimate that it was in, in 2020, someone estimated that on compute alone, it was $5 million to train GPT-3. And Mosaic just published a blog that you could now train a GPT quality model. So it's a little bit smaller, but quality model for less than 500K. So it's like a 10x reduction in, in two years. Do you think that trend will continue and we'll see another 10x reduction in the next two years or do you think that? You know, maybe there'll be some some sort of limits that we're hitting up in terms of

• 28:12Speaker 00

  because I bet there's some low hanging fruit that people were going after early.

• 28:15Speaker 01

  Yeah, I think there's still a lot of space to improve costs for the core model. So if you just want to take today's model architecture and like do it more cheaply, there is space. There are all these companies working on accelerators that will presumably generate a lot of competition there, especially if it ends up being a lot of transformers, right? It's just it's risky for some of them to go down that path because then if the architecture, you know, of choice changes, then then they're kind of stuck. But there will be some who do that. And there are also, you know, new ideas or new developments about the training process itself, like having bigger data sets, maybe having cleaner data sets, having, you know, different penalties.

• 28:58Speaker 01

  The the human feedback stuff, so it turns out like just a language model alone that just predicts the next word is kind of hard to use, but this instruction tuning thing makes it much easier to use, you know. There, there's a lot of room to do algorithmic things that might give you similar quality. And and then there are also pretty general techniques like model compression. Like, for example, with stable diffusion, they ended up compressing it to the point where you can run it on like a laptop or a phone or something, which was, you know, wasn't anticipated, I think, at the beginning. So, yeah, I think today's models will become way way more practical to use. The question I have is how much better do they get if if you put in way more parameters?

• 29:41Speaker 01

  And again, I think it also is tied to this knowledge versus semantic reasoning question, and if you could separate those two, then maybe you wouldn't need to put in more parameters. Because one interesting thing is in computer vision, we don't have like a GPT-3. We have, you know, excellent models, you can use them for pretty much anything you want, that are sort of moderate sized. And we we haven't had like, no one's adding like, you know, 100x more parameters to make it better. In language, we do have this, but How much of it is for the reasoning versus the knowledge stuff? Right, like you as a human, I can tell you, like, I can make up a story. I can make up like, you know, sci fi universe with my own like physics laws or my own like factions

• 30:25Speaker 01

  and tell you the rules about them and like you can then reason about them and tell them like, you know, like the rebels should definitely blow up the Death Star or like whatever. You know, you don't need to like spend lots of time like learning about it. So the question is, how much of that basic like reasoning capacity do you, have we maxed out with the current models and how much more do we, or how much are we gonna get? Yeah.

• 30:47Speaker 00

  Yeah, it's really cool. The stable diffusion running on the iPhone, I think Lama just came out or one of, one of those you can run on a single GPU and I can imagine there's just really interesting use cases, exactly what you were saying.

• 30:59Speaker 01

  Yeah, Lama is like a smaller transformer, but more care going into the training process. That makes the quality, at least on the standard like academic benchmarks, similar to GPT-3.

• 31:09Speaker 00

  Yeah, super cool.

• 31:11Jacob Effron?

  Yeah, I'm curious, given this discussion we're having about like how big these models need to get. You know, one thing I'm struck by of kind of momentum in the space right now is obviously you have the, the large foundation model companies, use your terms, that are, that are kind of closed source. And then you've got kind of the open source alternatives. And obviously you've spent a lot of time in the open source world. You know, curious how you kind of see this, this playing out in terms of the kind of extent of, of adoption of the open source solutions versus the closed source solutions.

• 31:35Speaker 01

  Yeah, it's a great question. I think there are a few factors here and again, it's, it's really hard to make predictions. So I can, I can give you like both the sort of the arguments for and against the, the closed source service approach. So one of the arguments for would be that it's just easier to use. So for, because it is a SaaS thing. So for example, like if I wanted to, you know, to train my own model to do something, I would have to first acquire some GPUs, not super easy even in the, in the cloud to get a lot of them. Or I can buy them and like. Wait, like, you know, I actually have like a bunch of like high-power GPU servers at Stanford, and I ordered a few more like a month ago, and you know, they're still not here yet, so I just gotta wait for them. So like, that's a pain.

• 32:19Speaker 01

  I then have to install software, you know, I have to call it. If I'm not using the GPUs or whatever, you know, it's a, it's an issue. So with, with something that's hosted as a service, you don't have any of this. I can just make a call and get a answer back. And then if they're working behind the scenes to like improve its price performance, or like upgrade to the latest version of CUDA, or even improve the model itself, the architecture, I just get those benefits. So like, that's one strong argument. That's like the SaaS versus, you know, packaged software type argument. And then the other one would be if they can learn a lot across customers and get this sort of unbeatable data advantage by being really good at that. But, and that I think, you know, kind of remains to be, to be seen,

• 33:04Speaker 01

  how much of that you need. Because the whole point of these, in, in sense, the ad like on the tin is like, you know, no, i- i- is basically like in-context learning. Like, I don't need to like see a lot of stuff. You can just apply me to your new thing. But then at the same, so that means, you know, you, maybe you don't need that much data. But those could be the reason. And I think there'll be some areas, and it could, could include language models, where like, you know, the best ones will be by companies that have that data mode. On the other hand, the, the counter-argument's like, one is, you know, the best models and, and the best training methods and stuff like that change very rapidly. And if your thing is closed, you don't get the benefit of that open, you know,

• 33:47Speaker 01

  innovation community that's happening, right? So you might invest super hard in like serving and, and hunting and, and training a transformer, and then like some grad student, or like undergrad somewhere these days, publishes a new thing that's better, and then everyone switches to that. And like, no one, maybe transformer was awesome too, but like, no one else is working on your architecture to, and you, you lose out. So that's like one of the risks, so. I think it will remain to be seen. And there is, I think, a tendency for more foundational things like public data sets. For example, all these things are trained in large part on public data sets. These things, there's a, there's a tendency for them to become like commons where, you

• 34:30Speaker 01

  know, no organization like wants to do that whole thing on their own. They decide, okay, we'll collaborate on that and we'll do something else. That's also what open source is, basically.

• 34:40Jacob Effron?

  Yeah, it's really interesting, you know, whether the underlying architectures stay the same, you know, as well as kind of, you know, alluding to the point you were mentioning earlier, like whether we can find ways to, you know, encode more reasoning into these models beyond just, you know, having to stop every bit of knowledge in there as well. It seems to have implications for whether you're going to need to throw billions of dollars at compute to kind of get a cutting edge model.

• 35:00Speaker 00

  We talked a little about the research that you were doing with DSP and information retrieval and bringing those together. It'd be awesome to hear more about Dawn and kind of what the charter is there and maybe some of the other projects that you and your team are working on in the research world.

• 35:14Speaker 01

  Yeah, so Dawn is this bigger lab at Stanford where a bunch of machine learning and systems folks got together. Actually, Dawn is, has basically just wrapped up. It was meant to be a three-year project, so it's kind of wrapped up now. Sorry, it was meant to be a five-year project. But we brought together like a bunch of faculty, you know, looking at, at different aspects of, and the goal was to look at democratizing machine learning and to create this environment where we can hear from people in industry. We had a bunch of like workshops and retreats and events like that with folks from industry. And we can hear from other people at Stanford who are applying ML. And we can also exchange ideas on different topics.

• 35:57Speaker 01

  So for example, two of the other professors in Dawn, Kunleer Lukatun and Chris Ray, are involved in Samba Nova, which is a hardware company. And then Chris is also involved, his lab started Snorkel, which is like a data, data-oriented ML company. And, you know, there are actually a whole bunch of like other startups or other connections with, with different people from that group. So it's, it was really nice for seeing these different perspectives. You don't often have people who work on hardware and people who work on parallelizing algorithms like my group and then people who like actually design, you know, their algorithms in the same room. I think to me, like, just trying to think of what's exciting.

• 36:41Speaker 01

  I mean, there are a number of, actually, at this point, like, pretty large companies that were founded out of Don, including Snorkel and Sam Benova. And I think there are also a whole bunch of, like, open source projects that have reached, like, some amount of use, or models that have reached use in industry. Actually, Colbert, which is the retrieval model we worked on, that is used in DSP, was one of those. That's like, we put out this model architecture, and now it's becoming one of the kind of standard things people consider for DNN-based information retrieval.

• 37:15Speaker 00

  Yeah, there's been some incredible companies, as you're mentioning, Snorkel and a bunch of other ones that have come out of Stanford. And I feel like that's, you know, kind of following the footsteps of the AMP Lab and what the folks at Berkeley did. Is there any rivalry?

• 37:28Speaker 01

  My Stanford colleagues would say Stanford was also doing that all along. But yeah.

• 37:32Speaker 00

  That's true.

• 37:33Speaker 01

  I like the big lab model there, yeah.

• 37:35Speaker 00

  I was curious about, is there a rivalry between Stanford and Cal for research? Like, you were at Cal and then you became a professor at Stanford. Are people pissed about that?

• 37:44Speaker 01

  I think people want their PhD students to go somewhere else and like to get, you know, to know the other folks, you know, like, and to get other ideas and so on. So like, they don't want sort of, you know, like, inbreeding of ideas in the same place. So I think it's okay. A lot of people collaborate across the two. But it's definitely interesting, because I think historically Stanford was seen as like the main place for startups out of academia. But now, I think Berkeley is, you know, I see more people at Berkeley honestly interested in starting companies than here. And partly is because there were a few large ones that started out of that, and like Databricks, and then there were alums that were really passionate about it.

• 38:29Speaker 01

  So like, last year, for example, like, a few of the Databricks co-founders and other faculty there ran a course on basically like, you know, doing a startup, a course for ex-PhD.

• 38:42Speaker 00

  Cool.

• 38:43Speaker 01

  You know, like graduate students. And I think there were like 100 people in the course or something, like, all thinking about it, which I have not seen here. There's no, there's no CS course here on doing a startup. You can go to other departments, but I don't think there's a CS course, so.

• 38:57Speaker 00

  Yeah, totally. One other fun thing I wanted to ask you about. Reynold Jin, when we were asking him for a different piece, he said that there was a time when you had done a ton of work helping a startup integrate Spark, this was in the early days of Spark, into their stack. And I guess after working on it for weeks, they gave you like a $50 Amazon gift card. What's the story there?

• 39:19Speaker 01

  Yeah, there were a few that I helped early on because, you know, I wanted, I was really excited to see people using the stuff we're building. And, you know, when you have a piece of software that's early on, there's also risk for them, right? Like, it might not work. They might have wasted weeks on something that doesn't work. Or, and, you know, it's out of a university. It might be buggy, like, you know, maybe I go and do a different thing next and like stop working on Spark. So there's some risks. So I was learning a lot by working with them. And there were a few of these. It was, it was really cool. You know, this is also like back then, I feel there were more in-person, you know, meetups and things like that in the Bay Area. I think now they're, they're still kind, they're starting to come back, but a lot of stuff is not exactly to the same level.

• 40:04Speaker 01

  So it was cool to have these like small communities and, you know, meet people there. We went, we gave talks about Spark at some of the big data and machine learning meetups. And then later, we actually started our own regular Spark meetup and like companies would ask to host it and stuff because they wanted to recruit people who know Spark or who work on it or whatever. So it was, it was fun. Yeah, but yeah, there were all kinds of things. I mean, I got, you know, I also got like free food and stuff. I didn't, I didn't really ask for much. It was just good to get people's time and to see the cool apps and honestly to, you know, to, you know, be able to talk about them to others.

• 40:44Speaker 00

  One thing I was struck by, you know, when you were talking about your Databricks founding story is obviously a big decision you guys made early on was deciding you were going to go cloud only. And I'm sure at the time, that felt like a big risk. Obviously, it paid enormous dividends. But how do you make that decision and kind of get comfortable deciding this is going to be the way forward?

• 41:04Speaker 01

  Yeah, I think there were a few things. So first of all, just the strategy, like even the short-term strategy of like you're starting a company. You're more likely to do well, I think, with this type of product. If you go after new workloads or like workloads that are being migrated than if you go and try to displace an established thing that's like tied into, you know, a million. Other processes in that enterprise. And so the cloud was good for that. In addition, the cloud allows you to have a very fast feedback loop and less divergence of like the software out there. So you don't have to support like 10 versions of the software because your old customers are refusing to upgrade or whatever, which was the issue for like packaged software. In the cloud, you just have the version that's deployed now.

• 41:49Speaker 01

  And maybe there's a second version that's like rolling out when you do an update. So you only really need like two versions at a time. So we thought, okay, fast feedback loop, which is needed to develop a great product. Plus this opportunity to go after, you know, green field applications or migrations where like someone else. Like we didn't convince that company to move to the cloud. You know, Amazon did or Microsoft did. But we say, look, if you're going to forklift move everything, why don't you try this thing here for this piece? Right? So we kind of piggyback on that. Yeah, long term though, and you know, we started with it and it was going well enough. And we were, you know, that we said, okay, we're just going to focus entirely on this instead of going to somewhere else.

• 42:31Speaker 01

  We could have also, in an alternate world, we could have used it to develop a really nice product and then sold that product somewhere else in another mode. But we decided not to do that. But yeah, in the long term, I think so much stuff is moving to, you know, to SaaS services and both for customers and for providers. Like they're a good arrangement because for customers, they just fundamentally have less stuff to worry about and to administer. And for providers, you get the very fast feedback. So it's going to be a major force. Like I think the big competitor, if you are launching a new, you know, software product that is not a SaaS thing, like say it's just open source bits someone downloads

• 43:15Speaker 01

  or, you know, thing you install a Docker container or something like the big competitor is going to be a managed service that does a similar thing. So.

• 43:22Jacob Effron?

  Well, we always like to end our interview with a quick fire round. So we ask you a few quick questions, get your thoughts. And maybe to start would just be curious. Obviously, there's a lot of things that are talked about on AI, Twitter, in the broader ecosystem. What's one thing you think is overhyped and one thing that's underhyped?

• 43:39Speaker 01

  Hmm. Yeah, I don't know if I want to say anything is overhyped. I mean, I do think. I would say, okay, if I am to say one thing, I would say like sort of demos that work like once, but then the thing, you know, doesn't reliably happen are risky in some, you know, depending what you're trying to do, right? So like it's in many areas, it's like easy to make something that works, you know, 50% of the time, 60, 70, but it's like really hard to close that gap. So you just have to be careful about it. But on the other hand, the demos show that there's potential that if you then do the additional engineering and stuff, you might get it to 100%. But it's, you know, I've heard people asking whether

• 44:22Speaker 01

  like different, you know, technologies have what they call the self-driving car problem, where like it's easy to make a demo, but it's hard to make the car like actually work. It's still hard to this day. So you gotta worry about that. And underhyped, I think, like data and incorporating knowledge or incorporating even like real time, like interactions between your model and something else, it's definitely becoming more relevant with things like the search engines that use LLMs and stuff. But like really thinking about how, what you feed in to get a great result, I think, is there's a lot left to do there.

• 44:58Jacob Effron?

  Awesome. All right. Next fire round question. What's one thing you wish you knew when starting Databricks that you know now?

• 45:05Speaker 01

  Oh, and there are many, many different things. Yeah. Yeah. I think there are just so many areas of business that like I didn't know about that, you know, we learn about, like including, you know, sales, marketing, like all these, you know, product management, all these things. So, you know, I mean, the other thing is just, you know, don't panic, basically, like you'll, you know, there'll be things along the way, but that are challenging. But if you have a great team and, you know, a solid sort of like strategic position, you'll, you'll, you'll do well. So.

• 45:34Jacob Effron?

  Awesome. Well, I mean, it just a fascinating, wide-ranging conversation. I feel like there are so many different threads we could pull on, but I know you're a busy person. So we'll, we'll leave you back to your day to day. I guess, you know, I'm sure folks want to learn more about, you know, Databricks, the work you're doing at Stanford. What's kind of the best way for them to, to dig in further?

• 45:51Speaker 01

  Yeah, I think if you just follow me on Twitter or LinkedIn, you know, something like that, you'll see a lot of the things I think are cool.

• 45:59Speaker 00

  Awesome. Momete, thank you so, so much for joining us. This is great. Super interesting.

• 46:04Speaker 01

  Thanks so much.