Databricks Founder Ion Stoica: Turning Academic Open Source into Startup Success

Berkeley professor Ion Stoica, co-founder of Databricks and Anyscale, transformed the open source projects Spark and Ray into successful AI infrastructure companies. He talks about what mattered most for Databricks' success -- the focus on making Spark win and making Databricks the best place to run Spark. He highlights the importance of striking key partnerships -- the Microsoft partnership in particular that accelerated Databricks' growth and contributed to Spark's dominance among data scientists and AI engineers. He also shares his perspective on finding new problems to work on, which holds lessons for aspiring founders and builders: 1) building systems in new areas that, if widely adopted, put you in the best position to understand the new problem space, and 2) focusing on a problem that is more important tomorrow than today. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode: Spark: The open source platform for data engineering that Databricks was originally based on. Ray: Open source framework to manage, executes and optimizes compute needs across AI workloads, now productized through Anyscale MosaicML: Generative AI startups founded by Naveen Rao that Databricks acquired in 2023. Unity Catalog: Data and AI governance solution from Databricks. CIB Berkeley: Multi-strategy hedge fund at UC Berkeley that commercializes research in the UC system. Hadoop: A long-time leading platform for large scale distributed computing. VLLM and Chatbot Arena: Two of Ion’s students’ projects that he wanted to highlight.

• 0:00Speaker 00

  In general, this was our approach. We are going to be aggressive also about partnerships, even though the partners could compete and overlap. Because you have to trust yourself that at least when it comes to Spark, you can build the best products. You're, you know, kind of saying internally, well, you know, if someone else is building a better product for Spark, then we deserve to lose, right? So that's kind of always was the confidence that we can build the best product for Spark. And eventually, if Spark wins, we are going to win.

• 0:52Speaker 01

  Hi, everyone. Welcome to Training Data. Today, we're excited to welcome Jan Stoica, professor of computer science at UC Berkeley and co-founder of both Databricks and AnyScale. He has a uniquely exceptional career as a leading professor and founder of companies at truly legendary scale. Today, we dig into questions like Databricks positioning in AI, how research projects like Spark and Ray have led to the founding of Databricks and AnyScale, how he ties his research projects closely to industry from day one, new projects out of his lab like VLLM, MemGPT, LMSys and Vicuna, and what research fields he's thinking about next.

• 1:31Speaker 01

  Jan, thank you so much for joining us today. We're really excited to have you on the pod. To kick things off, we'd love to hear a little bit about where Databricks aspires to fit into the overall ecosystem, especially with some of the recent launches. What are you personally most excited about?

• 1:45Speaker 00

  First, thanks for having me here. So I think with Databricks always, we wanted to provide the platform, an internal platform, which helps our customers get most of the value out of their data.

• 2:08Speaker 00

  And one of the best ways today to get value out of the data, it's using this kind of new developments in AI, including large language model and everything else. And the one thing is that to note that this was always our vision from day one. Actually, Spark was created, one of the main reasons to create it, when built it, it was to solve,

• 2:38Speaker 00

  to speed up classical machine learning algorithms, right? And to scale them up.

• 2:44Speaker 02

  Yeah.

• 2:44Speaker 00

  Right? And so in some sense, for us, it's full circle. We started this AI use machine learning. It's not classic machine learning. And right now, we are back and doing more and more AI because to take advantage and to create value out of the data.

• 3:05Speaker 02

  So you mentioned that, you know, you founded around, you know, enabling classical machine learning. How do you see the current AI moment as same? And how do you see it as different? And I'm curious, what are the specific things you're doing for this moment in time? Like the Mosaic acquisition, things like that?

• 3:21Speaker 00

  Yeah. So certainly, the momentum around AI, it's on a different level today, right? You just look at the investments out there, right?

• 3:33Speaker 00

  That should tell the, you know,

• 3:36Speaker 00

  a big part of the story. So I think that obviously is like, what the way we are looking at is that taking advantage and being successful is AI is not easy, right? It's like the AI ecosystem is growing in complexity. It's not just a simple call to a model, right? You need, you have so many techniques now like rag and raft and, right? To improve the accuracy of your application using AI, right? Obviously, everyone is excited about AI because it solves so many.

• 4:18Speaker 00

  Problems and there are so many, you know, generate so many headlines. But but still, it's it's not what you want. You want a product of AI. A lot of things still we are seeing today are fantastic demos, right? And demos are inspirational. And when people see demos, what, you know, whatever, chat GPT is all this kind of problems, Olympic math problem. It's very easy to think that, wow, it doing that is going to do everything, right? So but going from demo to production, it's a, like I said, it's a big step. A demo means that you need to find an instance, at least an instance to be really impressive, right?

• 5:00Speaker 00

  In production, so it's like, is there, is there exists such instance, right? But when you go from the demo to the product, you know, when you have a product, it has to work for all cases, right? So that's kind of the big gap. So that's why where the effort is to improve the accuracy, improve reliability, of course, eliminate hallucinations as much as you can. And really also to find where it provides the best value, you know, for, because, you know, you can apply AI to 1,000 use cases, but which are the use cases which are going to provide you the biggest value? I think that's what we are also trying to help our customers with.

• 5:41Speaker 00

  So to navigate that, how to successfully apply AI to their product, to theirs, to improve their services, to their business.

• 5:50Speaker 01

  In that vein, one of the most interesting things that I thought came with the Databricks AI launch was the new open general purpose LLM created by Databricks, D-Bricks. What was the reasoning behind training your own models, open sourcing them? And what do you think are some of the best use cases for that model?

• 6:07Speaker 00

  Yeah. So I think the main, if you look like our main market, it's enterprise and enterprise customers. They do have a lot of concerns about data privacy, confidentiality. And obviously, the kind of one control is not only that, the auditability, right? You want to be able to audit what data you used, what results have been, have been used, and what decision have been made with what data, the decision have been made. And.

• 6:40Speaker 00

  So that's why, in general, the enterprises, everything being equal, migrate towards the open source models. We can host, you know, on their machines or in their VPC and so forth. So it's one of the reasons for releasing the D-Bricks is to help our customers. And then the customers, you know, in many of these cases, they start from this model and then fine tune with their own data and to optimize for their particular use case, right? So, and it's again, so our enterprise customers, they want, you know, open source models.

• 7:25Speaker 00

  They want kind of control as much visibility as they can. They want privacy and confidentiality, especially in the, you know, in the recent light of, you know, of the breaches, which are widely published. The other thing is that, you know, the DNA of Databricks was, has been open source. It's not only Spark, but Delta and MLflow and many others.

• 7:52Speaker 01

  Yeah. The other thing I thought was fascinating about D-Bricks is its excellent programming abilities. What do you think has made it such a capable code model, especially even compared to something like CodeLlama 70B?

• 8:02Speaker 00

  Look, I think it's about, obviously, it's about data and how you train it. And it's one thing, I think, you know, you know, one of the major Databricks advantages, Mosaic, right? We have also the entire infrastructure for training. So we have not only about the data, but the entire infrastructure of training and fine tuning. And that is make it much easier and most cost effective to optimize models for different use cases. And obviously, the copilot programmability, it's one of the very important use cases in this enterprises,

• 8:47Speaker 00

  because software engineers are still very expensive.

• 8:49Speaker 01

  Yes, interesting.

• 8:50Speaker 00

  You know, so making the people you have, and it's not, it's not only about that. It's like hiring top software engineering is very difficult, right? If you are a large company like maybe, I don't know. 4G or something like that. And so making those people productive, it's extremely important and critical for their business.

• 9:14Speaker 02

  You mentioned Mosaic as a key part of the strategy. I guess, what do you think are your most important chess pieces in this kind of AI battleground? I imagine Mosaic is one of them.

• 9:26Speaker 02

  And are most of your enterprise customers looking to train their own models?

• 9:32Speaker 02

  How does Mosaic and your other acquisitions fit in with your customer needs?

• 9:36Speaker 00

  Yeah. So I think there are, it's like,

• 9:41Speaker 00

  there are a few enterprises who would like, so it's again, it's pre-training and then it's fine-tuning and using the model in your own, on your own hardware, in your own VPC, on your own machines you rented your own. So, as you may expect, there are a few which are doing pre-training, but there are still a few doing, want to be training on their data, have enough data. A lot of them, they want to do fine-tuning, right? Because, look, if you are an enterprise, right, and a company, right, and you want to improve your business, right? What do you have? What do you have and which others don't?

• 10:24Speaker 00

  The thing you have which others don't is the data. The data about your business, about your users, right? Right? So, therefore, because this is something you have and others do not, you want to take advantage of that, right? So how do you take advantage of that? Again, there are many ways and you try different ways to do it, right? One is about fine-tuning user data and you have an open source model. You fine-tune on that, right? The other one to use RAC, right? And things like that. And, but, and so you are going to do any of those. But then you want to do, like I said, in your own VPC, right?

• 11:08Speaker 00

  To preserve security, to have, you know, ten four security perimeter, you want to do it in order to protect confidentiality of your users. And obviously, right now, with GDPR, California Consumer Privacy Act and so forth, there are a lot of regulation. And the number of regulation is going to increase. So the fact that you can, having an open source models, you can fine tune, you can use your track in your own VPC. It's very compelling value proposition.

• 11:43Speaker 02

  And are you finding that most of your customers want to go that approach versus, you know, OpenAI and the very powerful closed source models on the market?

• 11:52Speaker 00

  I think enterprises, and still we are still early on. And there are many enterprises, obviously, using OpenAI for different use cases, different applications. And I'm sure OpenAI and Microsoft Azure will come with new products to provide better confidentiality, better, better, uh, security.

• 12:20Speaker 00

  But at the end of the day, what I was saying is that everything being equal, right? As an enterprise, I will prefer more control. Security and strategic, right? It's like, it's, it's, it's less kind of locking or something like that, right? So, um, so I think that's what, what, what, what we are saying. So if, as, as, as open source models are going to catch up with the proprietary models, in particular, in the use cases which matter for the enterprises, it doesn't need to be, you know, perfect, right? For all the particular use cases, it just need to be very competitive where it matters.

• 13:04Speaker 00

  Then, you know, enterprises will prefer these solutions on which they have, you know, more control and more secure.

• 13:15Speaker 02

  And how far away do you think we are from that moment? Like, do you think we're there today? Where all else is equal? Or when do you think we cross that threshold?

• 13:22Speaker 00

  In terms of the open source versus proprietary?

• 13:24Speaker 02

  Yeah, open source being, you know, on par, all else being equal for the core use cases.

• 13:31Speaker 00

  So you have a lot of use cases, you know, it's again, it's the open source plus the data. And right now, the application are more complex. It's not just a call to a large language model. You have this kind of, you know, it's very cold, you know, you compose, you have, you know, application which build from many components. And that's what we call compound AI. And so actually, it turns out that, you know, if you can, you know, you can build an applications for doing, you know, recommendation or things like that,

• 14:13Speaker 00

  or, or, you know, for programming for a particular compiler, for a particular tool, you can actually do better than even something like OpenAI on the latest ChatGPT, because you have more data. And the other things about, about, you know, data bricks, which I think there are also announcement about that, is that with a kind of Unity Catalog and things like that, you have access and you know also about the structure of the data, which helps you tremendously to improve the accuracy of your applications.

• 14:57Speaker 00

  But I say it's not only the model, right? It's everything else you have around it. And the quality of the data, you feed it in. So I think that's, it's, it's, is the data stupid, like you'd say, right? Like, at the end of the day.

• 15:15Speaker 01

  It sounds like control and security are two primary areas of, of things that enterprises really care about that you've noticed. And, and data bricks obviously has a tremendous advantage with having access to data as well to help these companies use models.

• 15:29Speaker 00

  Control and security for the customers.

• 15:32Speaker 01

  Exactly. And how, what about some other factors have you noticed that they also care about? Is, how much does cost matter? How much does diversity of other models matter?

• 15:42Speaker 00

  I think obviously cost is important. And.

• 15:48Speaker 00

  It's like this, right? First of all, you know, you want to, you know, it's the initially, the important thing is about the value.

• 15:56Speaker 02

  Yeah.

• 15:57Speaker 00

  Right? Can you provide the value? Right? It's like, so that's the first thing. And at that stage, cost is not as important.

• 16:03Speaker 02

  Yeah.

• 16:03Speaker 00

  Right? And and in here, actually, in some of these early stages, where people also try to use the most powerful model like OpenAI and things like that. But once you cross that and you have a use case, which you conclude that is good, add value to your business. Now you want to scale it up, right? And now you are talking about having more control and more security and all of these things, protect confidentiality of the data, privacy of your users and so forth.

• 16:37Speaker 02

  Yeah.

• 16:37Speaker 00

  And now, basically, people consider how they are going to deploy it. And this is where having multiple choices and having, like you said, more control, security, it matters. And where open source models and platforms like Databricks are very valuable.

• 16:56Speaker 02

  Yeah.

• 16:56Speaker 00

  Right? And again, it's also all the other components which are together in Databricks, like I mentioned, like Unity Catalog and everything else to increase the value of your applications.

• 17:09Speaker 02

  Yeah, super interesting. I'd love to talk to you about compound AI systems. I think you guys probably coined or popularized the term. And it seems like that's a lot of what the industry is latching on to now. Maybe for our audience, can you explain what is a compound AI system? And you know, what are enterprises thinking about when they're building these?

• 17:27Speaker 00

  Yeah. So a compound AI system is, it's basically consists of multiple components, multiple calls to large language models or agents. And it's very much you can think about when you write a program, you have multiple components. You have different function procedures to do different things. And then you then put them together to. To create a program. The same is very similar here, right? You can use maybe one model to parse the data, to extract the data.

• 18:06Speaker 00

  You can use, and then you can use, for instance, depending of the prompt, you may use a model. You know, if the prompt is about math problems, right? It's like you can use one model. If the prompt is about, maybe about programming, you may use a different model, right? And then you may, you may use, for instance, about formatting the result, right? That's another one. You may use that, for instance, now more and more you are talking about agents. And with agents, you have to, you know, you call external services or functions, like search, or you can use a calculator and things like that.

• 18:51Speaker 00

  So now you have different models. You can do a better job, and they are small models, actually, to take the prompt and convert it to a function call, right? So that's kind of what it is. But the way to think about it is like, the conceptually is like you write a program has different components and make it easier to develop and deploy and manage the same thing you want to apply to AI applications.

• 19:19Speaker 01

  So like a collection of smaller components that work together, where the, you know, the, some of the parts is greater than than the one monolith.

• 19:29Speaker 00

  Yeah.

• 19:30Speaker 01

  I'd love to dive a little bit into the Databricks story quickly, which I think is an incredible legendary journey over the last decade, but with a lot of nuances that I think maybe many folks don't yet understand. From today, it looks like you were building the right company, the right, at the right place, at the right time. But the actual nuance is that Databricks was originally started, originally for data scientists. It happened to cater well to machine learning workloads because of all the work that you were doing in data. And over time, you made the right strategic decisions to actually really grow with the AI market. Can you share a little bit of some of the learnings and journey that got you to where Databricks was today?

• 20:11Speaker 00

  Yeah, I do think that, very happy to, I do think that.

• 20:17Speaker 00

  A lot of things, a lot of, it's also about being the right time, the right place, and things like that. It's being lucky, or I think that's all, all of this is true. And there are many things needs to go right to be successful. Some of the things you control, some of the things you do not control.

• 20:37Speaker 00

  When we started, indeed, we are focusing to build a product, a cloud product, hosted product for data scientists. Right, we have this kind of notebook and, and we have provided hosted Spark and, and, and we, we targeted data scientists. And we targeted data scientists. It was one of the reasons we targeted data scientists because, again, we all have, like I mentioned, Spark early on, also targeting machine learning application, machine learning workloads. And at that time, there are not many data scientists, it was like 2013.

• 21:20Speaker 02

  Yeah.

• 21:20Speaker 00

  However, when you look around, most of the universities already have data science programs, right? Degrees, you're starting to offer data science degrees and you're saying, okay, we are, it seems a good, you know, pass forward.

• 21:38Speaker 00

  A good market. And I remember you're looking at LinkedIn to see how many data scientists are there because our users, not initially, they're not as many.

• 21:48Speaker 02

  Thousands.

• 21:50Speaker 00

  Especially, you know, especially when you compare the data with database analysts, analysts and engineers and so forth. He started to build and I think it was, you know, a reasonable, you know, good product. And then, you know, we started to grow quickly in terms of the number of, you know, customers. Initially, we had the small customers. And uh but then, you know, after and and and still, you know, the interactive analysis, the data science has been for a long time. It's actually one of the biggest workloads, in particular in terms of revenue, because the interactive workloads are priced higher

• 22:31Speaker 00

  than batch workloads. But and I I remember that, you know, I sold to these companies to use, you know, the data bricks and they are aspirationally buying it to do data, you know, to the data science AI. And when, you know, after a few months, we look up to them, because we didn't go much, you know, earlier, because we saw that they are doing well, they they are usually just growing and everything seems good. So no, no reason to worry. So we went back to them and said, you know, to see what they are doing. Maybe we can write a blog post or whatever they are doing, right? It's like marketing and

• 23:14Speaker 00

  everything. The surprise is that very few are doing actually machine learning at that time. And we asked them what what happens, right? Or still. Well, it turns out that obviously, that in order to do machine learning, you need data, like we discussed early on. And they realized that for the particular applications they wanted, they don't have the the data they need. So they need to now to put maybe to add new logs, you know, to to collect new logs in their products and things like that. And also, they need to clean up the data, you know, curate the data and so forth. So now, we are lucky because Spark was very good also for data engineering, right? For data processing,

• 23:58Speaker 00

  right? It's a data processing tool at the end of the day. And so they are using using using Spark for for for data engineering. And and then obviously, we we we start focusing also on serving data engineers much more than before. And and, you know, that's how it started. And and then obviously, when later now.

• 24:26Speaker 00

  You know, got the, you know, data engineering, right? And still, you know, have all these data scientists exploring and starting to build models. And then it is a very natural extension to start to add more products for our user, for our customers, like I mentioned, to get more value out of their data. And that meant building machine learning models and using the open source models.

• 24:57Speaker 02

  Very interesting. I want to talk about the Databricks Microsoft partnership. I think that was the stuff of legends and I think still probably one of the only case studies of a successful, like truly transformative partnership. Maybe walk us through what that partnership was. Was it a bet the company moment at the time? Do you think Databricks would have become what it became if you hadn't struck that partnership? Maybe just talk about that.

• 25:23Speaker 00

  Look, obviously, the partnership with Microsoft was, was, was a great partnership for us. We are, the one thing I want to say that in, and this is most visible. We are very, from the day one, we are actually very focused on the partnerships. Our idea was always, you know, like, we know, we make Spark successful, you know, hopefully de facto standard for data processing. And then we make Databricks the best place to run Spark. So early on, actually, you know, we, we started, you know, a few months in the life of the company. We had this kind of partnership with Cloudera.

• 26:08Speaker 00

  And, and then we had with Hortonorx. And this partnership was mainly because you are, you know, to advance Spark, right? Because Spark, it was created in

• 26:22Speaker 00

  Hadoop ecosystem, right? And these are the Hadoop companies, right? Right? And so we had this partnership or data stack and so forth. So this was like in the first year or two. So we had all this partnership. Despite the fact that some of these companies, we knew that they could become our competitors, right? Because, you know, just making them is making and helping them

• 26:49Speaker 00

  to deploy, to manage and to sell Spark by services, right? It's like, so, so in some sense, you know, the Microsoft is like, it's, it's, it fits our kind of approach of trying to, you know, to strike as much, you know, the partnerships with other, being aggressive, doing partnership with other, you know, organizations in the ecosystem. Even though, again, in some cases, it was not a clear cut whether we are, you know, they are going to compete with us or not, right?

• 27:32Speaker 00

  So we're very, just growing the ecosystem and growing Spark, that was the priority. We even had at some point a partnership with Snowflake. So it was very, it requires a lot of heavy lifting, lifting. So we always look for partnership which has, which are meaningful, right? And I think great negotiation, you know, Ali and so forth did a fantastic job there. But at the end of the day, we also need to commit to it. It's like it, it took to do, to build the Azure Data Bricks. So we were in IWS before that.

• 28:17Speaker 00

  It took, you know, tens of engineer, one year. And you are a small company at that point. So it was a huge commitment and a huge bet also from our perspective to do that. And yeah, and we, you know, I think engineering and everyone executed very well. And it was a successful product, right? And you know, Microsoft was great partners, you know? And yeah, this is what happens. We are obviously a bit lucky, but in general, this was our approach. We are going to be aggressive also about partnerships, even though the partners could compete and overlap.

• 29:01Speaker 00

  Because you have to trust yourself that at least when it comes to Spark, you can build the best products. We are, you know, kind of saying internally, well, you know, if someone else is building a better product for Spark than we, we deserve to lose, right? So that's kind of always was the confidence that we can build the best products for Spark. And eventually, if Spark wins, we are going to win.

• 29:27Speaker 02

  Do you think that Databricks would have become the company it is today without that Microsoft partnership?

• 29:32Speaker 00

  I think so. It maybe took, could have taken a little bit longer.

• 29:36Speaker 02

  Yeah.

• 29:37Speaker 00

  But yeah, I think so, right? We will have, still have a good, very good offering on Azure. Like we have with GCP, it have taken a bit longer. But I don't see any fundamental change in dynamics, right? Because one of the advantage of Databricks, of course, is when Spark won, and we could provide the best product for Spark, we are in a very strong position. And compared with other clouds, remember that one of our advantage, like everyone else's advantage, like Confluent and so forth, is that you can provide a service on multiple clouds.

• 30:20Speaker 00

  Right, and multiple clouds is, you know, multi-cloud has been, you know, more and more very strategic for, especially for large enterprises. We do not want necessarily to be locked in or to want the one choice. So, yeah.

• 30:38Speaker 01

  I love the confidence and conviction that you have in Spark and your own execution abilities internally, but that married with the practicality and aggressiveness of winning as a business and pursuing the right partnerships and doing whatever it takes to win.

• 30:53Speaker 00

  Yeah, yeah, I mean, you try to simplify things. That's what I was saying, you know, you like initially said, like, look, you know, like, we have to make Spark win. You know, there are many, I remember, you look at all these combinations. It's like, it's like Spark wins, the product fails, right? Or Spark loses, but we have a product which is successful. Or both fail, no, that's not very interesting. Or both are successful, right? And we convince ourselves that, you know, we need to bet on Spark to win because that's the most likely way also for the product to win. Again, for better or worse, right?

• 31:35Speaker 00

  But sometimes, you know, there could be many ways to success, right? And in retrospect, you cannot go back and, you know, and try other alternatives. Maybe there are better alternatives at that point, but it's important to commit to one thing, which, and which hopefully it's a reasonably good solution, a good path forward, right? Again, there are many, many passes to the peak, right? The most important is to commit to one, which leads to the peak, right? It's like, not the shortest one, not the easiest one, it may not be, but it has to be one to get there.

• 32:04Speaker 01

  And to the highest.

• 32:05Speaker 00

  And that's why I said, okay, Spark has to, has to win, right? And we have to. Build, you know, the best need to be the best place for Spark. And then we are saying, you know, to be the best place for data and AI. We need to, eventually we knew and we assume that if you are to be hugely successful, you are going to, you know, to go beyond Spark.

• 32:29Speaker 01

  Yeah.

• 32:29Speaker 00

  Right? It's like, that's why the name of the company, Data Bricks, is not Spark Labs or something like that.

• 32:33Speaker 01

  Yeah, yeah, interesting.

• 32:35Speaker 00

  So, so that's kind of, you try to simplify it. You know, once you, you do that and you start to execute. Okay, so you want to make Ray successful as an open source. So you want everyone to use it. So that's why you do Cloudera and try Hortonals and so forth to do, to, you know, to, to do this partnership. Because at that time, there are other solution, you know? People knew that Hadoop map reduce, you know, these times have passed, so to speak. So they are talking about new systems. There are like Tez, it's actually Hortonals, this project and so forth. So that was very important. And then, it was also for data science.

• 33:15Speaker 00

  It was kind of, when we bet, it's like, it's a niche and, you know, we, and we can, we saw that we can build the best product for it. So, you know, so that's, that's kind of, you need, and you need to have ultimately confidence in what you bet on, right? You have to bet, right? Right? Because you are a small company, right? If you don't bet, how you are going to win, right? Everyone, right? And then you need to have some level of conviction, right, to do it? And, and, yeah.

• 33:49Speaker 01

  I'd love to kind of pull on that thread and switch gears a little bit into tying a lot of the entrepreneurial path that you have with a lot of the academic and research background that, that were the roots for the beginnings of these companies. You have a very unique career as both a leading professor and a founder of multiple unicorn and decacorn companies. I don't think there's anyone who comes close to pursuing both disciplines at the scale of success that you have. Maybe specifically, you know, Ray with any scales, Ray to any scale, Spark to Databricks. Take us into your head, what, what is the process in which these research fields start to ruminate in your mind?

• 34:32Speaker 01

  When do you kind of continue to give them resources to develop? And then when do you know that it's the time to then start a company to pursue that in, in a better, more open and fast way?

• 34:46Speaker 00

  That's a, that's a good, but hard question. So I, I think that, and by the way, I like to profess, to, to, to profess, to, you know, to say that, to start with saying that.

• 35:01Speaker 00

  It's obviously also a lot of luck involved and, you know, being in a place at Berkeley, like Berkeley and having fantastic students and colleagues around you. It's, you couldn't, couldn't do without that, right? It's like, it's, it's, it's their merit probably more than mine. So, but one thing I think is that I've been always trying to focus.

• 35:25Speaker 00

  It's on the problem.

• 35:26Speaker 01

  Yeah.

• 35:26Speaker 00

  Right? And actually, when even to my students, I'm telling them is like, the most, one of the most important things you need to do is to figure out what problems you are going to work on.

• 35:39Speaker 01

  Yeah.

• 35:39Speaker 00

  Right? Because it's like, everyone which comes to Berkeley or these top schools, they are, one thing they have in common, they are good problem solvers. Right? They have good grades, you know, good scores, you know, write papers, paper is about solving a problem. So therefore, if all of them are good problem, problem solver, the differentiator is a problem you are working on. Right?

• 36:04Speaker 01

  Yes.

• 36:06Speaker 00

  So you, you, you start with that. And also, I think it's like, in the, especially at, at Berkeley, it's, you get.

• 36:19Speaker 00

  You get exposed of kind of not only new ideas, but willingness to take risk and get into new areas. In some sense, and this is what I like about Berkeley. If you look traditionally, Berkeley among top schools, they are the first to open new areas. It's like, of course, the risk processor, it was with Stanford as well. But databases, networking.

• 36:47Speaker 02

  Yeah.

• 36:47Speaker 00

  A sensor networks, even in open source with the Unix BSD, you know, TCP IP, right, part of the STI. So they are always kind of more, you know, a little bit trying to experiment. So I think that's kind of, you know, that's kind of culture. I really resonated with it. And then the other thing what happened is Berkeley, we have these labs, which are like five-year labs, which basically each lab has kind of a vision. And, you know, it's a group of faculty coming together, which believe in that vision and with their students, and try to make it happen over, you know, five, you know, five years.

• 37:30Speaker 00

  And this, you know, has a lot of great, you know, impact, you know. It's like all the way it started. This tradition started forty, fifty years ago, with David Patterson, Randy Katz and others. And they built, you know, risk rate redundant array of inexpensive disk. Now network of forestation, it's commodity. You know, this is, everyone is building now this huge cluster of commodity machines, servers, and, and again, and many more. So there are this kind of elements. And these labs are, it's a very strong relation with industry, right? Connections, they are funded. Since I came, when I came, actually, so as one change happened.

• 38:13Speaker 00

  Before, these labs are also supported by government, in particular DARPA. But it was a point in which that kind of. At least that particular DARPA funding dried.

• 38:26Speaker 01

  Yeah, yeah.

• 38:27Speaker 00

  So, kind of when I came to Berkeley, this kind of now you need to go and remember, you know, you know, getting and more money from industry. I remember from Google, first time we got, and it was unheard by then because you are asking for five hundred thousand per year, right, for four years. So, but you know, we got, and this is we got. So now you have also this kind of very tight connection with the industry. It's a very good environment to see the problems, right, to understand the problems. Right? And then you can, you can, you can, you can, you can see that. And you, you try to, to think about also about, obviously about trends, right?

• 39:12Speaker 00

  Because trends are important, right? You have to be aligned with the secular trends, right? You need to bet on the right trends because these are things you cannot change, right? Or it's very hard to change. So you are not aligned, it's, it's, it's, it's, it's not good.

• 39:26Speaker 01

  Yeah.

• 39:27Speaker 00

  And these trends, actually, there are multiple trends. And the multiple trends kind of open gaps between them. And these are kind of opportunities for problems. Like for instance, for, you know, big data. It was clear you have more and more data.

• 39:45Speaker 00

  And the amount of data people collecting were just growing, right? It was pretty clear, right? It's like Google has, has seen that years before and they built all these systems. But now everyone want to emulate that, right? That's why Hadoop was created, right? And then you see, you, you start to see again, you, you, you look at that and you are working in that area and we are, you know, we have all this Hadoop people coming to our retreats on this kind of labs and we are, you know, friends with them. And we started to see problems and you, and then you try to use them. And, and then you, you, you just like, for instance, there are two things which happened with Hadoop.

• 40:27Speaker 00

  One thing is about a group in, in, in our lab. Oh, by the way, and the other thing what happens in these labs, they are interdisciplinary.

• 40:35Speaker 01

  Yeah.

• 40:36Speaker 00

  Right? It was people, you know, we are in this kind of rat lab was, that it was people from machine learning. Systems, databases, networking, right? So there are these groups of Michael Jordan students, which wanted to compete to, um, to this Netflix challenge. That Netflix released some data, basically, and asked for people to provide recommendations, come up with recommendation, to build recommendation algorithm systems. So that, you know, to beat their own recommendation. So they come to us and, okay, it's a lot of data. How can we, what can we do about it, right? We want to, and, you know, tell them to do, yes, Hadoop. And, but Hadoop was very slow, right?

• 41:21Speaker 00

  Because, you know, for, and then, you know, Matt, they put together something quickly to, for solving this problem in which the data was kept in memory. The other thing I've seen, it's about, like, I have a previous company called Viva, and it's about this sort of analytics company. And kind of it was very slow and we tried to do it, you know, it's like you to, for Azure queries. And there is no way to do it. And again, the, keeping the data in memory was a solution. It's one solution. So that's kind of how we started. That's, that's, that's, that's, that's one thing, right? And you look at the trends, yeah, it's obvious, right? It's like on one hand, um, you, you know, have more and more data, growing faster than the more slow.

• 42:03Speaker 00

  So you need to have, it's not going to fit on one machine. And therefore you need to use multiple machines. And then the only other question is that are you going to have data set are going to fit, important data sets are going to fit in memory, right? It's like, that's kind of the first question. And they are, because people, when they, even for, when they are doing, for instance, Azure Query, you notice that when they look at the data from different cluster, Hadoop cluster, from Yahoo and Microsoft and, and others. And, and we notice that in a lot of cases, actually, when you do queries and you are doing analytics, they are doing not on the, very rare, on the huge, on the, all the data. You do, say, for the most recent data.

• 42:45Speaker 00

  You want to see what happens yesterday, what happens last week, something like that. So once you get that and, you know, you have a lot of cases in which the data is fitting in memory and memory is still growing quite quickly, um, at that time, you know, that's, you kind of, you are pretty, you know, you, you connect the dots.

• 43:02Speaker 02

  Yeah.

• 43:02Speaker 00

  Right? And then it's about. Solving the problem and and the other thing is happens and why why they are related. You know, I'm talking about academia and industry because I'm telling, you know, people, you know, some people, you know, there are people in academia pushback. You say, you know, it's a lot of engineering here. What this is not you should do in in academia. But one thing is what I've what I've always found it is very satisfying. If you build a system, right, in a new area and that system, it's used by other people. Then you can, you are in the in the best position to understand the new problems in that area.

• 43:44Speaker 00

  Because people are going to use your system in different ways, right? And then, you know, you you understand that, right? And going back, if you know the problem, you are also in a good position to solve it, right? So actually, it it directly helps you with your research, right? To be ahead, because otherwise, what is the choice? Well, you are finding the problems, right? Of course, there are problems, very good problems in theory, which are not solved by, you know, decades and so forth. But other things what people do, they go to Google and Microsoft and so forth, spend time and to understand what is the problem they have, right? Because to solve, but that's kind of a little bit unsatisfying, right? Because you go to someone to learn about their problems.

• 44:29Speaker 00

  But the question is, why don't people, the people solve those problems? Maybe they don't solve the problems because maybe, you know, they're not as important as they're given time. And maybe, you know, maybe for the right reasons, they are, you know, too much to to to in the future to to. But this is this is the thing, right? It's you have to focus about the problem on the problem and you have to focus on the trends. And the way they're connected, you want to solve a problem ideally, which is going to be more important tomorrow than today.

• 45:02Speaker 02

  What what are the problems that you're most excited about right now? Like Spark, Ray, like what what's going to be the next data breaks or any scale?

• 45:08Speaker 00

  So a few things, so I still think that it will going to be a lot of work. It's it's right now what happens is that.

• 45:19Speaker 00

  Need to rethink most of the software stack. Why? Because it's again, if going back to the trends is that the demands of this application, in particular AI application and so forth, growing much quicker than capabilities of a single processor, a single node, even if you considered accelerators, right? So on our hand, this happens. On the other hand, the infrastructure becomes much more complex. You need to run that application not only on one node, but on many nodes. It's distributed. But it's not only that, it's becoming very heterogeneous, right? Because in order to bridge a gap between the demand and capabilities of hardware, people build accelerators.

• 46:04Speaker 00

  Like that's why Nvidia is a trillion dollar company, right? And but now the infrastructure becomes even more complex, right? It's hetero- it's not only distributed, heterogeneous. When we started Spark, it was homogeneous. All the nodes are the same, some some storage, some CPUs, that sort. But right now, look at, look at the heterogeneity. You have Nvidia and you know, and you have many others. You have TPUs from Google. Every cloud, it's having their own chip, right? Now you have AMD and Intel say that it's everything, it's about AI, right? So that's kind of what happens.

• 46:46Speaker 00

  So now you have a huge gap between this application and this very complex infrastructure, just growing in complexity. And then it's not only about CPUs, the compute, it's about networking. You have infinite band and you have all of this, you know, RDMA and so forth, right? So huge heterogeneity. And the software stack has to abstract away that complexity for the developers. There is no way around. You want a single machine or you have this operating system to abstract away the complexity, right? That's why, that's what makes it easy, right?

• 47:20Speaker 02

  Yeah.

• 47:20Speaker 00

  To develop all this application. Now it's extremely hard. So something is going to happen there. I think the other one is about this building application. You're talking about Compound AI, we are talking about Compound AI and things like that. Right now, this.

• 47:36Speaker 00

  Application, a application in particular, large language models. Everyone is talking about large language models. You know, the application are like assistance for humans, right? Humans are in the loop, right? If you are thinking about customer support, if you are talking about Copilot, if you are thinking about Q and A question and answering, even summarization. You have a human helping a human to be much more productive, which is a fantastic application. But they cannot, they cannot be autonomous. They are not yet autonomous. And to, to, to go from having the human in the loop to being autonomous is a huge gap, right? Because with autonomy, to be, to have something autonomous,

• 48:19Speaker 00

  you need to have someone which, you know, it's, it's, it's running kind of, you know, it's, it's more deterministic, it's more reliable, right? It's like far more accurate, right? And, you know, you need to get there because if you don't get there, you know, you are still limited to having the system or the human is in the loop. And the human is a bottleneck, will become the bottleneck. It's, you know, it's, it's just certain number of people on, on the planet. And I think that there is about, a lot of work could be about how to make, I would say, at least some aspects of this large language, building the large language model application more like an engineering discipline, right?

• 49:05Speaker 00

  Where you can, you know, build much easier systems from smaller components.

• 49:10Speaker 02

  Okay, so the two next data bricks will be distributed compute across heterogeneous hardware and autonomous compound AI systems. Noted.

• 49:21Speaker 02

  I'd love to tell them another thread that you mentioned just now of kind of how funding constraints drive what you're working on. I'm curious what you think right now of, there's been a lot spoken about kind of the, almost the brain drain in AI right now because the universities don't, just don't have the funding that you could get if you went to work at one of the big research labs. How do you think about that? What do you think is ideal? Does it, does operating under constraints force creativity for you? Like, what do you make of all that?

• 49:51Speaker 00

  Yeah, that's, that's a great question. And.

• 49:57Speaker 00

  It's true. I mean, it's challenging. It's very challenging. And when I came to the United States, I came to do my PhD and I graduated from Carnegie Mellon University. I am originally from Romania. So one thing was admired, people admired about the United States is like, is that how well this kind of, this, you know, the collaboration and the partnership, the three-way partnership between academia and government and industry are working, right? And prime examples at that time was obviously the internet, right?

• 50:39Speaker 00

  Which was DARPA project and, and of course academia had a huge impact. And also industry was, you know, was whatever, third industrial revolution, people were saying.

• 50:56Speaker 00

  And that, when it comes to AI today, that kind of partnership is broken. It's like, industries, every company is doing this research in silos, right? Even, you know, they don't talk to each other as much.

• 51:13Speaker 00

  Academia, like you said, doesn't have resources. And the government doesn't invest as much, right? So I think that's something to be very concerned about, right? And that's why I am also a big proponent of open source models. And US and California kind of will lose in long term if this doesn't, is not fixed. So what happens now in, unfortunately, in academia, one thing that happens is that, of course, there are some, you know, bigger universities and labs which can still afford to maybe train. Spend one, two millions, maybe to train some models. It's still they are not in the same league like.

• 51:56Speaker 01

  Yeah.

• 51:57Speaker 00

  OpenAI, right? It's like OpenAI and Microsoft, they are talking about building the data centers about one hundred billion dollars, right?

• 52:06Speaker 00

  And I think there is, the danger is that you are going to, some of the academics are going to give up and try to innovate around the ages. Now, you can still innovate in this application and things like that and, you know, I think there's a lot of innovations there.

• 52:27Speaker 00

  But clearly, you know, you know, it's, it will be harder to come up with new model architectures and, you know,

• 52:38Speaker 00

  innovate.

• 52:41Speaker 00

  Have, it will be harder. It's not impossible. Nothing is impossible. So, yes, it's, it's, it's, it's, it's a challenge right now. I mean, there are people like, you know, and which are better, fortunate, more fortunate position, but maybe I'm one of them, and which have access to resources outside academia, right? But, you know, it's, you do want to level the playing field in order to

• 53:14Speaker 00

  maximize the innovation. The innovation comes from everywhere. Now, this being said, it's true that, you know, scarcity and always in the past scarcity, you know, spurs innovation, right, as well.

• 53:27Speaker 00

  But the concerns about

• 53:33Speaker 00

  not having access to resources to

• 53:40Speaker 00

  to play the same game like in industry, it's, it's, it's a concept.

• 53:44Speaker 01

  I'd love to switch gears into some rapid fire questions. If you're ready for it.

• 53:49Speaker 00

  Sure, go ahead.

• 53:50Speaker 01

  Will anyone take meaningful market share from Nvidia over the next five years?

• 53:55Speaker 00

  I think they, they, it will be, they will be at the minimum because they are going to, it's probably that it's.

• 54:06Speaker 00

  Because Nvidia would not like to be accused about monopolistic behavior. So their market share has to decrease under some percentage, whatever, seventy, eighty percent. So that one will be one of the reasons. But I think that if I have to name one company to...

• 54:28Speaker 00

  Of course, there are clouds and for strategic reasons, they are going to push their agenda to build their chips, like still probably the biggest competitor right now in terms of the market share, it's Google with TPOs.

• 54:41Speaker 02

  Yeah.

• 54:43Speaker 00

  Probably that will continue for a while.

• 54:44Speaker 02

  Yeah. What's one project or student in your lab right now that you'd want to highlight?

• 54:51Speaker 00

  You know, I'm going to cheat here because I think that both VLLM and Chatbot Arena has been tremendous. It's like, I'm not talking about Skype Pilot because Skype Pilot is like, they started a company, so I'm not talking about. But I think VLLM has been amazing. It's like, it's one year old project and I haven't, I've never seen such a rapid growth. And of course, it's also part of the AI. AI is kind of compresses the time. So it's, there is something about that. And I think the other one is Chatbot Arena because it just is fascinating to see the development in the space and to see how these different models, where they are strong, where they are weaker.

• 55:36Speaker 00

  And I think that having a front seat at that, to see that kind of development in the ecosystem and in the space, it's fascinating.

• 55:48Speaker 02

  Do you think the foundation models will commoditize?

• 55:50Speaker 00

  The foundation model was a rare...

• 55:53Speaker 02

  Like a GPT-4 or a Claude? Do you think, do you think there's a market to be made in, you know, providing these models over time or do you think it commoditizes?

• 56:02Speaker 00

  I think people will continue to build larger and larger models. I think when it comes to serving, it looks like...

• 56:11Speaker 00

  Model distillation works quite well. By the way, the model is where you train a smaller model on the outputs from the bigger model.

• 56:21Speaker 00

  There has been a lot of success with that. By the way, this in some way, it's, it says that it shows how important is the data, right? For training a model, right? Going back early on in our conversation, right? Because you have higher quality data from the, from the big model and use that to train the small model and it's working very well. So I think using multiple distillation models to reduce the cost of inference is going to be a way forward. But yes, I think for advancing and pushing the frontier, so to speak, right? Pun intended, you are still going to see a lot of, you know, a lot of effort on bigger, on bigger models.

• 57:02Speaker 01

  What are you most excited to see in the world of AI in one, five and 10 years?

• 57:06Speaker 00

  What I'm most excited about AI?

• 57:08Speaker 01

  Yeah.

• 57:09Speaker 00

  Look, there is no question is transformational, right? I think that, and it's, it will change a lot of, a lot of things. Everything maybe. I think the most excited I am about it, it's about how do you make this AI more C systems, more predictable, accurate, verifiable? How you can debug these systems? This, all of this is kind of in realm of software engineering like techniques. This is what I think is exciting.

• 57:50Speaker 01

  What advice do you have for founders building in AI?

• 57:54Speaker 00

  It's, it's the same thing, you know, focus on the problem.

• 57:56Speaker 01

  Yeah.

• 57:56Speaker 00

  Don't focus on the hype. So hype is emotional.

• 57:59Speaker 01

  Yeah.

• 58:00Speaker 00

  It's not reliable. Just look at the facts, right? It's like, and look at the problem, try to understand the problem and try to be truthful to yourself. It's, it's, and it's about, if you build an application, it's about production.

• 58:15Speaker 01

  Yeah.

• 58:16Speaker 00

  It's not about the demo.

• 58:17Speaker 01

  Yeah.

• 58:17Speaker 00

  Right? It's beyond the demo. Of course, the demo are important, don't get me wrong, are very important. But the production is the way, the mindset you have to, to have. And yeah, and it's, it's a dangerous thing, is there's so much hype? And you think you can solve everything and you can do everything. Probably in certain number of years. But now just focus on exactly what problems you are going to solve. Convince yourself it's a good problem. Convince yourself that you can solve it. Or at least you have, you can, you can have an MVP, right? You can solve a smaller, a smaller version of that problem, which is still very valuable for your customers. That's what I would say.

• 59:00Speaker 00

  Nothing, no silver bullet.

• 59:04Speaker 01

  Amazing. Thank you so much, Jan, for joining us today. I've loved hearing a lot about your own thinking and reasoning behind your own journey. A lot of the, the thought process behind finding the right problem to solve, building the right systems to actually be in a position to understand the best problems. And then applying that even to many of the bold decisions that you've had to make in founding multiple companies from research into commercialization and the incredible success of Databricks today. Thank you.

• 59:32Speaker 00

  Thank you for having me.