Ep 69: Co-Founder of Databricks & LMArena on Current Eval Limitations, Why China is Winning Open Source and Future of AI Infrastructure

Ion Stoica helped define the modern data stack. Now he’s coming for AI evaluation. From co-founding Databricks and Anyscale to launching LMArena, Ion has shaped the infrastructure underlying some of the biggest shifts in computing. In this conversation, he unpacks what most people get wrong about model evaluation, the infrastructure challenges ahead for agents and heterogeneous compute, and why he believes the U. S. is structurally disadvantaged in open-source AI compared to China.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Jan Stoiga has an incredible background. He's the co-founder of Databricks, AnyScale, and now Ella Marina, a company that's raised $100 million to help other companies with evals. He's also a professor over at Berkeley. And today I was joined by guest host Rob Taves, general partner at Radical Ventures. And Jan, Rob and I talked about a bunch of things. We hit on Ella Marina, Jan's new company, and where he thinks the opportunities are. We talked about the future of AI infrastructure, where the gaps are in the space today. We also talked about what the US can learn from China to better improve open source model efforts within the country. This was a super fun opportunity to talk with one of the most brilliant minds in computer science about everything AI. I think people will really enjoy it. Without further ado, here's Jan. Well, thanks so much for coming on the podcast. Really appreciate it.

• 0:43Speaker 01

  Thanks for having me.

• 0:44Jacob Effron?

  Yeah, been looking forward to this one. And I was, we were joking beforehand, we timed this recording pretty nicely. I think we'll start with, obviously, the big news announced yesterday, the launch of your new startup Ella Marina, you know, $100 million fundraise. Tell us a bit about the company. You know, the product, the vision. What are y'all building?

• 1:00Speaker 01

  Yeah, indeed. So Ella Marina is based on the project we had at Berkeley, started almost two years ago. Well, two years ago. And actually, that was driven by a need to evaluate a model which we released, I think, March 2024, called Vicunia. So it was a model which was basically a fine-tuned Llama, the first version of Llama, using the shared GPT data. If you remember, the data which was people sharing their conversation with GPT, shared GPT, and someone making them publicly available. So we did that and, you know, Vicunia was,

• 1:43Speaker 01

  a few students, actually, the students did that even without me knowing. I realized later that they just, you know, we can fine-tune it, this model.

• 1:54Speaker 01

  And then there's a question, it's about evaluating it, right? It's like, of course, you know, you have a few benchmarks back then and so forth, like today, a lot of benchmarks. But also, it was a little bit harder because it was this kind of conversational, right? There is chat, right? So that was kind of how do you evaluate it? How you show that you are meaningfully better than what is out there at that time? And our initial effort was, what do you do? Like, you know, at university, you buy some pizza, get some students, ask some questions and evaluate them, right? On different models. The problems that was, didn't, didn't scale.

• 2:39Jacob Effron?

  I thought they could scale pretty well on a cognitive canvas.

• 2:42Speaker 01

  Yeah, yeah, yeah, but you know. And then, actually, we, we did a tour. It was two weeks before GPT-4 was released. So we had, we thought, okay, why don't we use GPT-4 to evaluate? And that was what became eventually known as LLM as a judge. So we did that, and to our surprise, it performed pretty well, right? So I look at that and, you know, did it. We got good numbers for our model. And, but then still people asking, because again, using LLM to evaluate was very early, right? The first or maybe among the first to do that.

• 3:24Speaker 01

  And I'm saying among the first because I think that some of the companies, they did it internally before, maybe, you know, Microsoft and so forth. And, but people are still asking, okay, you know, yes, I see that for this example, it looks it's doing well, but still, how does it compare with people? So that's why we started, you know, Chatbot Arena, and trying to scale up the kind of evaluation of humans. That was one reason. The other reason, it was these things about, it's like today, it's very hard to evaluate this like language model, and already there, there was evidence of contamination.

• 4:04Speaker 01

  Right, basically, you know, G P T four, I remember doing very well for some different kind of problems, which are before the cutoff for training, but not doing very well after that, right? So it was this kind of static benchmarks, right? So then we also thought about, okay, that if this is static of a benchmark, how do, it is almost equivalent if you think about people, is that you take over and over the same exam, right? It's like what it is, right? So, you know, how you make it and with humans, you know, it do make exams differently and different exam and so forth. So that's why the second, this is sort of the kind of the two reasons, right?

• 4:48Speaker 01

  We wanted to have a benchmark which is not static, it's more dynamic and also which captures the preference of the users, of the humans, right? That's all the two reasons. And then we tried to figure out how to do that. There are multiple ways to do it. You can do a little bit of a tournament, which is what the students are doing before, because you give, okay, this is a question, this is a prompt, and you have these like language models of, you know, whatever, three or four of them, you compare and see the answer to each of them and you rank them. Unfortunately, that kind of more like it's a tournament like almost, it doesn't scale as well. And it's not only that, but assume that while you are doing this tournament, the number of models is fixed. So that's one.

• 5:32Speaker 01

  And again, we look always inspiration of how humans do evaluation. And then it was, you know, this thing, okay, like in where are, you know, similar situation in which not everyone is playing with everyone and the number of players is kind of dynamic. And one is obviously chess. And but then many other sports like tennis, A T P, and you also team sports, they do that. So, yeah, so that yellow rating and kind of yellow rating we used. And then the model was very simple. You come, you ask your prompt. And we provide answers from two randomized anonymized light language models.

• 6:17Speaker 01

  And then you can pick one which you believe is better. And you may vote.

• 6:22Jacob Effron?

  Yeah.

• 6:22Speaker 01

  You are not, it's not mandatory to vote, but you may vote. And we get all this data and we compute these ratings.

• 6:29Jacob Effron?

  Right, and obviously it became wildly popular. When did you decide it was a company?

• 6:34Speaker 01

  Yeah, well, to their credit, you know, Waylin and then Anastasio who work on this. They believe more that it's a company before I did. I said, you know, this is just, this is just a valuation, it's just a leaderboard. What, what, what company is here, right? You know, but, you know, to their credit, they really believe in it. And then I think that you, you get to a point, and of course, there are so many things you need to do. And it becomes more popular. Some light language providers, light language model providers, like Frontier Labs, started to evaluate the models, also asking us to evaluate before the release.

• 7:19Speaker 01

  So it kind of becomes bigger and bigger, then people start to ask, okay, what about categories? People started to ask about, well, what about, you know, the humans is very subjective, right? Hey, this model gets high rating because it's doing, it's showing more emojis, right? So we introduced kind of style control to try to factor out some of these things. And or, you know. And so it kind of grew, and then another dimension, it started to grow, it started to text. And then multimodal, right, it was most text to image, image to text, text to web, to, you know, to generate code, and many more. So it kind of grew into an evaluation platform.

• 8:04Speaker 01

  And then as the data we got, it started to realize that it's quite valuable in terms of what you can do with it. And I'm not sure you've seen, but one of the features we launched one month or two months ago, something like that, is called prompt leaderboard. And prompt leaderboard, what it is, is that you come and you give your prompt. I may not, never have seen your prompt, but I can give you back. It's a leaderboard for your prompt.

• 8:37Speaker 01

  Right, what are the best models to answer your prompt? What is their estimated rating? And the way you do it is although I didn't, you know intuitively, it's not, it's not very accurate, but I think intuitively, I think it's, you know, it's reasonable, I think. So, although I never seen your prompt, I've seen a lot of prompts which looks like your prompt. And then I can use the votes on these similar prompts as a proxy to estimate the rating of the models on your prompt, right? You know, that's kind of exciting. And there are many other questions, you know, when you, you, you start realizing there is a potential here.

• 9:20Speaker 01

  So first of all, we did a company because there was no way to scale what we are doing, right? It's all, it's costly, yeah, right? Because what we provide the users is free access to these powerful models, right? And even, you know, like we spent for the past year, maybe, you know, probably close to two millions to run this. Of course, a lot of money comes in terms of credits, in terms of some gifts, grants, things like that, right? But if you think about right now, you know, want to scale 10x or something like that, it's clearly, it's a lot of, you know, it's not going to be cheap.

• 10:01Speaker 01

  The other thing it is was that, like I mentioned to you, it's a lot of, a lot of requests to do more of kind of evaluations. Right now, you have agents and you have all of this. We have a search now, you know, evaluate the search engines, not the search, you know, yeah, like perplexity and the OpenAI search and things like that. And then you need to build really a scalable backend. And you need to build something, you know, a much more responsive UI UX, you know, all of these things. You know, there is no way you can, you can do that with a few people.

• 10:39Speaker 01

  And, and then you have a model and providers and other people, both open source and proprietary, asking for evaluations. Right, and recently, you know, I have a student, Waylin, basically, you know, is like doing, he's a backend. Yeah. Right, okay, request, you know, like you put, so all of these things. And then you see that, again, once you have this kind of data, there are a lot of questions you can answer, right? Like I mentioned, prompt leaderboard, maybe you can answer questions about, which many people have, I build my application on a particular model, and now I want to swap the model.

• 11:20Speaker 01

  You know, maybe it's a new one, it's available, maybe something cheaper. So, but what would be the impact of swapping the model on my application? Many, many of these questions. And then, but what convinced me about that its potential here, and us in general, maybe not only me, is that if you look at AI, I think that one of the main challenges today with AI, it's reliability, right? You like, that's a key question, and it's very hard to see how AI will really achieve its potential without figuring out how to build more reliable application, right?

• 12:04Speaker 01

  Now, if you look about software, a lot of things we do, it's about reliability, right? It's actually most of the energy we put in software development is about reliability. Because it doesn't take a lot of time to write the code to provide a feature or a service. But you spend a lot of more time in order to test it, debug it, and so forth. And there are a lot of innovations, you'll have a CI/CD, all of these things. So, and that, in some sense, it's an easier problem, because the software is pretty wide box, right? You see the instruction, you change the instruction, you can see what is the state of the program after each instruction. If you don't use a debugger, you can do printf's and things like that.

• 12:49Speaker 01

  But models are more black box, right? So if you look at the analogy, and we still want reliable application, and because even if you look now, the most successful AI applications are the ones which have a human in the loop, yeah, for the same reason, to validate the answer, like code assistance, customer support.

• 13:11Speaker 01

  And many others. So then if you think kind of the analogy, you know, you have to have ways to to test and validate and build more reliable AI applications. Again, big problem. And it's seems like obviously there are many things you need to do, but also something like LM Arena will help.

• 13:34Speaker 02

  Super interesting, Yana. It makes a lot of sense. And I think as you've touched on evaluations and like validation of reliability for AI systems is one of the biggest unsolved challenges. I think it's a it's a massive opportunity you guys are tackling. Is your vision and is the thesis of of the company LM Arena that human based evaluations will always be the best way to go and that that needs to be the core of it?

• 13:57Speaker 01

  Yeah, that's a great question. Happy to answer that. That's why we have our guests here. They ask the great questions. Yeah, so I think that it is very interesting here. Like we have a lot of discussion on this and probably you've seen and so forth. I think it's it's it's one evaluation. There are many other evaluations and you should look at many evaluations, many benchmarks. However, if you think today, like we discussed early on, that most of the application today they have a human in the loop. So this makes the human evaluation particularly important, right? And there are people saying, well, you know, like we mentioned to you about this

• 14:39Speaker 01

  style control, but you know, it's it's it's people like that model, not because it's better, but because it's funnier or because, you know, or emojis or something like that. And the answer to that question, well, first of all, even that is relevant. If you build application which interacts with humans, you want to know that.

• 15:02Speaker 02

  Yeah.

• 15:02Speaker 01

  Right? And by the way, guys, you know, like, we as a human, right, is like, you know, we like people who give better presentations. Yeah, all sorts of biases. Sort of biases, right? So you want to understand that. So that's kind of very legitimate, right? And of course, now the other thing is that you have enough data. If you know a bias, you can factor it out. Right, and this is what we do a little bit with style control. You know, for instance, formatting, right? If you have nicer formatted author, you like it better, right? Like, so you try to factor that out, that you can speak style control. Maybe your style control will be by default in the future, right? In which you factor out some of these things.

• 15:43Speaker 01

  Or so, um, so and then, you know, that's having a platform in which you can try to remove certain biases. Of course, you can remove only the biases, you know, they exist, right? That's a problem. Um, but definitely. By the way, you know, this is funny about the biases. Um, but you know, when we look, um, I mentioned to you that when we do this, when we, when we, um, um, build, uh, when we, when we, when we use this G P T four for as a, as a judge early on. And then we build, uh, you know, chat bot arena, one of the reason to answer the question about how well and how does it compare with G P T four, because people are asking.

• 16:26Speaker 01

  We did a, we wrote a paper and we did, uh, we made a study. And the funniest thing is that, um, is that these L L M's have also biases as a judge. Like, for instance, if they have position bias, they in general prefer the first answer. They have verboity bias, the more they prefer the more verbose answers, right? Um, uh, at this time, maybe now they are better. They are not very good at math, like people, right? They are not very good at math. So it's very, a lot of biases of, um, of, or liking their own answers, right? Or, you know, from their own family, right? The family of models, right? It's like Lama model, it's because the judge is going to like other version of Lama models.

• 17:11Speaker 01

  Um, so, uh, but that's kind of a very interesting, because even of these models, maybe, maybe not surprisingly, again, they are. Trained on artifacts, on what are produced by the humans. So, yeah.

• 17:25Jacob Effron?

  That's interesting. I feel like, obviously evaluations, I feel like any startup you talk to, this is like, you know, one of the top things they're thinking about. And I think a question's always been, as people have been building tooling in this space, you know, how generalizable can you make an evaluations tool for a company? Because I think every, you know, in some sense, the most valuable thing companies have is their own evals, right, that are specific to their own use cases. And so, how do you think about like providing a general set of tooling to companies in various different end domains?

• 17:52Speaker 01

  Yeah, that's a good question. I don't think I know the answer to that question.

• 17:55Jacob Effron?

  Good, I don't either.

• 17:56Speaker 01

  We are happy. But, you know, things like that, like I mentioned to you, the prompt leaderboard, are going to help, right? Because in that particular case, we don't know, again, your, we haven't seen your prompt or your question before. But we can still tell you, you know, with some statistical meaningful margins, that which are the best models for that question. So that's kind of generalized easily. You give me the set of data, and I can give you, you know, the best models for your set of data. I can give you also, probably you can do personalized, right? You can do what are the best models

• 18:36Speaker 01

  for you, right? So I think you can do, that's why the key for that is to scale, right? To get more data. The more data you're going to have, the more kind of this kind of micro categories you can have.

• 18:50Jacob Effron?

  Yeah. Makes total sense. What do you think happens to the, you know, model landscape over the next few years? Maybe starting with the, like, most state-of-the-art closed-source models?

• 18:59Speaker 01

  Yeah. I think really, if you think about right now, the trend, obviously, and that was...

• 19:13Speaker 01

  The open source model caught up the progress or quite impressive was the past year.

• 19:20Speaker 01

  Of course, it was.

• 19:24Speaker 01

  Not of course, but maybe surprising and also surprising thing is that these models are not necessarily from US or from China.

• 19:34Speaker 01

  But definitely at this point, if you're talking about the open source model is the best car coming from China. And if you look at the trends, you know, the open source should be able to catch up with proprietary models within one year or so.

• 19:51Speaker 01

  And yeah, now the question where this open source model are going to come from.

• 19:57Speaker 01

  And what I can say is that in China, it's a lot of, right now, a lot of momentum and for, I think for structural and reasons. And you know how the ecosystem is right now grew in China versus US. So I think that we in US are a little bit of disadvantage now. Say more about that, why is that? Structural. So first, if you think about what do you need to develop these models, right? You need three things. You need experts, you need data, you need infrastructure, right?

• 20:37Speaker 01

  And I think that if you look at China, they have a lot of experts, probably sheer number, they have more than US. Data, they have data, they have, you know, they don't have as much infrastructure, but they are making progress, right? If you look at the latest announcement from Huawei and so forth and, you know, export controls, you know, everyone guess about how effective they are going to be or they are. That's, you know, that's to be, you know, to debate.

• 21:12Speaker 01

  But the one thing there is that the, the open source, it's much more prevalent, right? It's almost by default. Now, if you go to US, is that you have huge resources and so forth and you have lots of smart people. The problem is that the development, what have, how happens in US is siloed.

• 21:42Speaker 01

  Right, each, you know, this frontier lab, right? Everyone is doing the same thing, basically. That's number one. The number two is that academia, when you talk about pre-training and building the models, doesn't play a major role because of lack of resources. You know, there are companies like AI2, there is a group from Stanford, Percy, Liang, trying to develop, and Berkeley tried to develop this, you know, pre-training, post-training, full open source pipelines and so forth. But it's a challenge.

• 22:27Speaker 01

  And unless that changes, we are going to be at structural disadvantage because the diffusion of innovation is very limited. Right? It's like you have people, so the way you can, if you want to maximize the rate of progress, you have to have all the smart people collaborate. Right? But if you have kind of this silos, and a big part of the researcher, I'm talking about academia, not being able to contribute in a meaningful way to developing these models, then it's a significant disadvantage. The main diffusion of innovation here in the US is that through people who leave one company

• 23:12Speaker 01

  go to another or start new companies. But if you look, for instance, in China, there is a much stronger collaboration between academia and industry, you know, like Baidu or Alibaba and obviously DeepSick. And that really helps them. What is the reason academia is so much more effectively involved in China? How does that come about? Because it's much closer collaboration with industry. Here it's very hard to collaborate with the frontier lab because everything is secret. If you really want to maximize the rate of progress, again, you have to have all your researchers or your experts collaborating.

• 23:57Speaker 01

  The only way they can collaborate is by shared artifacts, which is means. As a software side, open source models. And they need also to have a shared infrastructure.

• 24:08Jacob Effron?

  Yeah.

• 24:09Speaker 01

  Right.

• 24:09Jacob Effron?

  I mean, obviously, the motivation behind a lot of the closed labs is, you know, this belief that, you know, they, you know, being the first to find these state of the art models, there's all these dangers of what they can be done in bio or cyber, and they need to get to them first. Like, what do you make of that?

• 24:22Speaker 01

  Look, I mean, I am on, I am optimistic in general. I am not, I think it's very, you know, you always need to remember that as a human, we are driven by emotions, and one of the most, by far, powerful emotion is fear. So it's, you are always going to respond to that much more, much stronger, right? But then providing you some optimistic view, which is not palpable. But the other one, you know, is you are, you know, like, kill you, right, in my hero. So you are going to react very strongly. So I think to start with, whenever you are seeing this kind of discussion, you need

• 25:04Speaker 01

  to discount the negative one, because you, as a human, you are going to be much more prone to respond to that. Yeah, right? Just to keep a little bit of objective, objectivity.

• 25:16Speaker 01

  The other thing I would say is that I still do, you know, you are talking about, and this was SB 1047 last year and so forth.

• 25:24Speaker 01

  You are talking about, seeing about the marginal risks, right? And marginal risk is about risks which are not present before, but they are enabled by this technology. And right now, I still, still need to see real marginal risk enabled by AI. You know, it's, it makes the risk, the previous risk more prevalent. They are making much worse, maybe. But if you talk about, you know, deep fake and so forth, I mean, you could do that before with Adobe and so forth, and impersonating people, it's all the way from antiquity, right? So, so that's one, you know, obviously, it's like, they speak, oh, you can, tell you how to build a bomb.

• 26:08Speaker 01

  So first of all, what it tells you, it can make it maybe easier in the best case scenario. But if I am going to kind of any library and so forth, I should be able to find that information if I am determined. So it makes this much easier, maybe it's ten times easier, maybe one hundred times easier. But if you look at end to end, what you need to do to apply to make that a reality, acquiring the knowledge is like very little, right? Then you need to get the materials, you need to assemble without anyone detect it, you need to deliver that or something like that. So if you look at the US one, that's kind of dominating. So yeah, I'm the ten percent of acquiring, which was before, I'm going to do it to one percent or zero point one percent. Yeah, sure, but the rest is not going to get better, right?

• 26:53Speaker 01

  It's like, it's still gay. So that's kind of, I am, when you think of this kind of level of details, not details, it's, doesn't convince me that is this existential risk and so forth. Of course, everything, you know, there are things you cannot imagine, granted.

• 27:12Speaker 02

  You mentioned physical infrastructure as one area where China is still lagging and export controls are a piece of that and so forth. In the US, obviously in the West more broadly, a massive physical infrastructure build out is underway and all of the big hyperscalers are pouring tens of billions of dollars into building out massive data centers, one gigawatt, five gigawatt data centers, et cetera. Do you think that, like, that's the right approach? And that's a good, good trend to be happening in the US? Do you think there's some risk of overbuilding infrastructure? I'm curious how you think about that over like.

• 27:45Speaker 01

  I think overbuilding, it's very likely that will happen, like with the internet.

• 27:49Speaker 02

  Yeah.

• 27:49Speaker 01

  That's what happened with the internet. We overbuild the internet and a lot of companies were overbuilding it, you know, we went under and there are a lot of other companies which really took advantage of that infrastructure, like, you know, Google and Amazon and so forth, right? So probably very likely this is something like this could happen now, right? It's like, clearly, look, it's much easier to get GPUs today than it was one year and a half ago. Okay, for one, that's, that's a fact, right? I'm not, you know, it's not about housings. But I do think that. China is not going to, you know, is going to be two years, three years. I don't know, it's like, but they are going to build and the advantage is that,

• 28:32Speaker 01

  you know, they have an economy which is in the same ballpark. I don't know, whatever numbers you look at with the US economy, right? And the other thing what they do, they, they have the ability to fund strategic initiatives, you know, many, many years, decades if needed, right? So that's kind of gives them maybe a structural advantage there. Um, so we'll see, we'll see. But and also they have a lot of experts and like we see DeepSeek, right? They are going to, you know, do optimization at the lower level. And some people say, oh yeah, but you know, they do that optimization. If we go for them to spend their intellectual cycles to do that optimization, sure. But you have enough resources, it's going to work out, work out for you.

• 29:15Speaker 01

  But you know, look, look at and, and look, you know, this is also, um, always amazes me that people say, oh, you know, it's like this kind of belief, which is confidence that we are always going to be ahead. But there are many, many, um, high-tech industries which we are no longer at. Like solar cells, car batteries, drones, uh, electric cars.

• 29:46Speaker 02

  Yeah.

• 29:47Speaker 01

  Um, robot, you know, at least the industrial robotics.

• 29:52Speaker 02

  Maybe transitioning topics slightly. You have co-founded and helped build many defining infrastructure companies over the years, software infrastructure companies, Databricks, AnyScale, now Almarina. Aside from the evaluations topic, which obviously you guys are focusing on with the new company, what do you think the biggest opportunities and the most important unsolved issues are at the infrastructure layer for AI right now?

• 30:20Speaker 01

  Again, you look, you look at the trends and, and clearly right now, what we see and every lab is doing, um, we are seeing a lot of, um, the infrastructure evolving towards being very, you know, vertical integrated, you know, co-design across all the layers, all the way from the application to the hardware. And I think that you are going to see more and more on that. That's one I'm going to say again, based on the trends where I think there are. Some opportunities. Sorry. Clearly, what we are seeing here, right? It's like, one thing it's about, you have this kind of distributed heterogeneous infrastructures, right?

• 31:05Speaker 01

  And if anything, you know, you have at the accelerator level, you have GPUs, Nvidia, and MD, and you have many other accelerators like TPU, Tranium, and others. Then you have at the networking, right? You have Ethernet, you have InfiniBand, and then a little bit higher level RDMA. So you have a lot of this. And then collective communication like Nickel, Recall, and so forth. So huge heterogeneity. So you need to master that. And the one thing I would say is that, and there is quite a bit of work, I'm hopeful,

• 31:46Speaker 01

  it's about automatically optimizing and generating optimized toll-level code kernels, right, for these accelerators. I think that's, hopefully, it will happen. So that will help us support much easier news and optimize for a large variety of hardware and networking. I think that it's going to see a lot of more optimization at the intersection between networking and compute, right? Fine-grained, overlapping the communication with computation and so forth. Obviously, load balancing is going to be very important. So that's one.

• 32:28Speaker 01

  And when you look about, in order to optimize this kind of models and workloads, it's so complicated, right? It's like, if you have, if you look at this kind of paralymp, like for model, model serving and training, you have, the last time I was looking, it was like seven. You know, model data parallelism, model parallelism, or tensor parallelism, pipeline parallelism, context parallelism, and the token parallelism, and expert parallelism, sequence parallelism, I forgot, so I think this or seven.

• 33:10Speaker 01

  So it is, you know, it's like you need to do this in automated way, right? Because there are too many, and then is there very fine grain again between communication and computation, because you want to overlap the computation with communication to increase the utilization of GPUs. Obviously, we'll see what happens as an agentic level.

• 33:34Jacob Effron?

  Yeah. What do you think the like, infrastructure needs will end up being for agents? I feel like there's been some early attempts at broad frameworks at other, you know, air source areas. It feels like the model companies themselves are building lots of things.

• 33:47Speaker 01

  It's hard, because when you have a field which is so, moves so fast, it's very hard to come with good frameworks and which are going to be stable over time, right? Because it's just, the needs, you know, change every month, every week, every day, right? So that's kind of makes it difficult. So I think this will be typically when you start to build good frameworks or good software abstractions, is when the, the speed of evolution of the, at the application level, it's a little bit, it's kind of slowing down.

• 34:30Jacob Effron?

  Is that ever going to happen, given model progress?

• 34:33Speaker 01

  Well, you know, it's like, look, the different layers, it does happen. I mean, it's like now everyone is using transformers, everyone is using kind of PyTorch, right? Almost. There are things like.

• 34:51Speaker 01

  We are using right now, you know, when you talk about inference, we are using OpenAI API, more or less. So yeah, I think there are, you are seeing that, you're kind of the standardization at a low level. If you look about, for instance, a lot of new post-training frameworks, every day you get another one, but most of them are built on Ray, Vialan, and maybe, you know, Edge lang also. Yeah, so I think you see at different points, some kind of standardization, but it's still, it's pretty like dispersed.

• 35:31Jacob Effron?

  Since the, you know, generative AI explosion, I feel like it was this, you know, huge moment for Databricks, and in many ways, you guys have kind of met it head on. Reflecting back maybe on the last two and a half years, like, what do you think the company got like most right in the kind of immediate post-chat TBT moment? And then maybe, if you could do it over again, something you might have done differently?

• 35:49Speaker 01

  So I think that one thing was, you got right, it's about, the data is as important as ever, right? So getting access to all your data in a seamless and performant way, while having all the governance on top is very important, right? And this is what we've done with Lake House now, with the Unity Catalog. I think that's the key, because if you are a company and enterprise, a large enterprise, you are going to have the data in a myriad of storage, right? Legacy or newer, you know, like used to be data, like something like

• 36:32Speaker 01

  that. So having access, uniformly accessing the data, and not only that, to have the metadata associated with the data you are going to access, is super, super important. Okay? So I think that was, that was right, and you see even today, it's, that's why, you know,

• 36:53Speaker 01

  Our main conference is called Data Plus AI, right? Data Intelligence is a new category. So, and you'll see, it's a huge push in that direction. I think the other one is about, also after acquiring Mosaic and so forth, we are very early on, very aggressively pursuing AI for enterprises, right? And it's perfect, right? Because you have enterprise, they have the data. In general, the data, it's one of their big crown jewel, right? It's like, it's unique.

• 37:38Speaker 01

  And then you help them to extract a lot of value from the data and build new products powered by AI on their data, right? So that's one thing. Now, the one thing I want to mention here is that, and people may not realize that, but this AI was in the DNA of Databricks. When we started Databricks, with Spark, one of the main libraries on top of Spark was machine learning library. It was MLA, then SparkML. And it was, of course, it wasn't deep learning at that stage.

• 38:22Speaker 01

  It was classic machine learning, random forest, linear regression, anything like that. But that's one. And early customers actually were buying Databricks, you know, products and Spark, because they wanted to do AI, right, early on. So it's coming, in some sense, full circle. But I think, you know, that's, I think, you know, it's hard to know what you would have run differently. Of course, we tried to build, you know, we built, and it was a pretty high-performance model, D-bricks, right? So if you go back, whether you are going to do that again, it's a question.

• 39:03Speaker 01

  Given the power, given how many powerful open-source models have been released. Of course, I cannot talk about everything here, but I think we did.

• 39:15Speaker 01

  For the last two years and so forth, I think that I don't think I'll go back. I would go back and say we should have done differently. And of course, we need to talk with Ali and so forth. Much better than I know. And he will be better to answer these questions.

• 39:31Jacob Effron?

  You're so close to all this, you know, cutting edge stuff in the AI world. What's one thing you've changed your mind on in the last year?

• 39:37Speaker 01

  Maybe the way I would answer the question, what things are I thought will happen and didn't happen. And by definition, you'll change a little bit your mind on that. I thought that we are going to see more alternative to Nvidia, more right? And we haven't seen that yet. That's number, that's one. The other thing I was pleasantly surprised by the progress of the open source. Now, this came from where I maybe one more than a year ago, I haven't expected, you know, came more from China rather than US.

• 40:18Speaker 01

  I thought that will make more progress on reliability, hallucination and so forth. But that still remain a problem, especially with the reasoning models. I think that certainly reasoning models are probably more effective than, not reasoning, post-training was more effective than I thought. And one particular thing which, okay, one I know now, one thing I changed my mind. I was not very big early on on quantization. Because I saw that everyone, you know, like people, quantization makes a trade off, right?

• 40:58Speaker 01

  It's like a little bit hit the performance to provide your better efficiency.

• 41:07Speaker 01

  And I thought that people will not be willing to do, to make that trade off. But that's there. And quantization, without question, is very, has been very successful and a game changer.

• 41:19Jacob Effron?

  Yeah. There's a lot in there that you said that's interesting. I mean, I guess to go back to your first one, like, do you still think over time, you know, in the next few years, there does emerge real challengers to Nvidia? Or like, has what's happened over the last year changed your mind on the viability of that?

• 41:31Speaker 01

  I think that could be for, you know, it's.

• 41:37Speaker 01

  Certainly, you know, probably China will put a lot of effort into Huawei because they don't have any other choice if things do not change. Otherwise, the export control will get worse or something like that. That could happen. I think that, yeah, and I think that clearly we see a lot of investments. Google always invested on DPOs, then AWS makes a lot of effort on Tranium, and then AMD.

• 42:06Speaker 01

  Very good hardware. I think the biggest challenge, as you know, is like it's softer. The software stack, right? One of the biggest challenges. We'll see. I'm hopeful because, you know, it's like it's obviously from the software layer, it's like, well, you never know. Like, on one hand, if you have a lot of more choices, you need to put a lot more effort in the software to do these different choices.

• 42:30Speaker 02

  Continuing with the crystal ball gazing a little bit here. I'm curious to hear what's your view on our current trajectory toward superintelligence AGI timeline there, if it's even a coherent concept? How do you think about that debate?

• 42:45Speaker 01

  You know, I used to joke that with AGI, everyone will be right because they have no good definition.

• 42:50Speaker 02

  Right, right.

• 42:51Speaker 01

  So I said, I am right because what I call is I said is AGI, right? I don't know. It's like, and this is that, and what is AGI? Like what, but people say that you have one artifact that is going to

• 43:06Speaker 01

  do majority of tasks better than humans, right? But how that artifact will be built? I can, you can build even probably today, something like that, or maybe very few years. But I don't think that intuitively what people mean by, you know, AGI. But if you look historically, right, there are, you know, it's like there are an increasing number of tasks at which computers are better than humans. I think about calculators, right? That was 70s or 60s, right? They are better than humans for a long time, right? Then playing games, right? It's like.

• 43:44Speaker 01

  Chess, right? It was Deep Blue, ninety-seven, something. And then it was, obviously, Go, two thousand seventeen. And now you have kind of more ImageNet. It was for a lot convolutional tricks for recognizing, image recognition, which is arguably better than humans. So you always have these kind of things which are going to add up, right? Maybe you're going to prove and so forth. So, you know, you're going to have more and more tasks, which computers are better than humans. And you can package them in an artifact. And depending on the task, you can invoke different things under the hood.

• 44:26Speaker 01

  But from the user perspective, it's the same. Although people, you know, that, some of them will not call that AGI. But one thing I say, so I'm not going to, I don't know, right? But one thing I was going to say is that what you see there, this goes back to the discussion about reliability a little bit. But you see progress. You see progress where you have a good validation, good test for the answers. You have ground truth. That's what you see, okay? Like, like you're talking about calculator, right? It's like, you know, it's like, it's only one right answer. If you look at games, it's very easy to test whether

• 45:11Speaker 01

  you're successful or not, you win or lose. It's also, the rules are pretty clear, yeah, right? If you think even reasoning models, where are they more successful, right? Problem solving, coding, right? Where you have ground truth. But this is natural, it shouldn't be a surprise. Because if you look, forget about, you know, computers. If you look at the progress was done in the history of humanity, just over the past, you know, say two hundred years, where did it happen? In sciences, engineering, where you have miserable outcomes.

• 45:51Speaker 01

  Chemistry, mechanical engineering, electrical engineering,

• 45:57Speaker 01

  physics and so forth. This is what the problem, right? The world for these sciences is today very different than 200 years ago. Where things are not as measurable. Like novels, books writing, creative writing, right? Arguably, you don't see the same progress. Actually, some people will argue the other way around. Right? So that's kind of what I want to say. So this is what I'm trying to say is that I think that we are going to see continuously for the measurable, you know, the task or use cases measurable outcome. You are going to see very rapid progress. But for the other one, it will be more subjective.

• 46:38Jacob Effron?

  Yeah. Do you think we'll see a proliferation of what measurable tasks are? I mean, obviously, a lot of people in the lab seem to think you can build reward models for almost any domain.

• 46:47Speaker 01

  Yes, but there are things which are going to be more called kind of objective, what is true or false, right? You can have formal specification and so forth.

• 46:58Speaker 01

  And things which are going to be more subjective, right? And because what is a good book, right? How you are going to do, develop a reward model for that? That's what I think that you may have, you know, your personal reward model. Maybe you can, someone write a book for your model, right? Your own model.

• 47:19Speaker 01

  So, yeah, you are going to do. But reward models are not as efficient as I said, right? Like, for instance, if you have even for,

• 47:29Speaker 01

  say, you build, you know, so you do, you solve mass problems. And you can use to have a reward models to kind of learn what is good or bad

• 47:40Speaker 01

  result. And now you compare that if you have the ground truth. I know this is the results, the correct results, right?

• 47:52Speaker 01

  If you have the ground rules, it's much more efficient. Maybe one order of magnitude. More efficient in terms of the compute to get to a particular accuracy for whatever benchmarks, AME or whatever.

• 48:07Speaker 02

  This is almost a philosophical question, but within the hard sciences, which are verifiable, do you think that the current paradigm of AI that we're in, it's possible for AI to generate novel ideas and breakthroughs?

• 48:21Speaker 01

  Definitely. But there is a bottleneck will be testing them. Right? Right? Because now you are talking about a little bit more reinforcement learning and so forth. Right? Because I generated, that's a generative AI. I generate a solution.

• 48:35Speaker 02

  Yep.

• 48:36Speaker 01

  Right? Now you need to test it.

• 48:38Speaker 02

  Yep.

• 48:38Speaker 01

  Right? So that's kind of the, will be the bottleneck.

• 48:42Speaker 02

  It seems like AI research will be like the first place.

• 48:44Speaker 01

  Yeah. Yeah. AI research, yeah. It's like, well, yeah. But then it's, what is, what is, what does it mean to be, to generate a good, you know, a good artifact, research artifact, right? Because right now we have so many papers, right? It's like.

• 48:59Speaker 02

  Yep. And so in that sense is, can you think of AI creativity as basically just being, generating a ton of ideas, some, most of which are awful, some of which are good, and then just being able to accurately.

• 49:10Speaker 01

  There are two parts of it, right?

• 49:11Speaker 02

  Sounds like my typical branch.

• 49:12Speaker 01

  Yeah. That's, that's sort of things, right? It's like, yeah, it's like brainstorming, right? You know, on all hand, you need to, even to solve a problem, right? It's like this method and many, many, like all the way alpha, you know, alpha geometry and so forth, it's the same kind of pattern. You generate lots of solution and then you select the good solution. So it has two parts, right? First part is generating solution. Now this means that you, you, a condition to find a good solution, good solution is that at least one solution that is generated should be good.

• 49:52Speaker 02

  Yep.

• 49:52Speaker 01

  Right? And then it's selection. Selection is also very hard. You need to, the, you need to live in the haystack from all these myriad of solutions you generated. You need to identify that good solution. And that's very tricky because even our probability of identifying the solution, you say is 99%. But if you have to pick from one million solution or good solution. You are guaranteed you are going to pick the wrong one. With very high probability.

• 50:19Jacob Effron?

  Yeah.

• 50:19Speaker 01

  Right? So that's kind of, these are the two, the two things. And, but yeah, I think that right now, even today, when you have a human in the loop, the best application are the one in which the solution, generating the solution is hard, but verifying it is relatively easy.

• 50:39Jacob Effron?

  Well, it's been fascinating. We always like to end our interviews with a quickfire round where we get your thoughts on a standard set of questions. So maybe to start, what's one thing that's overhyped and underhyped in the AI world today?

• 50:49Speaker 01

  I think underhyped is reliability. Still, people discuss, but it's not enough. Because if you believe that that's one of the main challenges which will hold back the proliferation of AI and solving real problems, or I think that's, that's all. Overhyped, maybe a little bit, all this kind of scale, maybe scaling laws. Every now and then is looking for scaling laws. Especially as a post-training recently, we've seen that if you have, of course, a powerful base model, just with a very small set of high-quality data, you can unleash new capabilities in that base model.

• 51:35Speaker 02

  Another rapid-fire question for you. What's one AI startup outside of your area of focus that you're really excited about or bullish on?

• 51:43Speaker 01

  This is probably one of the most successful applications, but I still be very curious about, you know, startups like Courser or Insur or something like that, call assistants, how far they can push this. Because it's clearly that, you know, they are, they are great for some of the use cases, you know, but when you have to have bigger.

• 52:11Speaker 01

  To have, to work with the context of the entire code base and things like that is much more difficult. And I think that the other point there is about, for instance, once you generate more and more code, then it's going to, how easy is to maintain it? Right, the maintainability. But the reason I'm saying, you know, obviously, Curse, it's, and they made a lot of progress. And here, maybe it was more than I expected. That's what I would say. And also, it's interesting because there are a lot of still open, interesting questions there in the space. And if successful, obviously, they are going to have, there'll be a massive impact.

• 52:55Speaker 01

  But in that space, no, it's, the area, the space is going to get, it's quite crowded now with the big players also entering. But I think that space, it's, we are going to learn a lot. Because in some sense, this is like canary in the mine. Because, you know, the code assistance is like, it's almost a perfect application because you have developers which are early adopters of technology. Right? So they are going to adopt. The other thing is that this code assistance, coding assistance, they already, they fit in the existing workflow. Right?

• 53:38Speaker 01

  So I think that this kind of having early adopters and fitting naturally in the workflow are big, big advantages which you are not going to see for other examples, like doctors or lawyers and things like that. So I think that it's very interesting to see how far this will go.

• 53:56Jacob Effron?

  Well, I'm sure there's all sorts of threads folks will want to pull on. You obviously are involved with so many interesting parts of the AI world. I want to leave the last word to you. Any place you'd point our listeners where they can go to learn more about you, the work you're doing?

• 54:09Speaker 01

  So, you know, we are doing still a lot of work. A lot of work is happening at Berkeley, Sky Computing Lab. And obviously, the companies, all of them, they have these websites. Good websites, presumably. But I think that the cutting edge research we are doing, it's people should go to Sky Computing Lab. We have blogs and everything. Amazing. Well, thanks so much. This was a ton of fun. Thank you.