Ep 47: Chief AI Scientist of Databricks Jonathan Frankle on Why New Model Architectures are Unlikely, When to Pre-Train or Fine Tune, and Hopes for Future AI Policy

Jonathan Frankle is the Chief AI Scientist at Databricks ($43B), which he joined through the acquisition of MosaicML in July 2023. Databricks has over 12,000 customers on the cutting edge of AI; Jonathan works to anticipate their needs and offer solutions even as the tech is rapidly evolving.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Jonathan Frankle is the chief AI scientist at Databricks, which he joined via the Mosaic acquisition where he was chief scientist. Today on Unsupervised Learning, I had a really interesting conversation with him, including how he works with enterprises to think through when they should train their own models, fine tune, or just use prompt engineering. We talked about the lessons he and the team have learned from working with Databricks 12,000 customers on AI and the new evaluations product they're releasing. We talked about Jonathan's reaction to O1, Anthropic's work on computer use and more. And we also hit on policy and what Jonathan thinks is required for us as a society to feel comfortable using AI in medicine, self-driving broadly, and more. A really interesting conversation that took a lot of turns. I think folks will really enjoy it. Without further ado, here's Jonathan.

• 0:39Jacob Effron?

  Jonathan, thanks so much for coming on the podcast.

• 0:41Speaker 01

  Yeah, thank you for having me.

• 0:42Jacob Effron?

  Been looking forward to this one. I feel like I've seen you on a bunch of panels and other podcasts, and you always have some good takes. So I'm sure our listeners will enjoy this.

• 0:50Speaker 01

  Now I'm nervous. The bar's been set really high here.

• 0:52Jacob Effron?

  I'm kind of bummed because as I understand it, you know, maybe I forgot exactly when you guys released your open source model initially, but I believe you dyed your hair blue as part of this. And I feel like now it's, I guess it's grown back fully and we don't get the full kind of visual effect here.

• 1:06Speaker 01

  Nope, just the gray at this point. I tell my team, like, you know, I'm happy to put my body on the line to incentivize them to do awesome work. I found like I buy swords for people when they do great acts of service for our research team. You think I'm joking, but I like engineers, external partners, our legal team after we came through the DBRX launch, they all have swords with the Mosaic logo and their name on them.

• 1:28Jacob Effron?

  Wow.

• 1:29Speaker 01

  Like, there is a Databricks-approved sword vendor now that I had to get through procurement. But, you know, I'm serious when I look for good incentives for people. And my team hasn't proposed anything. Like, they haven't offered a tribute that would qualify for the hair to get dyed again. But I'm always open-minded.

• 1:43Jacob Effron?

  What would it take? What do you think is the next?

• 1:45Speaker 01

  That's not my job. That's the team's job. The team set some bars for DBRX and what performance it was going to hit. And, you know, when I offered that as incentive, suddenly they got to their scores really quickly. And I checked, they didn't cook the books. Like, they actually got there the old-fashioned way. But, you know, as a manager, my job is to get my team excited and make them like really want to go the extra mile. There are lots of ways to do that. You can just throw money at them. You can threaten them. But my view is like, give them fun incentives and make them. Them really want to do it.

• 2:11Jacob Effron?

  Yeah, as I understand it, what is the last two weeks of basically before you ship this model that that you kind of blew through a bunch of these benchmarks?

• 2:16Speaker 01

  Yeah, they just like they really put in a ton of work on the post-training and did a phenomenal job. And it was a really solid model. Of course, Meta went and dropped a half billion dollars on some awesome models after that. And, you know, we love the Lama three models like they're phenomenal, Lama 3.1 and 3.2 models now. But, you know, sometimes you get blown out of the water after two weeks, but two weeks is king of the hill is, you know, not the worst thing in the world.

• 2:36Jacob Effron?

  Yeah, exactly. Well, we'll definitely get into some of that stuff on the model side. I was thinking about kind of where to start, and I think, you know, one interesting angle would be, I feel like you have this kind of like famous bet on whether transformers are going to be the dominant architecture, you know, over, you know, I forget what exactly the period of time is that this bet ends. But I'm kind of curious, like now that you're a year plus into this, how do you feel about that bet?

• 2:58Speaker 01

  Oh, well, you know, I really take the long view like I'm not looking at this day to day. I'm I'm much more, you know, I take

• 3:04Jacob Effron?

  You're not getting existential with every paper that drops.

• 3:06Speaker 01

  I mean, you know, I it's kind of funny, like in the days after the transformer paper came out and BERT came out, there were a million papers of just like this tweak to BERT, that tweak to BERT, you know, this kind of BERT, that kind of BERT. At the end of the day, the models we train are basically the Vaswani at all transformer with different positional encodings and just the decoder. Like, at the end of the day, they kind of found a sweet spot in the hyperparameter space. And I don't think it's really been worth it to anyone to shake things up too much. And I, you know, this is a challenge I'll issue to you. This is, you know, a fun thing I love to do on these podcasts. What came before the transformer in terms of NLP kind of state of the art architecture?

• 3:40Jacob Effron?

  Yeah, there were recurrent neural networks.

• 3:42Speaker 01

  Okay, recurrent neural networks, but specifically LSTMs. Now, this is the awkward part, which is older, me or the LSTM?

• 3:48Jacob Effron?

  I guess the LSTM.

• 3:50Speaker 01

  Ah, you're, you know, actually, I'm older by one year. I think the way you should think about this is good architectures are really hard to find. It took us a basically a generation to go from, you know, what were then the state of the art architectures to what we have today. There's kind of this alternate journey we didn't get to go on of what would have happened had we just scaled the hell out of LSTMs. Transformers maybe came along at the right time, they had some beneficial features. Are they really fundamentally better than LSTMs or is it just where we put our energy? And so that's what panned out. It's kind of hard to say, but new architectures kind of hard to come by. The architectures get simpler over time, not more complicated, just, you know, in general transformers are a lot simpler than LSTMs. AI newcomers come along and they see, oh, the transformer happened recently, therefore there must be some other big thing. Oh, this is the next big thing.

• 4:33Speaker 01

  That's not quite how the field works. That's not how science works for that matter. Science is not like this nice linear scaling law where everything just moves up and to the right on a log log axis. You know, science tends to move in these big leaps and then, you know, we consolidate for a while. At least that's, you know, we can get into philosophy of science, but that's my take on at least what I've experienced and what I've read about. So with that in mind, like the idea that suddenly something's going to come along that'll blow the transformer out of the water seems ahistorical to me.

• 4:58Jacob Effron?

  Yeah, all right. So still feeling good?

• 5:00Speaker 01

  Still feeling very good about that bet. And, you know, Sasha knows exactly what my equity is worth now after the Databricks acquisition.

• 5:06Jacob Effron?

  Oh, you bet. You bet, actually, equity.

• 5:07Speaker 01

  Oh, yeah. We bet, you know, hugging face versus mosaic equity to charity. So, you know, Sasha kind of has a sense for what he might be getting now that Databricks' valuation is a little, you know, easier to discern than the mosaic valuation was. Hugging face, you know, harder to tell. So, you know, really Sasha's got his incentive there. I want to know what my incentive is.

• 5:25Jacob Effron?

  Well, and, yeah, in due time. I mean, I guess on the mosaic side, I'm curious, like, you know, obviously you had an incredible journey as a standalone company and then ultimately, you know, chose to combine forces with Databricks. You know, why do you think it made sense for Mosaic to come together with Databricks? How'd that all come about?

• 5:41Speaker 01

  I mean, do the math. Mosaic had an awesome AI platform and AI makes no sense without data. Databricks had an awesome data platform and was making some really great progress on AI and we could accelerate that journey. It just made too much sense. It wasn't something, quite frankly, we were thinking about. It wasn't something we wanted to do. I mean, you know, you want to stay independent. Things were going really well and continue to be going even better, but it just made too much sense. And we kind of, I think both sides begrudgingly came to the conclusion that this was great for everybody. And I don't think any, you don't want to, you know, acquire a company. You don't want to get acquired. You want everything to just work organically within your organization and everything to be sunshine and rainbows. But we just realized like, wait, this makes too much sense. Okay,

• 6:25Speaker 01

  fine, we'll do this. There was also just good alignment on the teams. Like, their leadership is a bunch of academics and scientists. Our leadership, a bunch of academics and scientists. You know, all of our founders, PhDs. All their founders, for the most part, PhDs, professors. Like, it just, we understood each other really well and, you know, it wasn't Something anybody was thinking about, but after that initial meeting between Ali and Naveen.

• 6:46Jacob Effron?

  Is it true that happened at the Central Valley Conference?

• 6:48Speaker 01

  It did. Yeah, that actually did happen. I wasn't there, but I can vouch for it because, you know.

• 6:52Jacob Effron?

  I remember seeing them talking at that conference and I felt like it's where you get to witness, you know, a moment of history or something, but I always wondered if that was actually true.

• 6:59Speaker 01

  Yeah, no, it's true. I heard about it soon thereafter and it was just like, well, okay. Like, I don't know who these Databricks people are, but I, you know, I know Mate and revere Mate. And, you know, I'm excited to learn about the other folks and, you know, it's, it's kind of been a dream come true in that sense. Like, they are these legendary people, both, you know, in the startup world, but especially in the academic world.

• 7:18Jacob Effron?

  Well, I guess obviously a lot of what you, you know, did at Mosaic and now do as part of Databricks is kind of helping folks think through, you know, what makes sense to do around their own models and whether that's kind of, you know, a massive pre-training or, you know, fine tuning or kind of just taking models off the shelf as is. I'm curious, like, you know, the kind of basic guidance you're using today when you talk to folks about how they should think through the decisions that they have.

• 7:38Speaker 01

  Yeah, I think, you know, the first place I start is kind of, it's hard to know from first principles what to do and, you know, where you're going to land on this journey or kind of where your journey is going to end. Will it end with doing a little bit of prompting of, you know, an OpenAI model? Will it end with pre-training your own model from scratch? Like, it's, it's not obvious at the beginning of the journey. And so the first thing is kind of keep your options open. I think that's one of the most important things we've worked to provide at Databricks is just, let's make sure nobody has to compromise if they need to go on the whole journey. We can pre-train a model from scratch. We don't do it very often. I try to scare people off from doing this because it is, it's an undertaking. It's expensive. It's a lot of work. But for those who need it, you can't really do it anywhere else to this day and I'm really proud of that.

• 8:21Speaker 01

  But the journey starts at just prompting a model. The journey starts at spending 20 cents on OpenAI or on Lama on Databricks or, you know, what have you. And just kind of like litmus testing, is AI going to be good at this? It's really hard to know. We don't have good predictability as to like, is AI going to be really good for this use case? Is it not? And the answer I tell people is like, you're a scientist. This is data science in, you know, in the literal sense. Go run an experiment and try it and try to set things up. Give yourself the best chance of success. Try it on the best possible model you can get your hands on and just prompt it. Or don't even do rag. Pull in a few documents you know are going to be helpful into the context manually and just see what happens. Like, test the waters. The recommendations I make to people are always start small and work your way up.

• 9:03Speaker 01

  And kind of, you know, working your way up should be justified by rigorous ROI. Like, are you making progress on things that matter to you? You know, testing it, build some benchmarks, test it on those benchmarks, realize your benchmarks suck and build better ones at some point. And there's a cycle I'll mention in a moment that I see all the time. And then, you know, from there, you, you know, see if there's a there there. And then work your way up and realize, okay, maybe it's time for some hardcore rack. Like, I need to bring my data to bear. The model is not going to just have telepathy and know about my internal enterprise data. And if it does know about my internal enterprise data, you better look at your security, quite frankly. And so bring your data to bear. It's what we like to call data intelligence around Databricks. It's kind of like, of course, like, you know, a generic model is not going to know about your data.

• 9:48Speaker 01

  You have to bring it to bear somehow. And then from there, we can talk about fine tuning. If you're getting value, fine tuning is a way to bake a lot of this into the model. And that's gonna, you know, it'll be a little more upfront cost, but you'll get better quality in a smaller package. It's a way to reduce cost or get more. And, you know, then from there, we can talk about continued pre-training. And then, you know, maybe, maybe, maybe at some point in the future, we can talk about pre-training. Again, not for the faint of heart. Yeah. But for a lot of our customers, it's made a huge difference. But the one, one other thing I want to mention is kind of, I see this mistake all the time. People think, well, I can't do AI until my data's perfect. Or I can't do AI until I have the perfect evaluation. And the measure of your data is can you build a useful AI system? The measure of your evaluation is, did the system you build actually work for some human being

• 10:33Speaker 01

  in your real scenario? And so the only way that you can figure out how to make your data great, or make your eval great, or make your model great, is to just test it against the real world. So don't go and make your data perfect, because you don't even know what that means until you've gotten it into a system. Like, be agile and do a little work in your data, just enough that you can get it to interact with the model. Build the crappiest model you can, build a really quick evaluation, and test it. And then go back and figure out, is it the model? Is it the data? Is the task just impossible based on the data and the model I have? Am I expecting too much from AI these days? And then iterate on that cycle. But it's really like you don't try to get your data perfect in advance. It's all kind of one process.

• 11:11Jacob Effron?

  You mentioned, I guess, this kind of classic thing where you start with some sort of eval benchmarks. And then you kind of quickly realize, okay, like we need better ones. Any tips or like, you know, things you've seen around kind of getting better evals out the gate or better setting that up?

• 11:23Speaker 01

  So I think the first part is knowing it's not going to be perfect. And knowing that any eval you have is going to be a proxy for the real world. Identifying what the real world is and being able to make contact with that. Even if it's like one friend you have at the company who is not involved in the project, who can be the human tester for this process. Just having that is closer to the real world than any synthetic benchmark you're going to make. And, you know, I actually do this. I will make my team sit down from time to time and we will do A B testing of models. Or we will do A B testing of the outputs of our image generation model, for example, to look for certain properties the team cares about. We'll do this for natural language outputs. We'll do literally the RLHF thing where you look at pairwise outputs and pick the best one. And we won't tell you whether it came from our model or whether it came from Llama

• 12:05Speaker 01

  or this version of Llama we created or, you know, OpenAI or what have you. It's not fun. I have to really incentivize them. It takes a lot of cake.

• 12:13Jacob Effron?

  A lot of swords?

• 12:14Speaker 01

  Swords not, my team doesn't get swords. Swords go to the people who provide service to our team. My team gets other incentives, such as my hair. There's a system here. There's a, there's a whole process. I'm going to write a book on management someday. No swords for your team, only swords for people who help your team. I'll get them cookies. I'll get them cake. I'll get them like Chipotle, whatever. Chipotle, also Databricks customer. We love them.

• 12:35Jacob Effron?

  Really? I didn't know that.

• 12:35Speaker 01

  And I'm a big fan.

• 12:37Jacob Effron?

  Do they have any AI models yet?

• 12:38Speaker 01

  Yeah, I don't want to talk too much about what our customers do. But I will say as a very loyal Chipotle fan, I'm loyal to three things in life. Other than Databricks, of course. One is New York. One is New York bagels and the other is Chipotle, burritos, you know?

• 12:50Jacob Effron?

  That's a great, that's a great combination.

• 12:52Speaker 01

  Yeah, I'm a, I'm a very simple man with simple tastes. But, you know, where was I even? But yeah, you've got to have, you know, somebody to test this against. But when you build the eval, start really simple. Write five examples. And, you know, those examples don't even have to have right and wrong answers. Try taking an input and writing a few responses and just saying, these are good responses. These are like, these are five out of five. These are three out of five. These are one out of five. With that alone, you can calibrate an LLM judge and you can have an LLM judge decide like, is this more of a three out of five answer, a five out of five answer? Why? You can kind of inspect. Five examples is enough to just like get you off the ground. But I will say beyond that, we have a product that we've just released that helps you build this.

• 13:37Speaker 01

  This is actually a key focus for me personally, because until you have a measuring stick, anything else you do is kind of, you know, it's, you're just making things up. And so we're working on a product to help people have measuring sticks. It's a way for you to create your own evaluation data. It's a set of automated tools that help you along the way. I'm not a big believer that you'll just push a button and synthetic data will get generated that will just evaluate your system. I think that's, you know, we're, we're over promising if we say that. But the idea is like, human time is really valuable. I want someone to be able to, in an afternoon, create a meaningful eval set of a few dozen examples that they believe in enough that they can measure their model. And I think we're honestly there right now with the product. So if you want to check out our, you know, our agent platform and our agent eval product, they're there. I believe going into public preview next week.

• 14:22Jacob Effron?

  Oh, nice.

• 14:22Speaker 01

  By the time this podcast airs, it's in public preview. So, you know, hit me up. We'd love to get you on it, but it's something my team is using now. And I think the highest endorsement I have of Databricks' products across the board is my team uses all of them.

• 14:35Jacob Effron?

  Yeah, I mean, I obviously used a lot of the core Mosaic tooling to build your own model.

• 14:39Speaker 01

  We did, but we also, you know, I look at the DBRX journey, our, you know, our last language model that we built, that we released publicly at least. You know, how did we ingest the data? We used Spark. We did ETL on that data. How do we store it with Delta tables? How do we keep track of it? Unity Catalog. Unity Catalog was actually incredibly useful. Like, I can list off the Databricks products, but all of these things saved us time. Like, we, my team had been resisting Spark for months. I was trying to like quietly get them to start thinking about it.

• 15:05Jacob Effron?

  That feels tough to do within the company.

• 15:07Speaker 01

  It was like, I mean, well, this was pre-acquisition. I'd been trying to tell my team, like, maybe rolling our own stuff on these giant 128 core CPU machines is not a modern way to do data processing. Got into Spark and like jobs that took us a week were like minutes. It was kind of, you know, honestly embarrassing for how we had been doing work previously. Unity Catalog, suddenly I found there were like 20 copies of the same data sets everywhere. And it was Incredibly helpful for me to keep tabs on what my team was doing. Then, of course, we used the Mosaic tools to train the model. We used MLflow as our experiment tracker. MLflow is phenomenal and it's free. So we've been working with the team really closely. We're putting in feature requests every day around our workflow and they're getting out like a week later. So it's been great on that front. We're using the Mosaic inference service.

• 15:46Speaker 01

  So it's kind of, you know, across the board, it's just like, we work with our product teams to build products that we want to use.

• 15:53Jacob Effron?

  I mean, I guess, you know, on the, uh, these kind of, you know, when eventually folks do get to, you know, doing their own large scale pre-training, like, have you seen examples of that that have made sense? I mean, I feel like there's, there's this question out there about like, when kind of domain or company-specific models make sense versus just, you ride the trends of the larger models getting better and better and it kind of subsumes whatever advances you've made on your own specific data. What have you kind of seen of when the company-specific data or, you know, domain-specific models make sense?

• 16:18Speaker 01

  There are a lot of cases where this makes sense. One is just the model, you know, isn't that good at your task? If you chat with a lot of our customers in, you know, Japan or Korea, they'll tell you that these models really haven't been tuned for this. A lot of people blame the tokenizer. I think it's just, you know, there's less data available in those languages than there is in English and these models are really data hungry. You train on what you have. You focus on the things that are most important to your target audience. And I think for a lot of the companies building open source models, including, you know, the, the DBRX model we built, you know, English is the first language. It's what you focus on. It's what your team speaks, so it's what you can curate. But that's leaving a lot of our really important friends in India, Japan, Korea, kind of high and dry. And they're focused a lot on building their own models because they have to.

• 17:00Speaker 01

  And, you know, that's where we've done a lot of really good work and I'm excited about what our partners have created. You know, a lot of folks, NTT has released some amazing models that they've talked about publicly. Our friends at Ola have released some really awesome models for India. Like, it's, it's impressive. Scenario number two is just, you're literally doing a whole different thing. You know, our friends who are doing protein models.

• 17:18Jacob Effron?

  Yeah.

• 17:19Speaker 01

  Go ahead and, you know, try to, try to get Lama to do protein modeling. Okay, fine. Number three is, you know, cases where You just need a really, really fast model that is really specific. Our friends at Replit fall into that category. We worked together and basically built like a Lamborghini of models. It was just like every detail was built for speed and built just for their task because cost was an issue. They wanted to release code completion to all of their users, including their free tier users. And the model had to be really good and it had to be really fast. And you know, Databricks has a lot of code completion tools. That constraint is not unique to our external customers. That's what I'll say. And the fourth scenario is it's honestly just a cost thing. So it's expensive to pre-train a model, sure.

• 18:02Speaker 01

  But if you have a model that's getting a ton of use by pre-training your own model, you're going to get a really good model. But I think of this as a cost quality trade-off. You can either think of it as you're going to get the same quality model in a way cheaper package for inference, or you're going to get a way better model in the same package. Take your pick. It kind of works either way. But if you care about just maxing out quality for a budget, fine, you can do that. If you care about minimizing your budget for a given level of quality, fine, you can do that. But either way, there's just a point where the upfront investment pays for itself very quickly. This is kind of how I describe it to people. If you switch from, say, prompting to fine-tuning, you're making a slightly bigger upfront investment, but you are going to improve that cost quality trade-off. And that upfront investment, if you have enough use, will pay for itself really quickly.

• 18:47Speaker 01

  Continued pre-training, takes a little more use to pay for it, but for a lot of folks, they're getting that use. And then full-on pre-training, same thing. It's just kind of, you're literally moving the cost quality curve up. It just takes a little upfront investment to get there. First, justify the ROI. Like, make sure this is going to make sense for you. Don't, you know, jump right into this. But as you go on that journey from prompting to fine-tuning to continued pre-training to pre-training, you'll know if it makes sense because you're getting usage to justify the fact that you'd rather have a better kind of cost quality trade-off.

• 19:13Jacob Effron?

  Right. And I imagine one of the key assumptions is just how often you have to, you know, pre-train again or fine-tune again.

• 19:19Speaker 01

  Sort of. I think once you've got a really specific application where you're seeing use, I don't see people needing to like go back to the drawing board that often.

• 19:27Jacob Effron?

  Yeah.

• 19:27Speaker 01

  It's kind of, you know, once you have product market fit, you have product market fit. And now it's a matter of, you know, like optimizing for both quality and cost. And that's where you start to move up this, this chain over time.

• 19:37Jacob Effron?

  If you were to think of the broad categories today that you'd say have product market fit, like just in broader AI, like what are, what are the areas that come to mind for you?

• 19:43Speaker 01

  I kind of, I think of it as two like patterns and you know, those patterns you can apply to a lot of different places. One is cases where you just don't have to be right, where you know, you're brainstorming, where there are a lot of right answers. You know, you can think of this as, you know, even like a situation like Glean is a great example, actually. You're just helping surface information for people. It doesn't have to be perfect. You just have to help people be a little more productive. This is in, you know, creative applications, marketing applications, media where there's no right answer. The other scenario is cases where, you know, the AI doesn't have to be right because it's going to get checked. And specifically, it's got to be scenarios where it's relatively costly for a human to produce a proposed answer to the problem, but it's relatively quick for a human to check that answer.

• 20:27Speaker 01

  So a case where this might be really risky. Let's suppose you're dealing with a hundred page legal opinion and the details matter. And the model produces a proposed answer. A lawyer can check that proposed answer, but the devil is in the details.

• 20:40Jacob Effron?

  Kind of to recreate the work just to check it.

• 20:42Speaker 01

  Yeah, it's kind of, this is kind of a P versus NP kind of thing. You want, you know, NP is a class of problems that are, you know, polynomial time checkable. You know, P is a class of, you know, problems at a polynomial time solvable. And I want things that are at least in NP. Like, it doesn't have to be, you know, easy to get to a solution, but it does have to be easy to check that solution. And there are tons of scenarios like this. Co-pilots are a great example of this. Because, you know, all you do is tab complete or not tab complete. It's, you know, it's a lot harder for a human to come up with code than it is for them to look at code and say, nope, that's not what I was about to type. So in some sense, like, you know, GitHub Copilot, awesome PMF there. So, you know, I've mentioned two companies right, right off the bat that I really respect for the work they've done and think, you know, they've kind of captured those two use cases pretty well.

• 21:25Jacob Effron?

  Yeah, no, it's super interesting. I feel like coding obviously seems to have great fit. Support seems like another example where relatively easy to, to, to check some of the stuff. And then, you know, your Glean example is interesting too, where it's not, you know, existential if you miss something. Whereas, you know, you have similar kind of semantic search problems in kind of legal or other spaces. And it's like, you better be comprehensive. Missing one link, you know, in your internal company search is okay. In like a search for precedents where one precedent is the one that really matters is, is not.

• 21:52Speaker 01

  So I, the way I think about it is it's not really like an industry. It's use case by use case and it's honestly, you know, how comfortable you feel depending on like, you know, your customers better than anybody else. You know, whether these scenarios will make sense, whether being a little bit wrong is okay sometimes. AI is fuzzy. It's its superpower and it's Achilles heel. And I think that's kind of a fundamental property of where we're at with the technology right now. And that's okay. Like it's unlocked tons and tons of value. You don't have to write a regular expression anymore to try to parse a document. AI can just do that. It's incredibly powerful. The downside is, you know, if you can't make your intentions as crystal clear as you can encode, you know, you're not always going to get exactly what you expected.

• 22:32Jacob Effron?

  Yeah. What do you think? I mean, obviously there's some companies that are, you know, focused kind of in the agent space and you'll be able to customer support, for example. And you know, I think that the, it seems like the main focus right now is getting around this, uh, you know, this like fuzziness of the answers, right? And chaining together like a ton of models that are checking your initial models and finding ways to basically, with much more fidelity, say, we're confident that your, our customer support model is going to output, you know, uh, something that's pretty damn near what your people would do. Are you kind of bullish on that long-term with the current generation of models or?

• 23:01Speaker 01

  I'm definitely bullish on that. I think it's really, I look at this as more of walking the cost quality trade-off. Like in some sense, there's a model, like you take one monolithic model with a single model call. There's only so much quality you're going to be able to get out of that. So you pay more by running more tokens through the model and running it through in creative ways. And you get more quality. It's kind of pushing us further along the cost quality curve than we can get in a single monolithic model. So, you know, I look at this as just inputs and outputs and, you know, same thing, we're getting better quality for a given cost. And as cost comes down in tokens, things that might not have made sense before make a lot of sense now. So I'm very bullish on this. Now, I'm not bullish on that getting to perfect answers. I still think you have the fundamental property that the system is fuzzy. We're pushing to higher quality, but pushing to perfection.

• 23:47Speaker 01

  Or pushing to kind of, you know, getting that, you know, that fifth nine or what have you of quality. I think that's going to be hard with the current generation of the technology and we should be realistic about that. That's okay. That's not, that doesn't mean, you know, we're all doomed and, you know, all the stock prices for all the hardware companies are going to zero. No, this is incredibly valuable technology. We're just still learning what its strengths and weaknesses are and how to make the most of its strengths and cover for its weaknesses, just like we had to do with software engineering. We had to learn how to, you know, build structured programs. We had to learn about security. We had to learn where vulnerabilities were. We had to learn how to build, you know, million or tens of millions or hundreds of millions of line code bases and manage that among thousands or tens of thousands of engineers. And that wasn't easy. That was a decades long process of learning to work with this medium.

• 24:31Speaker 01

  And that's a medium that in many ways is simpler and easier to understand than AI. It's going to be a journey for us and that's okay. It doesn't mean people weren't getting value out of digital technology until we had object oriented programming. We were getting a lot of value back to the 50s and we're getting that value today. But, you know, Tik Tok was impossible to contemplate in 1950 because of how much changed and it took a lot of getting better at building these systems. So even if you freeze the technology today and just say GPT four, that's all you've got for the next 20 years, we're going to see enormous creativity and mind blowing stuff happen just on the basis of us learning how to use this tool better.

• 25:06Jacob Effron?

  No, it's super interesting. Also, I think you have to figure out kind of what we are comfortable with from a, you know, from a quality standpoint across different use cases. Like I always think of like healthcare where we've had models that are better than doctors for quite some period of time. And it feels like the, you know, in radiology and in other places. And the question is always, well, okay, how much, you know, when it's my doctor making a mistake, that's one thing. When it's the AI doctor making a mistake, you know, that's a totally a different thing. And it's almost like, what is the, the point at which it becomes almost unethical not to use the model versus?

• 25:33Speaker 01

  I think it's more complicated than that. I think, I think that glosses over, you know, a bunch of things. One is that like with humans, I think we have good intuitions as to when humans fail. Like even with autonomous vehicles, we have really good intuitions about when a human is going to be a worse driver. And not only does that, you know, it gives us better uncertainty bars, we know how to interact with human drivers. It also just helps us rationalize the mistakes and come to terms with them.

• 25:57Jacob Effron?

  Yeah.

• 25:57Speaker 01

  Just as human beings, it helps us cope with the world that we, we have predictability over that. Like we, we don't just know when doctors are going to be right and wrong. We have uncertainty bounds. With AI right now, we don't have that. And I think, you know, that's something. For us as humanity, either we're going to have to start to build those intuitions over time, and that will take time, because as a scientist, I don't have good intuitions for that right now, let alone as a person interacting with these systems in the world. But it's kind of inexplicable and unpredictable. Look at some of the autonomous vehicle failure cases. They are in situations where a human driver would probably have struggled, but sometimes they're also thinking of us as a cloud. And that's hard to predict. So I think we're more comfortable with having models of uncertainty, and we can just make

• 26:41Speaker 01

  peace with that as human beings when things go wrong or when tragic things happen. So again, we're getting into philosophy, but my take is kind of, we're going to have to wrap our heads around this. And I mean, the other piece that I keep coming back to is by holding AI to these kinds of rigorous standards, people say things like, well, humans aren't that good. Maybe we should hold humans to higher standards, too. Just drawing back on my time in the policy world, where I was looking at law enforcement use of facial recognition. We held facial recognition systems, or automated ones, to very high standards. Turns out, humans are really bad at this. And having a human check this is actually not great. There's a lot of work on super recognizers who are very good at

• 27:24Speaker 01

  this, and how to distinguish super recognizers. There's a lot of work on how to train people to do better facial recognition. There's a lot of work on just when humans are most fallible in facial recognition. It's people who don't look like us, which that's not racism at work. There are biological processes at play in your brain where you see a bunch of people who look like you during formative periods, and people believe that that's what makes you better at recognizing people who look like you. So we raised our standards for humans, or at least I kind of rethought the way that I thought about the whole system of how we recognize whether two people are the same person, or two images are of the same person, due to having this automated system that was uninterpretable. And I think

• 28:06Speaker 01

  the same thing is probably going to happen around autonomous vehicles. We're holding these vehicles to very high standards. Why does the driver's test look the way it does? Should we not be tracking a human in their car with all these sensors for a set amount of time while they're driving and kind of look at what their performance is over time? That seems to make a lot more sense to me. I remember right after I got my driver's license, the state of Maryland moved from this little closed course to like actually having you get out on the road for a little bit with, you know, the person who was testing you. That seems like a step in the right direction. So, you know, again, I don't want to oversimplify here and say, you know, it's, you know, AI systems are better than humans, so maybe we're irrational. It's, I think there's, there's method to the madness on both fronts, but it's going to lead to better human performance as well, and I think that's really exciting.

• 28:49Jacob Effron?

  That's super interesting. I think the point particularly around, you know, the lack of explainability on where these things go wrong, really, you know, being something that's hard to, to make peace with or live with, I think is, is really interesting. I haven't thought about it before, but makes a, makes a ton of sense. On the Databricks side, like, how do you think about, you know, obviously you guys have this end-to-end platform. I feel like you're, you're kind of going head-to-head with some of the offerings from hyperscalers, from Snowflake, like, how do you think about how this whole space plays out and where you guys fit in?

• 29:13Speaker 01

  So, I don't really, you know, the way that I look at it is I like to speak for Databricks and, you know, you're welcome to invite folks from other companies to speak for, you know, for their companies, but I'm much more focused on my customers than on anybody else.

• 29:24Jacob Effron?

  Yeah.

• 29:25Speaker 01

  And what I'm seeing from my customers is, you know, they're just trying to get traction. They're just trying to get from zero to one in this space and make sense of not only how to figure out how to get AI to work on their problem, but how to get that into production in any meaningful way. And so, our goal is just make sure we have all the tools available and they work really well together, and we have journeys that we figured out, and we can work with them on those journeys. We work hand-in-hand with them. I work hand-in-hand with customers. As a researcher who, you know, in some of my former research lives, I would have been segregated off, like, you know, many degrees away from customers. I'm talking to our friends at these companies all the time and looking at what they're going through, because I don't think anybody knows what the right product is yet. So having the right collection of pieces, making sure they fit together really well,

• 30:10Speaker 01

  all in one platform, all under one roof. So you don't have to worry about data transiting in and out of different places or cobbling together five different tools. And then going on these journeys with folks and trying to figure out what the like, the well-trod paths are and making those paths really great.

• 30:24Jacob Effron?

  When you say nobody knows like what the right product is yet, what do you feel feel like are some of the big unanswered questions about like what these offerings should look like long term?

• 30:30Speaker 01

  Oh, everything. I don't think we, I don't think we know how to help someone build a way to measure their product yet. I think, you know, at Databricks, we've made a really good dent in that. And there are also a bunch of startups who have done a phenomenal job that I really respect in this space. And, you know, I think they're doing a great job. And I hear my customers saying that. I don't think we've figured out how to help people like, you know, navigate the space of RAG versus prompting versus fine tuning yet. I don't think we know the answer as to what works in what cases. And a lot of my thinking these days is how do we help customers do the all of the above solution and then pick the best thing? It's not automl, but it's kind of helping people realize that every time you train a model, there's enough there that we could actually create 20 models, not out of, you know, doing any more work, but there are a lot

• 31:12Speaker 01

  of ways to mix and match models and mix and match different approaches to RAG. I was chatting with my fine tuning team yesterday and I said, actually, I think that in one or two fine tuning runs, we could get you 200 models. And from there, and 200 diverse models that are going to have different properties, because I don't know which learning rate is going to work perfectly for you. I don't know which prompting approach or which sampling approach is going to work well for you. And I don't think I have a principled way to say for this industry or this task, this is going to be the one. So the answer is helping people figure out how to measure and then helping people get a lot of options. And there's going to be one good one. Even with my team, we have to do this a bunch to figure out where the right one is. And I don't think it's consistent necessarily, or at least in a predictable way. So for me right now, it's really about

• 31:53Speaker 01

  like helping people have as much choice as possible with as little cost as possible, and then helping them choose well.

• 31:59Jacob Effron?

  Yeah.

• 32:00Speaker 01

  And then helping them get it into production and feel comfortable that it's running in production. Those are kind of all places where I think we have a lot of work to do as a field.

• 32:07Jacob Effron?

  I mean, part of what I'm struck by is you're obviously building this like incredibly powerful end-to-end platform incorporating, you know, everything from the data ingestion side, all the way through, you know, to evals now. And I'm curious because obviously there's a lot of startups tackling AI infrastructure in different ways and sometimes I think one could look at it and be like, okay, well, that's a part of what we'll ultimately live in some end-to-end offering here. And I think even these startups would like to think, hey, whatever we're doing here is like some wedge to then build out like a broader set of things. So as you look at the infrastructure landscape, do you feel like there's place for some of these standalone like eval companies or data injection companies or some of the other folks that are inference providers or all the other categories? Or does this all just converge on like end-to-end platforms long-term?

• 32:46Speaker 01

  I think the answer is yes. I mean, you can look at any other part of the software space and the question is like, is there an end-to-end platform? Or are there also a bunch of amazing point solutions? The answer is yes. Databricks is not a self-contained universe. We partner with anyone and everyone and you can go and look at our partner list. It includes a lot of these amazing startups because quite frankly, they're building phenomenal tools. And I want my customers to have access to them. The wonderful thing about being at Databricks has been so far that it's been yes and every time I've said, well, I would love it if my customer used Databricks for this, but also can they use this other amazing tool for data annotation, eval creation, data exploration? And the answer has been, of course, let's make them a partner, integrate them really well so that it's a great experience

• 33:31Speaker 01

  for our customers, and let's have a win-win. We want to build an amazing image generation model. Well, who has the best library of images in the world? Shutterstock. Let's call them up and build something together. It doesn't have to be this story of it's this or that. I don't think that's how the world works. So my view is like, let's make Databricks amazing for the things we're great at. Let's find the best stuff out there and make it as easy as possible for our customers to use it.

• 33:57Jacob Effron?

  How do you think about the acquisition side of things? Obviously, you guys have Mosaic being the first, then you've bought some other things too. Where does that kind of fit into the platform strategy?

• 34:06Speaker 01

  I mean, when something is amazing, I want it to be part of Databricks. And there are two ways to do that. Either we can partner or we can acquire. And I think looking from the other side now, we try to figure out where it makes sense. And that can mean makes sense for us, makes sense for the partner, what have you. But there's one acquisition I participated in pretty closely, which was an acquisition of a company called Lilac this spring. And why did we acquire Lilac? Well, we used their stuff on DBRX and we loved it and we wanted to partner with them. And I went to our leadership and said, this company is amazing. Can we keep it? And you know, we worked with them for a little while and figured out like how to make it make sense for both sides. And I'm glad we came to a place where that team is now

• 34:50Speaker 01

  part of our team and we're building some incredible stuff together, not just Lilac, but a bunch of other amazing products. That eval creation product that I just mentioned, it's that team because they, they have an amazing product sense for what people need and they built a beautiful, beautiful product that we loved, the open source community loved, a bunch of our partners loved. And you know, now they're part of the Databricks family, same as Mosaic got to join the family, you know, a little over a year ago. And so if you know, from my perspective, it doesn't really matter whether you know, we acquire them, whether we partner with them, whether you know, we just make a recommendation because we want our customers to be successful.

• 35:24Jacob Effron?

  With DBRX, I mean, you mentioned earlier, obviously you've got Zuck spending a ton of money on Llama models. How do you think about like the future of that and kind of, you know, the continued investment there? And you know, as you know, people are spending more and more compute, like what makes sense for, for your team?

• 35:38Speaker 01

  I think it's a matter of just figuring out where the gaps are and what our customers need. The Llama models are phenomenal. I mean, I don't think I could build something better than that with, you know, the, the position I'm in, you know, in the situation I'm in. Meta has amazing resources to bring to bear, amazing talent to bring to bear. We do too, but their resources far outstrip ours. And this is a game of incredible investment. You can look at the amounts. I have my own kind of back calculations of what Meta probably spent. And I have, I'm pretty sure they're, you know, they're in the right ballpark. And those are amounts that quite frankly, it doesn't make sense for Databricks to spend when our friends at Meta are doing an amazing job and giving this just gift to the ecosystem. I, you know, I'm really grateful for that. But not just Meta, our friends at the Allen

• 36:20Speaker 01

  Institute for AI are building these models that are kind of the paragon of openness. They're open about their data, they're open about their process and their code and everything. They're releasing intermediate checkpoints. Those models are phenomenal, too. I think the choice in the open source model space is so incredible that there are other things we should be focused on right now. If we want to make our customers successful, there are other gaps in the space. And so, you know, I'm spending a lot of time looking at like personally right now, first of all, the eval creation work. Like that's really near and dear to my heart because that's the beginning of the journey for everyone. Without that, Really hard to get anywhere. But also, how do we start to kind of, I think I mentioned before, navigate this space of fine tuning, rag, like, our team is looking at that really closely, and trying to have good answers, even if the answer is taken all of the above approach,

• 37:05Speaker 01

  but make it so you're not overwhelmed by choice. I think that's kind of an underappreciated part of the space. And then there's meeting customers where they are in terms of what data they bring to bear. Sometimes the data hasn't been cleaned and is very difficult to clean. Sometimes they have a lot of inputs, but not a lot of outputs, or a lot of like example outputs, but not a lot of inputs, or a lot of documents, or a lot of code. And we need to help them build an AI system that solves a specific problem on the basis of what they have. And so it requires a very creative approach to how we take that data and connect it to a process by which we can build an AI system. We're thinking a lot about compound AI systems and agents for kind of connecting the dots between lots of different pieces, including Databricks pieces. So that's where I see the gaps

• 37:45Speaker 01

  right now. It's kind of in some sense, that last mile of, we have amazing, you know, amazing substrates. How do we actually make that useful to customers? How do we turn that raw material into something that people can really build systems out of? And personally, when I look at the field right now, I think the open source model world is exceedingly well covered.

• 38:03Jacob Effron?

  Yeah.

• 38:04Speaker 01

  But there's a lot of room in these places. And I'm going to go where my customers need me, and I think they need me here right now.

• 38:09Jacob Effron?

  Yeah, no, that really, that really resonates. I guess switching gears to kind of the broader AI research landscape, I'm curious, like, if there's an example of something you've changed your mind on in the AI world in the last year.

• 38:19Speaker 01

  In the last year, I mean, it's, I, the way I look at it is kind of, I try to keep an open mind at this point. I think I'm wrong more than I'm right. And when I try to predict the future. So, you know, I know that you're planning to ask me for some predictions about the future.

• 38:32Jacob Effron?

  Wouldn't be a good podcast outside it.

• 38:34Speaker 01

  And I'm going to refuse to answer those questions, quite frankly, because not only, I think the future is hard to predict. And as a scientist, I don't think, you know, it's in my profession to try to predict the future and how technology develops. But also, I'm wrong a lot. My hypotheses were wrong a lot during my PhD. You know, my initial dissertation work, this, you know, it was called the lottery ticket hypothesis. There were a bunch of other hypotheses that predated that one that I was really wrong about. And that one, you know, thankfully happened to work and, you know, show us a lot of really interesting science and helped me build my career. You know, I remember GPT-3 coming out and I was like, eh, you know, they made GPT-2 bigger. You know, so it's going to generate mediocre paragraphs slightly better than GPT-2 did. What's the point? There was a bigger picture that I, that I don't think I fully appreciated at the time, and I'm happy to admit that.

• 39:20Speaker 01

  I think a lot of other folks would say the same thing if you, you know, maybe didn't get them recorded on something that'll be visible on the internet. But I think a lot of us would say, we didn't see where that was going. And I don't think we could have predicted where that was going, even if we fully appreciated it. I think it went somewhere a little unexpected. It worked. Very few things work in this field. I advise PhD students, if your project you're working on is asking the scientific question, I wonder if blank will work? 99% of the time, the answer is no. And you're not going to get a paper. And so I coach students on a way to do experiments where you'll always get an answer that will create scientific knowledge. Scaling worked. It worked better than I expected. And if you were to look at a lot of other trends that people were predicting, a lot of them didn't work.

• 40:05Speaker 01

  But, you know, it's the things that work that we all look back on and wish we'd been right about. I was wrong. So I'm probably wrong about a bunch of things right now. And that's okay. Maybe, you know, I've just said a bunch of things about how these systems are fuzzy and, you know, I don't know if we're going to get that fifth nine of reliability. Maybe we will. Right now, I don't know if scaling is really serving us that well. I don't know if we're scaling to the next, the next place. Yeah. I think there's some doubt right now in the community. Maybe I'm going to be wrong again. You know, GPT-3.5 to GPT-4? Okay, that one I bought into, like, you know, it's like, okay, you know, you've convinced me, you know, like, Text Avicii 003 was already very impressive, putting into the chat GPT product. And then, you know, from then on, awesome. But, you know, who knows? Like, it's really...

• 40:50Speaker 01

  I think we should keep an open mind, and I try my best to. So, you know, it's hard to be surprised if you don't go in with too many preconceived notions, but a lot impresses me. And, you know, I think everything in this field impresses me these days. It's just, it's fun to, you know, experience the joy of seeing technology develop in real time, seeing that it's still a world where students can come in and make a huge impact on the field in real time. And, you know, we haven't reached the point where you need to build a Large Hadron Collider to do important work and have 8,000 author paper, even if we're seeing plenty of that. And that, to me, is just kind of, I love sharing that joy with students.

• 41:24Jacob Effron?

  I love that. Well, I'll try to, I'll try to keep it grounded in the present then and not too, too, too, too in the future. Because if you can't predict where scaling laws are going, I don't know what hope us mere mortals have. But I guess, you know, for some reason stuff has happened. I'm curious for your thoughts on, obviously, you've got the new O1 model from OpenAI. Curious your reactions to that, to what extent you thought it was a breakthrough.

• 41:44Speaker 01

  I think it's exciting. You know, breakthroughs are only things we say with a lot of hindsight. Transformer looks like a breakthrough today.

• 41:53Speaker 01

  At the time, it was one among many architectures that people were looking at, and, you know, nobody remembers XLNet. Nobody remembers a lot of things that were floating around back then. So, you never know. GANs were the hot new thing, and diffusion models were kind of this weird curiosity until they weren't. But there have also been a lot of kind of false starts in various directions and all sorts of things like that. O1 is exciting to me. You know, there are a lot of questions. Is it going to scale? Is the kind of juice going to be worth the squeeze in some sense in terms of the cost that you have to pay? Is this going to generalize from kind of constrained and more mathematical encoding reasoning problems where you can automatically generate problems and automatically check them to more open-ended problems, the kind that I see with my customers?

• 42:34Speaker 01

  But it's an exciting direction. There's certainly been, like, it's not unique. It's unique in terms of the scale. And I think the impressive part, you know, for my friends at OpenAI continues to be that they take these ideas and they're able to scale them up. In some sense, it's You know, as much or even more in engineering achievement that they built that model than it is, you know, just a scientific achievement. But those ideas are kicking around and certainly, you know, we have our own elements of those ideas that are floating around Databricks right now that, you know, you may not see publicly, but are, you know, showing up in various ways in our products already. And, you know, those models are working well for some people. They're not working, you know, they're not a step function for other folks. And time will tell. This could be a GPT-3 moment where a bunch of people kind of, you know, look at this and go, oh, it's not that impressive.

• 43:19Speaker 01

  And, you know, haven't learned the lesson I've learned that sometimes these, you know, it's new ideas that can be the seed of really great things. And sometimes they're dead ends. We should keep an open mind, but, you know, I take my friends at OpenAI very seriously when they say something is important. And, you know, I'll take them seriously on this one until proven otherwise.

• 43:37Jacob Effron?

  Yeah. What do you do like on the day like O1's released? Like, how do you mess around with these models and start to figure out what they, that they kind of can't do besides just reading the evals?

• 43:44Speaker 01

  Call in favors and, you know, ask for access so we can tinker with them. We have our own internal eval suites that are kind of more enterprise centric and more relevant to our customers that allow us to kind of get a quick handle. And then we just kind of sit down as a team and ask ourselves like, how do they do this? We generally have a pretty good idea because I don't think there are that many mind blowing new ideas in this space. Once you see the result and see that it works, like it's a confirmation. That's the hardest thing to come by. There are plenty of ideas floating around, but we sit down and we ask, you know, A, what does this model mean for us? But B, what do these ideas mean for us? We've now gotten a demonstration that these ideas work at a certain scale that we didn't know before. How can we put these ideas into practice today for our customers? And, you know, we already have products that, you know, actually we were working on a lot of these ideas prior to O1. I think, you know, in some sense, there's nothing new under the sun, but OpenAI took

• 44:27Speaker 01

  it to an extreme level and was willing to bet on it. And that's the really truly impressive part in some sense. Great science and great engineering and the willingness to bet on big ideas. But, you know, it was kind of like, okay, let's, you know, these ideas have been demonstrated at scale. Let's get them into production. They're in production and our customers are already working with kind of a lot of the ideas that relate to O1 today. My team is like, has a bunch of stuff cooking and we're going to write about that shortly.

• 44:53Jacob Effron?

  Awesome. I'll look forward to reading. What about the stuff that Anthropic released around like kind of computer use, you know, obviously early still in that in that area. But what do you kind of think of that work and what's going on there?

• 45:04Speaker 01

  I think it's an important application. I mean, you can look at companies like UiPath, you know, the startup Adept AI that did some really impressive work here. You know, it's it's something that's clearly an important problem for folks. The question of, you know, whether it's the right time to address that problem, hard to tell. I mean, you know, is it the right time for VR? Is it the right time for AI? Like, these things have come and gone over the years and people have taken a new stab with a new generation of technology and a new outlook. And, you know, maybe it wasn't the right time two years ago and it is the right time now. Maybe not. But what I love about the folks at Anthropic is they're experimenting with new products. They're really creative on that front. We haven't, I like, you know, in some sense, we haven't had our, you know, Google moment or our, you

• 45:44Speaker 01

  know, Facebook moment for this generation of technology yet. And so I just love seeing the like, the garden of ideas that's blooming right now in terms of interesting products. And I love the fact that the folks at Anthropic specifically have been really creative about trying to get stuff out there. And not all of it's been a hit, but that's fine. That's like, we're in the phase of just trying things out. And I love that they're willing to take risks. And I'm learning a lot from that as well. Like, we're all trying things right now. It's cool to watch, you know, those really smart folks try things as well.

• 46:12Jacob Effron?

  Any other products you've seen that kind of have caught your eyes? Like, oh, that's a really different way of of doing things.

• 46:17Speaker 01

  I mean, there, there's a lot right now. It's kind of everything, to be honest. I mean, like, Glean is the kind of thing where I look at that and I'm like, wow, they had that idea before this generation of the technology. They were really ahead of the curve and I really respect that. I'm looking at a lot of companies right now that are building amazing application products that are kind of hybrid human AI with the idea that maybe the AI will take on more of the work and make this a lot more scalable over time. Kind of the original thesis for Uber, that, you know, you have a human driven process and eventually, you know, autonomous vehicles will, you know, reduce the need for human labor and make it a lot more scalable and a lot more cost effective than it was. And that thesis has taken a long time to play out and still hasn't really played out. But I think the idea is there with a bunch of other applications that are AI enabled today.

• 46:59Speaker 01

  So across the board, I'm just, I'm just having fun seeing new things. And tinkering with it, it's kind of it. I think we shouldn't just lose sight of how incredible it is to just be a part of this and watch it all happen. And it's, you know, it's a lot of the work, honestly, my customers are doing right now. That's got me most excited. You know, my friends at 12 Labs who are building this amazing video search tool. Like, I went to Korea about a month ago, basically to hang out with them, because I had never actually gotten to meet them in person. And they're doing phenomenal work. We've been, they're one of our oldest Mosaic customers. It's just, it's kind of exciting to be a part of this, you know, even like, there's a startup I'll highlight Osmo AI. They're doing AI for smell.

• 47:40Jacob Effron?

  That's cool.

• 47:41Speaker 01

  Which, you know, you'd think that sounds crazy. They just announced yesterday, three new scents that are kind of part of what they call a perfumer's palette. That's awesome. Can you imagine? Like, they've had to set up human data annotation for smell. They held a scent salon a couple of months ago where they invited, you know, a bunch of perfumers and a bunch of just friends in the AI community to come and try out a bunch of new smells that were built with the aid of AI. That's so cool. I wouldn't have thought of that. I don't have the expertise. It's, you know, really, we're creating new tools for, you know, truly creative people who have domain expertise and real problems they need to solve

• 48:21Speaker 01

  to do incredible things. And I'm just, I'm enjoying being along for the ride and having Databricks' 12,000 customers to get to watch, iterate on this.

• 48:27Jacob Effron?

  Yeah. Obviously, you know, another hat you wear is you still obviously work with PhDs and do a fair amount in the academic world. I'm curious like how you think about, you know, the role of academia and AI research today and kind of where it makes sense for, it sounds like, you know, obviously, you have some good thoughts on where it makes sense for academics to spend time and where maybe, you know, work is better done in kind of the large research labs with access to tons of compute.

• 48:49Speaker 01

  I think it's the usual thing in academia. When industry is zigging, you have to zag. I mean, you know, you're not going to compete with Google scale resources. Don't try. But you're able to ask hard questions that companies may not want to ask themselves publicly about their systems. You're able to build amazing benchmarks, amazing leaderboards that kind of force us to move the ball forward. And you're seeing that among a lot of folks. You're able to just take risks. New kinds of technology, new kinds of models, if you really want to go there. You're able to measure things. Just the mere act of measuring the world, that's astronomy and physics in a nutshell. And I still think there's a lot of work to do there. Work on data, work on product, work on HCI.

• 49:31Speaker 01

  I think human AI interaction is a space that like the HCI folks are the ones I'm thinking about paying big salaries to right now because I need that expertise on my team. So there's tons of work to do in academia. The important part is not looking to industry for leadership on precisely what to do. But chatting with folks in industry about problems. I have so many problems right now and I wish I had the time horizon to look at all of them and look at all of them rigorously, not just get to kind of an okay solution and then ship it. And, you know, my friends in academia can spend the time to do that. And in doing so, they can push the boundaries in a really meaningful way. So I still think academia has a very bright future in this space. But I think academics need to be careful about what problems they choose to work on even more so than usual.

• 50:14Jacob Effron?

  You kind of mentioned in the kind of of Osmo, like, you know, I guess they have their own interesting data labeling challenges on the on the small side. I think you've tweeted before about the data labeling market at large. It feels like, obviously, you know, it's exploded with with just the amount of of data folks need. What do you think like happened to that space long term? Is there a way for some of these folks to differentiate? What are you kind of seeing on the ground?

• 50:35Speaker 01

  It's it's honestly hard for me to say because, you know, I'm a, you know, I'm a consumer of this space, not someone who's in the space day to day. So I don't want to like, I don't want to speak out of turn for someone else's space and where someone else is the expert, not me. What I will say is like data annotation is really important for me, for my customers. There's a question as to whether you need external labelers or whether, you know, actually your best data annotators are in-house, you need a platform to help them. Turning human labor into high quality data annotations is incredibly hard. There's a ton of expertise there and it's not expertise that's widely known. It's certainly not expertise I have. If I had that expertise, maybe I'd do it myself, but I don't. That's why I pay someone else to do it in the same way that, you know, folks come to Databricks because they don't want to build their own data platform from scratch, nor

• 51:20Speaker 01

  should they. They'll work with us and we'll provide them with a great service there. So, you know, I'm really grateful to my partners in this space, Superannuate and Surge specifically are the two companies that I work with. You know, most closely and, you know, they've done incredible work. I think the one other thing I like to say about this space is it takes trust. Because there is so much work that needs to be done to turn raw data into high quality data. There's a lot of trust there in kind of the artisanal aspect of that process. And there's a lot of trust that, you know, when you're paying for human annotation, you're getting human annotation. And that takes, you know, not only a lot of trust in the company, but a lot of trust in their supply chain. And how they're working with and incentivizing their annotators to do that. It's not an easy thing. I have such respect for the people who have built really good operations in this space.

• 52:04Speaker 01

  Because it's incredibly hard. I wouldn't know where to begin if you handed me a bunch of money and said, Jonathan, go do this.

• 52:09Jacob Effron?

  Yeah.

• 52:11Jacob Effron?

  You obviously have like this, you know, this clear interest in kind of previous work in kind of, you know, policy, philosophy. What are like some of the questions that we should be talking about more with regards to that and AI?

• 52:23Speaker 01

  I mean, philosophy, you know, we can leave that off to the side. I like the more practical side, the policy side. I've spent time in that world. I still spend time in that world. We're building these systems. We're the best experts on these systems. We owe it to society to make sure that these systems are used responsibly. You know, policy is our way of, you know, trying to make sure that the, you know, we shape the world in ways that are good for us as a society. And folks in the AI world need to participate in that conversation. Not just with self-interest in mind and in trying to shape policy to their, you know, capitalist desires. But because we're a part of that society too, and we need to help that conversation move forward. So. You know, it's really important to me to just be a part of that at all times and make sure, you know, we're not there.

• 53:05Speaker 01

  Out of self-interest, we're there to be of service to, you know, to our country, to the world, depending on the context. You know, even to our local jurisdictions in a lot of cases, like finding ways to get involved in New York and San Francisco or, you know, in in Boston and Cambridge back in the day. Like, it's really important that we're we're of service. And I don't think we should lose sight of that. And it's.

• 53:27Speaker 01

  You know, it's a very fine line because why should you trust someone from Databricks that has enormous self-interest? Why should you trust someone from OpenAI or Anthropic who has enormous self-interest? It's, you know, in many cases, it's, the way I think about it is, A, it's pretty obvious when you're shilling. Like, a lot of people think they're doing a really good job shilling. They're not. Like, people know when you're coming in with self-interest. And the other part is, trust, let me find the right way to put it. In policy, there's no right and wrong. There's just trust. And trust in you and your integrity as a person. And trust is very, you know, hard-won and easily lost. And I've spent the past decade or so working really hard to build that trust. And it's important to me that I don't violate that trust because these are lifelong relationships

• 54:10Speaker 01

  with me and my colleagues in policy, me and these organizations, me and society. And so, you know, I, I hope my friends, you know, despite the fact that I come in, obviously, with clear financial interest in the outcomes of one particular company and with a particular industry, I hope, you know, I can come in and make my biases clear and still share useful information. And, you know, I wish everyone in, in the technology space kind of was willing to just come in and be open and put all their cards on the table and earn that trust over time. I think there are a lot of folks who have done that well and a lot of folks who haven't done that well. And a lot more we owe society than I think we've given to date.

• 54:43Jacob Effron?

  I guess, keeping in mind, I'm not allowed to ask future-facing questions. So I guess in the, in the, like, what, you know, what policy issues do you feel like are most important to be talking about right now?

• 54:50Speaker 01

  I mean, I think we need to figure out when we want to allow these systems and when we don't. It is an okay thing to sometimes say, maybe this system is not reliable enough, and we don't want to use it in a certain context. For law enforcement use of facial recognition, that was a position that, you know, that I had and, you know, held to very strongly because I think it's, you know, it's really risky. You can affect someone's life and liberty. You can, even if you don't actually throw them in jail, even if you don't arrest them, you can still turn their home upside down by searching them. You can still cause them to miss a day of work and lose their job over a false match if you're not careful with how you handle that. So. You know, it's, these are, these are not things to be taken lightly and we should be very careful when we automate. Maybe automation will be better than humans, maybe it won't,

• 55:35Speaker 01

  but we need to think about that incredibly carefully. We need to think in certain contexts about what standards we hold these systems to, and then what standards we hold humans to. And maybe we'll realize we haven't held humans to very good standards either. You know, I've mentioned that before, but it's, I don't think that should be lost on us. But, you know, I, I really think about this context by context, application by application. I don't think we should just, you know, blanket regulate the hell out of everything. But I think we should be thoughtful about where we want to use this technology, or at least where we really don't want to use this technology. And it's okay to say, we're gonna be a little bit technologically backwards, perhaps, in very specific areas, if it gives us the time to be careful and be thoughtful and, you know, not completely ruin someone's

• 56:18Speaker 01

  life inadvertently because we put too much trust in a technology. I think there are very few situations where that's true with AI these days. I think there's like incredible greenfield where, you know, from a regulatory perspective, we should let people run, let people be creative. Amazing things are going to happen, but there are also cases where we should exercise extraordinary caution. And the places where you're hearing about this most, law enforcement, medicine, autonomous vehicles, are situations where, when you make a mistake, it costs a lot. It may cost someone their life. And that's not something to be trifled with.

• 56:49Jacob Effron?

  Yeah. Well, I think it also goes back to what you were saying earlier about, obviously, I feel like as, as an industry, there's the opportunity to build trust and that trust is easily, you know, broken. And that, you know, these high stakes issues are where you kind of build that.

• 57:01Speaker 01

  Yeah, it's, I mean, these, knowing when to stop, knowing when to make reasonable promises to people, to not promise the moon, to not promise superintelligence in everything, when we clearly don't have that. And not to promise what that future is going to look like in a specific timeframe because we don't know. Like, scientific honesty.

• 57:20Jacob Effron?

  Yeah.

• 57:20Speaker 01

  Not, not to say people have been dishonest, but honesty in terms of like putting all of our cards on the table and saying, here's what we know and here's what we don't know. I think that builds trust very quickly. And you've heard me say that a bunch of times here. I've been wrong a lot. That's okay. Nobody fired me over it. Being wrong is okay. Being honest about my level of uncertainty about different things and not being willing to predict the future because I don't think I'd have good predictions right now. I think that's a level of scientific honesty we all owe.

• 57:48Jacob Effron?

  Well, fascinating conversation. We always like to end our interviews with a quickfire round. And so maybe to start, what like area of AI applications do you feel like is underexplored today?

• 57:57Speaker 01

  It's hard to say areas of applications because I think everybody's exploring everywhere. I mean, again, 12,000 customers at Databricks. We're working with all of them on AI right now. Nothing is, I would say, not being explored right now. So I don't want to point my finger and say, gosh, I really wish someone would build AI for blank. I mean, I really wish I had AI that washed my dishes for me.

• 58:17Jacob Effron?

  Sounds nice.

• 58:19Speaker 01

  You know, there are a lot of things where just I wish I had something that, you know, AI that made my bed. It would be, you know, robotics is something that I think, I wouldn't call it underexplored, but it's something I'm excited about. Not that I know much about it, but that I think there's enormous potential in embodied systems. There is also enormous risk. And, you know, I have a smart vacuum cleaner. I love it. That was like the best money I ever spent. My floors get vacuumed every day when I leave the apartment. It is, you know, I'm a bit of a neat freak, but, you know, that to me is just, you know, incredible. So, you know, I think, you know, robotics, who knows when, you know, whether it's going to be this decade or the next decade, you know, and many people said it was the last decade or the decade before when it was going to happen, kind of like VR, kind of like a lot of things. But when that happens, I'm really excited.

• 59:02Speaker 01

  The ability to kind of do things that are unscalable right now with work in the same way that having access to, you know, the steam engine. It can be very powerful. Not that I'm going to make that metaphor that it will change the world in that way, but it's another iteration on, you know, allowing us to interact with our physical world in the way we can with our information world. And that could be very exciting to me, but I don't think it's, you know, underhyped.

• 59:23Jacob Effron?

  Yeah.

• 59:24Speaker 01

  There's plenty of money pouring into that. I just hope it works.

• 59:27Jacob Effron?

  Yeah. That would certainly be powerful for the world. You know, there's rumors of kind of this next wave of models, folks spending, you know, 50 billion plus on compute. I want to ask you to comment on whether it will work on the scaling laws, but do you think that amount of compute spend at this point makes sense?

• 59:42Speaker 01

  It's hard to say. I mean, suppose that happens and you get superintelligence out of it. I think a lot of people would say yes. Suppose it happens and actually you get something incrementally better than the models today and it turns out we've kind of saturated, then you'd say no. How am I to know? That's predicting the future as far as I'm concerned. I'm glad someone's running the experiment and I'm really curious to find out.

• 1:00:02Jacob Effron?

  Obviously you have a pretty fascinating job today at Databricks. If you weren't at Databricks and you could kind of parachute into any organization, where do you think would be most interesting to be in the AI world right now?

• 1:00:14Speaker 01

  Databricks. Quite frankly, Databricks, I'm seeing where the rubber meets the road.

• 1:00:19Jacob Effron?

  Yeah.

• 1:00:19Speaker 01

  12,000 customers, all of whom are trying to figure out what to do with AI and trying to figure out, like, in some sense, we've built this incredible set of technology, but what is the sum total of it? What does it mean for the world? I mean, you've seen my interest in both science and policy and society in the world. This is the nexus of all that right now. I'm, I'm there on the ground floor seeing this happen. So if you made me parachute, if you took me out of Databricks and said parachute somewhere, I'd parachute right back to my current job. You know, I'm really, it's, it's just fascinating every single day to, to see a new customer, a new scenario. The, if you really, you know, if you want me to put it in like very glamorous terms,

• 1:00:59Speaker 01

  you know, OpenAI's definition of AGI is something to the effect of, you know, AI that is better than most humans at most economically useful tasks or something like that. I am seeing all the economically useful tasks right now. I get to be on the ground floor of watching, like, if, you know, by OpenAI's definition, and we can quibble with that definition, how you measure it, but I am seeing most economically useful tasks and I'm seeing where AI meets that and I'm seeing how to make AI effective at those tasks. In some sense of AGI is going to happen by that definition. It's going to happen at Databricks.

• 1:01:32Jacob Effron?

  Well, I love it. I feel like that's a great place to end up. Maybe I'll just leave the last word to you. Like, where can folks go to learn more about you, about the work you're doing at Databricks? Feel free to point them wherever.

• 1:01:39Speaker 01

  Yeah, Databricks.com. Come check us out. I mean, if you're interested in working with us, you know, you can go to our jobs page and you can see tons of jobs across the organization. I would say there is no job at Databricks that doesn't involve AI at this point. There are jobs like mine where AI is the primary thing that I worry about on a day-to-day basis, but you can't avoid it at Databricks because we are the data and AI company. That is what we do. There's no separating data and AI at this point. You know, if you want to be a customer or want to, you know, come work with us, you know, come check us out. I'll give a couple of plugs to specific products, actually, that I think, you know, all of these are in some phase of preview right now. So really, you know, my audience is our 12,000 amazing Databricks customers who might listen to this, but this is one way to get to them. You know, come check out our AI model gateway, which is an amazing way not only to interact

• 1:02:24Speaker 01

  with a wide variety of models. Kind of in a multi-cloud way, you know, just classic Databricks, but with a lot of amazing features that build on top of those models like monitoring and, you know, being able to switch between models or A/B test really easily. Come check out our agent platform or agent evaluation product. Agent evaluation product going into public preview by the time, you know, you listen to this podcast probably. And, you know, that's how I think people are going to build the next generation of their evaluations and get up and running with a real AI project in an afternoon. Come check out, we have some cool new fine tuning techniques. There's one we call soft. You know, if you want to learn more about that, come find me. I'm not going to say any more than that, but it is a unique new way of fine tuning models that takes advantage of the fact that people often have fragments of data,

• 1:03:08Speaker 01

  but are missing key pieces that someone would expect if you wanted to do fine tuning the old fashioned way. We've got a solution to that, that my team's been working on really closely. That is in preview right now. And, you know, if you manage to track me down, I'm very easy to find online. I'm very online through any medium except TikTok and Snapchat essentially. Find me or find your Databricks rep and let's chat about that. I'm looking for a handful of customers to work closely with on that. And that's just the day to day for us is we're always tinkering with new things. I want to find folks who will come hang out with us and help us figure out when this makes sense, help us go on that journey together. And then that becomes a product. That's the right way to build it. So, you know, come find me if you want to work with me as a customer, as a colleague. And, you know, even as a collaborator, if you're, you know, in academia

• 1:03:52Speaker 01

  or you're at another company where you think we could do some great things together. We just want to make our customers successful and there are lots of paths to do that.

• 1:03:59Jacob Effron?

  I love it. Well, Jonathan, thanks so much for coming on. Seriously. Fascinating conversation.

• 1:04:02Speaker 01

  Yeah, thank you so much for having me. This has been great.