#261 Jonathan Frankle: How Databricks is Disrupting AI Model Training

This episode is sponsored by Oracle. OCI is the next-generation cloud designed for every workload – where you can run any application, including any AI projects, faster and more securely for less. On average, OCI costs 50% less for compute, 70% less for storage, and 80% less for networking. Join Modal, Skydance Animation, and today's innovative AI tech companies who upgraded to OCI…and saved. Try OCI for free at http://oracle.com/eyeonai What if you could fine-tune an AI model without any labeled data—and still outperform traditional training methods? In this episode of Eye on AI, we sit down with Jonathan Frankle, Chief Scientist at Databricks and co-founder of MosaicML, to explore TAO (Test-time Adaptive Optimization)—Databricks' breakthrough tuning method that's transforming how enterprises build and scale large language models (LLMs). Jonathan explains how TAO uses reinforcement learning and synthetic data to train models without the need for expensive, time-consuming annotation. We dive into how TAO compares to supervised fine-tuning, why Databricks built their own reward model (DBRM), and how this system allows for continual improvement, lower inference costs, and faster enterprise AI deployment. Whether you're an AI researcher, enterprise leader, or someone curious about the future of model customization, this episode will change how you think about training and deploying AI. Explore the latest breakthroughs in data and AI from Databricks: https://www.databricks.com/events/dataaisummit-2025-announcements Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The idea is that you can, if you really want to be reductive about it, this is a synthetic data generation and usage technique where you give us the prompts. We generate synthetic data based on those prompts and then we integrate that synthetic data back into the model. And then you can repeat this process many times with more prompts as you collect them over time. And the nice thing about synthetic data generation at training time is like, I usually make recommendations to customers to be kind of as conservative as possible when they're doing something with AI. They're already doing something with AI and that's an aggressive choice to make.

• 0:30Craig S. Smith?

  Yeah.

• 0:30Speaker 00

  We're still learning how to do that. So, you know, don't push your luck in some sense. So what I typically tell people is don't deploy something that you haven't tested. And so my usual recommendation is deploy the model, you know, collect the data and then redo tau on all the data you've collected so far. Test that model and then, you know, even consider A B testing it to make sure that, you know, for whatever metrics you're looking at in production, this model isn't going to make things worse and then deploy it.

• 0:54Craig S. Smith?

  In business, they say you can have better, cheaper or faster. You only get to pick two. What if you could have all three at the same time? That's exactly what Kohair, Thompson Reuters and Specialized Bikes have since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development and AI needs where you can run any workload in a high availability, consistently high performance environment and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper?

• 1:39Craig S. Smith?

  OCI costs up to 50% less for compute, 70% less for storage and 80% less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is a cloud built for AI and all your biggest workloads. Right now with zero commitment, try OCI for free. Head to oracle.com/eyeonai. Eye on AI, all run together, E Y E O N A I. That's oracle.com/eyeonai.

• 2:22Craig S. Smith?

  But why don't you begin by introducing yourself? Go ahead.

• 2:26Speaker 00

  Definitely. So I'm Jonathan Frankel. I'm Chief AI Scientist here at Databricks. I oversee a lot of our research efforts on trying to make AI more useful for our customers and understanding how AI and data interact in a productive way. I got here, Databricks acquired my startup Mosaic ML, I guess about a year and a half ago now. We were kind of dedicated to helping everybody train their own AI models from scratch, founded kind of out of my PhD along with, you know, co-founders Naveen Rao and Hanlin Tang and Michael Carbon. And before that, I was a PhD student studying how to make neural network training more efficient. So it's been a wild journey over the past few years. But, you know, if you're looking for one through line,

• 3:08Speaker 00

  I care a lot about making sure everybody can use and control this technology and do with it what they want. I think a lot about kind of the internet where everybody could have a website. And that led to just such an explosion of creativity, a lot of really bad ideas and a few absolutely brilliant ideas. And, you know, if I told you from the beginning, oh, why don't we create an encyclopedia online that anyone can edit and contribute to, you'd say, definitely one of the bad ideas. And it turns out it'll change the world. So the idea that AI is similar and everybody kind of has the chance to do creative things with it to solve their problems, you know, it'll change the world again.

• 3:47Craig S. Smith?

  Yeah, absolutely. I agree, which is why I have this podcast. So I was reading in at Databricks, what's your, I mean, describe for listeners what Databricks does, first of all. I mean, I've, I've talked, I think, before on the podcast about the data lakes and or data, what's it called? Yeah, data lakes, is that the term?

• 4:16Speaker 00

  Yeah, we're the data lake house.

• 4:18Craig S. Smith?

  Right, exactly. Data lake house. That's right. And what that means and, and how what you're doing on tuning models fits in that.

• 4:31Speaker 00

  Yeah, so I think the the Databricks journey, actually, from the very beginning, you know, the the stories have lived in legend for me as someone who's watched Databricks from afar. And now I get to be a part of it and hear all the inside stories. But the story began kind of where it is today. People have lots of data. They need a way to manage it and they need a way to understand it. And kind of data and AI are intrinsically linked to each other. Because what do you do with a bunch of data? You understand it. And Databricks is, as far as I can tell, looking through the history books, always been about using whatever the latest AI was to understand your data. Now, that AI wasn't always, you know, large language models or neural networks or anything like that. But and machine learning have been fundamental. Because what do you do with your data?

• 5:15Speaker 00

  You don't just, you know, stick it in a data lake house and just let it sit there collecting dust. You want to understand it. It's only useful. It's only worth having there if you can do something useful with it. So this is kind of this concept we like to refer to these days as data intelligence. The idea that, you know, you want to be able to talk to your data, understand it, figure out what's there, interact with it, learn what it's trying to tell you, kind of make effective use of it. So, you know, we offer ways to manage both unstructured data like documents and structured data. You can think of that as like, you know, SQL or kind of individual entries into a database. And a lot of the work these days is on the question of how do we use all the cool, modern AI

• 5:55Speaker 00

  that's out there to give people new ways to understand it and to give new people ways of understanding it? There are a lot of people who in yesteryear would have had to like, write a notebook and write Python and do a SQL query or write in Scala or Spark to be able to get insights from their data. And today, I think AI is going to make it possible for pretty much everybody to do this. I don't know SQL, but I've been able to do this for my data, Databricks.

• 6:20Craig S. Smith?

  Yeah, and and the the data lake, uh, lake house concept, uh, it really started during the supervised learning phase of AI, when everyone was labeling data and, uh, training models with labeled data, most largely, uh, sort of text, uh, you know, reading models or, uh, image reading models. Uh, and and you came in after the the generative AI revolution, where suddenly

• 7:00Craig S. Smith?

  Really, the, uh, those silos are breaking down. I mean, at the time during the supervised learning phase, I used to talk to people about, you know, you train your, uh, data, but there are all these pools or lakes of datas around and sooner or later, they're going to merge and we're going to have an ocean of data, a lot of it labeled, uh, but the science kind of overtook that. You, you don't need to do that any longer. Uh, I wanted to talk to you about, uh, Tao, which, uh, I've got to call it up, uh,

• 7:45Craig S. Smith?

  let me see here, Tao, Tao, so test time adaptive optimization, uh, which I found fascinating. I read your paper. Uh, before I start asking about it, what really fascinates me is it looks like it's a way to create an ever improving model, at least in, uh, narrow domains, which is something that I haven't seen before. Maybe it exists and I just haven't stumbled on it. But, uh, but first of all, can you talk about, uh, training and tuning, uh, how you, you, is that what, what, I mean, I guess that's what

• 8:29Craig S. Smith?

  Mosaic was doing, right? And that's what Databricks bought you for, but is, what's, what's, uh, talk a little bit about your role in, in that at Databricks?

• 8:42Speaker 00

  Yeah. So I think it's worth taking a step back and just asking, you know, yeah, why am I even here? Like what is, what is my purpose and what do I contribute? You know, I'm an AI researcher.

• 8:53Craig S. Smith?

  Every day, yeah.

• 8:54Speaker 00

  Yeah. We're trying to solve real problems for customers. Like, why, why do you need a guy like me around Databricks? And, you know, my best answer is, There are a lot of cool things happening in the research world that are clearly very valuable somehow. And there are a lot of important problems our customers have that they need to get solved and they really wish they had the magic to do it. And my job is to connect the dots between these. And Tao is one of those examples. So, you know, customers want to get their data and their models to interact with each other. They want to be able to bring all that data to their models and have the models be able to speak fluently or answer questions about their model or reason in whatever domain they care about. And there are lots of ways of doing that.

• 9:38Speaker 00

  And those ways are things like you can prompt or do rag to kind of bring your data in. You could fine tune a model on your data. There are lots of ways to do this. You could even have a model write calls to a sequel database. And fine tuning is probably the most powerful of these techniques. It lets you literally change the model and, you know, teach it more about your data directly and intrinsically as opposed to having the model kind of start fresh every time you interact with it. And, you know, fine tuning is a bit of a tricky thing to do for a lot of reasons. But one of the reasons we saw again and again was that fine tuning is really demanding in terms of data. You basically to fine tune a model to do something, you need examples of doing that exact thing with your data.

• 10:22Speaker 00

  So if you want a model to serve as a customer service bot, you need to have examples of user asking question and what the customer service agent should say in return. And you need thousands of those examples. And what we found was that people have lots of incredible data. It's very rare that they magically have the perfect fine tuning set. That's just not the kind of data that gets created in nature. In nature, you have lots of documents that accumulate and you have, you know, some telemetry data and you have some user inputs and you have, you know, some manuals on how to respond or some guides or some decision trees or whatever else you have floating around, plus a sequel database of relevant information. But in nature, you don't have this perfect pristine data showing what a, what an interaction should look like with an LLM.

• 11:08Speaker 00

  It's just a very, it's a hard, it's a lot to ask of someone. And, you know, the correct answer, when you come to a customer and they say, I really want an LLM that's fine tuned for my task is not, well, go annotate 10,000 examples and come back to me when you're done. That's not an adequate answer. And I think to date, that's been the answer that the research community has had for all of our customers. Just, you know, it's your problem. You didn't give me the right data. And I think that's a bad answer. That's not an answer that's going to lead to people successfully deploying AI. Nobody has time for that. Nobody has resources for that. By the time they do that, the field will have moved on or their business will have moved on. And so the question we asked ourselves with Tao was kind of, on the one hand, what can we do to simplify this?

• 11:50Speaker 00

  Like, is there something we can take away or something we can, some constraint we can remove that will make this easier? And on the flip side, does the technology exist to actually do anything useful with that? If I have the customer give me just inputs, but no outputs for how the LLM should behave? Well, is, do I have science that can help me to actually tune a model on that? Or that, it sounds kind of unreasonable, like you have no idea what someone actually wants the LLM to do. You know, there could be many good responses to one input, depending on what the user wants. And so my job, in a sense, is to move back and forth between these worlds. On the one hand, I'm always asking like, what's making our customers' lives difficult? Or what do they wish they could do?

• 12:32Speaker 00

  And if I had a magic wand and I could make their life a little easier, what would I do to make that easier? And on the other hand, okay, my magic wand is really not magic, it's science. What techniques exist? What techniques can I modify? What can I build that will bridge these gaps? So half my time, I feel like I'm a researcher. Half my time, I feel like I'm a product manager. Actually spend a huge amount of time with our customers, just talking to them about what they're trying to do and understanding their problems. Not the ones in abstract that I think many of us think people have, but really where they're stuck. And if only I could remove this one thing. So with Tao, that's exactly what happened. We said, getting labeled data is really hard. How can we simplify this? And there are lots of ways you could simplify it. You could ask someone to write a detailed rubric of how something should behave,

• 13:17Speaker 00

  or write a bunch of judges or graders that tell you like, for any arbitrary output, how good is it? It turns out those things are also really hard. And it would be nice if we could just simplify it further. You don't even need any labels at all. That means that all you have to do is show up with a bunch of inputs reflecting what people might ask your LLM. And you just get an LLM that's good at those inputs. Now, the nice thing about that is it's really easy to get inputs. Just set up a really basic LLM. You can set up Llama, give it a really basic prompt and ask your friends or your teammates or yourself to just interact with it and ask it a bunch of questions. It's probably going to do a bad job, but you don't care whether it gives you a good answer. You just care that you've asked it a good question. And so this is great.

• 14:01Speaker 00

  This means that it's really easy to get the training data because it's something that people can naturally provide to the system. There's no real work involved. We're not asking someone to sit down and literally write 10,000 questions. Just put this in the wild, beta test it, prototype it. Ask your team to do an hour hackathon and just ask it all the questions that they might want to pretend to be the customer. Ask a customer to prototype it with you and you can collect the data you need. Then you get to the really tricky part, which is what the hell do you do with that data? Cause it sounds, I mean, I, I hope I'm not like being overly dramatic, but it sounds kind of insane, like all I know are the inputs and you're going to tune a model that will give me the outputs I want. That's like, what's going on? And you know, this is a place where we turn to techniques from reinforcement learning.

• 14:45Speaker 00

  And reinforcement learning is a set of techniques where rather than giving yourself inputs and good outputs, you give yourself inputs and just an environment where a model can explore. You can think of this like, you know, when, um, when you know, people have built models for Go, for example, DeepMind built this model that was really good at Go or game playing. Let it explore the environment and figure out for itself. You need to have an environment it can explore and some kind of signal that tells you what outcomes are good and which outcomes aren't. This is why people love games so much. Cause you just, you explore the environment, you know, when you're scoring points, you know, when you're winning and you know, you get good signal. And here what we did is the environment is in some sense, this special model we call DBRM or the Databricks reward model.

• 15:28Speaker 00

  This is a type of model that's very common in the reinforcement learning world. And all it's meant to do is tell you how good is a particular output. And you know, it's that sounds very generic, like just generically, how good is it? But the way you train this is you show it lots of examples of, you know, pairs of outputs for a given input and which one a human liked more. And you just show it tons and tons of this data. And it turns out it actually generalizes pretty well. So, you know, you combine that with a bunch of inputs you have, do some synthetic data generation and kind of filter it with this reward model. Use the right techniques to train the model after that, you know, a lot of, and this sounds really easy, a lot of very careful work on the part of the scientists who worked on this to just get the details right.

• 16:13Speaker 00

  And it turns out that it actually, there are very reasonable choices you can make a lot of the time. If I showed you two outputs and I showed you the input, you'll probably have a pretty good intuition as to which one is better. You won't know which one is right. But as long as you can kind of keep steering things in the right direction, the model will get better and better. And that's what we saw. Not only did this actually lead to improvements, even though you had no examples of how the model should behave, but it even beat training on labeled data, which is kind of crazy. And I can say a bit more about that at some point if you'd like, but it's, you know, it's not as crazy as it sounds, I think. But it was just very surprising to us. And this worked across a pretty wide range of tasks, like document question answering and SQL, which again was very surprising to me

• 16:57Speaker 00

  that a reward model could kind of generalize in that way. It's, it's really exciting for the future. Maybe enterprise tasks are more similar than we think.

• 17:05Craig S. Smith?

  Yeah, I mean, but, but supervised fine tuning, you need these question answer pairs in, in,

• 17:14Craig S. Smith?

  for a language model.

• 17:19Craig S. Smith?

  And what you're talking about, you, you still need question answer pairs, right? But you're getting them through reinforcement learning with human feedback.

• 17:29Speaker 00

  I, I wouldn't call them question answer pairs here because I think it's, it ends up being more complicated than that. I would call it questions and kind of steering on which answers are better or worse. Cause it's not really, none of them are, there's nothing in Tao that ever checks, is the answer right?

• 17:49Speaker 00

  Or even is the answer good? There's just asking, is the answer better? And which answers are better and trying to continually move the model toward better answers, which is very, I don't know, I found that to be a little bit mind bending when I started this project. I have a bunch of reinforcement learning kind of enthusiasts on my team and specialists on my team, who have been, you know, spouting the gospel of reinforcement learning for as long as they've been here. And I've just been politely ignoring them and patting them on the head. And, you know, I've learned, actually, maybe there's something to what they do and I should listen to them a little bit more.

• 18:19Craig S. Smith?

  Yeah. So in this process, you do use RLHF?

• 18:24Speaker 00

  That's correct.

• 18:25Craig S. Smith?

  Yeah. And then you, once you have that, that data, then you use pure reinforcement learning?

• 18:34Speaker 00

  That's correct. So we're kind of using, I would think of this as techniques from the RLHF literature. And just kind of combining them in the right way. I won't claim that there's anything we invented that's mind bending from like a scientific perspective. I, you know, I think there were, when we released this, there were some folks on Twitter who were like, this doesn't sound very novel. And my response was, it works. And getting it to work is the hard part. Like, that's the novelty is easy. Like, getting stuff to work for real customers is hard. So the details of like, okay, we're gonna take some inputs, we're gonna generate in the right way, we're gonna select in the right way, build the right reward model, all that stuff took time.

• 19:09Craig S. Smith?

  This reinforcement learning step, after you've gathered the data, is happening at test time. Is that right? It's very similar to DeepSeek and, and this test time compute training that, that has become kind of a

• 19:34Craig S. Smith?

  paradigm for refining models.

• 19:40Craig S. Smith?

  But, but it's different in that you're only doing it during the tuning.

• 19:47Craig S. Smith?

  Phase, you don't do it during the inference phase. Can you, can you talk about why that's an advantage? And what really fascinates me is unlike test time, compute, training

• 20:06Craig S. Smith?

  is temporary, it's, it's, it improves the inference accuracy, but once you're done with that inference, the model reverts to its

• 20:20Craig S. Smith?

  training state and you lose that improvement, right? But in your case, it continually improves the model. Can you, can you talk about that?

• 20:29Speaker 00

  Yeah, yeah. So I think the, there was actually enormous debate internally about whether we should use the word test time. Because it's really, you know, because it's not really, it's like using test time compute for training. And so is it really test time? There was kind of, you know, there, there was some debate over this word. So the way I think about it is even ignoring the word test time. The idea is that you can, if you really want to be reductive about it, this is a synthetic data generation and usage technique where you give us the prompts. We generate synthetic data based on those prompts. And then we integrate that synthetic data back into the model. And then you can repeat this process many times with more prompts as you collect them over time.

• 21:10Speaker 00

  And the nice thing about synthetic data generation at training time is like, well, let's take a step back. If you were to just use, let's say, O1 or R1, which are both excellent models. You're going to use kind of an uncertain amount of compute at inference time. You don't know how much is going to get used. The model may explore for a while. It kind of, you know, there's some uncertainty there. And, you know, that can get expensive as well. We, we saw kind of in practice, O1 and O3 were using about five times as many tokens, somewhere around then to solve problems, which is not unexpected. They're designed to do that. That's the whole point. That's why they're so powerful. And one other way of looking at this is, what if you use all that extra compute to generate the synthetic data?

• 21:55Speaker 00

  And then you create a standard model that when you actually go to use it at inference time, doesn't do and doesn't use any extra compute, just behaves normally. And the idea here is we expect that for our customers who are using this technique, if they have a bunch of inputs already, they've probably already got an application deployed. They at least have some amount of confidence that, you know, it's worth doing this training. And so they're willing to invest a little more in training to make inference a lot cheaper. So the idea is, okay, let's spend some more compute at training time to generate better synthetic data. And then let's generate a normal model that when we use it at inference time, has exactly the same inference cost as, you know, Lama had we not done Tau.

• 22:38Speaker 00

  So it's a bit of kind of a, it's taking this test time compute paradigm and flipping it on its head a little bit. The test time compute is to help you build the training set. But at the end of the day, you're training a model that you use normally. And we think this is kind of the right mix for customers who are deploying applications at high volume. If you're doing something like Tau, you know, you're, you know, you're trying to cut down on the cost at inference time anyway. Often you're trying to train a custom model to replace something else that you might have. So let's really lean into the fact that we can help you reduce your inference cost and a little more upfront investment, probably worth it to you.

• 23:11Craig S. Smith?

  Yeah. And then the, once you're in inference mode, you're collecting more queries, more prompts that then you can use in tuning mode, just the tuning time compute. Does, do you periodically go back and retune the model on the data that's built up? Or is there a way of doing that in the background, even if the model is being used to, for inference?

• 23:50Craig S. Smith?

  Or, or do you split it into a, two instances of model and one, you're improving through this tune time compute, the others doing inference out in the wild? And then you swap the models periodically? I mean, how does, how does that work?

• 24:10Speaker 00

  It's a great question. It's something we're still kind of experimenting with customers on, just to kind of get it in exactly the right shape. I usually make recommendations to customers to be kind of as conservative as possible when they're doing something with AI. They're already doing something with AI and that's an aggressive choice to make.

• 24:25Craig S. Smith?

  Yeah.

• 24:26Speaker 00

  We're still learning how to do that. So, you know, don't push your luck in some sense. So what I typically tell people is don't deploy something that you haven't tested. And so my usual recommendation is deploy the model, you know, collect the data. And then redo Tao on all the data you've collected so far. Test that model and then, you know, even consider A B testing it to make sure that, you know, for whatever metrics you're looking at in production. This model isn't going to make things worse and then deploy it. And the idea there is, you know, first of all, it's just be playing it safe and. You know, again, you're making a very aggressive choice to use AI at all right now. It's exciting. But, you know, be careful.

• 25:04Speaker 00

  And the other piece is that you don't necessarily want to just want to train a model that you've already done Tao on on more recent data. You can end up in this interesting place where the model will kind of get extra good at the latest stuff. And may forget things because you're emphasizing the latest data. Sometimes that makes a ton of sense. If you're in a chatbot application, let's say for retail. You may actually want to use the latest data because, you know, you don't want it to be talking about Valentine's Day when it's Thanksgiving. Sometimes you do want to mix all the data together just to, you know, make sure that you're fully representing the earlier stuff. So there. This kind of gets into more application specific considerations. We're like, I'm guessing the big recommender system is powering like ad placement at Google and Facebook. I'm guessing those are actually trained mostly on most recent data because human behavior changes so quickly.

• 25:50Speaker 00

  But for a lot of our enterprise applications, taking things, you know, one step at a time and kind of combining everything together. Again, it's the safest possible choice to make. And then you can experiment and try lots of different things, but I wouldn't. I wouldn't recommend kind of trying to update in real time, especially without testing the models first.

• 26:07Craig S. Smith?

  Yeah.

• 26:08Speaker 00

  Just to be extra safe.

• 26:09Craig S. Smith?

  Yeah. And so you you mentioned sort of overriding previous parameters. And which has been the the problem that's.

• 26:21Craig S. Smith?

  The restricts of sort of continuous learning of catastrophic forgetting. So yours doesn't get around the catastrophic forgetting problem.

• 26:35Speaker 00

  No, I wouldn't. I wouldn't make that claim. I would say, you know, very narrowly, this gets around the need for labeled data, or at least, you know, assists you by reducing the need for labeled data. When it comes to catastrophic forgetting, it, you know, it still puts you in the same trade off that you're always in of like, do you train on the latest data? And hope that the model doesn't forget. Do you combine all the data together? Do you up weight more recent data? There's kind of, I don't think we have an amazing answer for catastrophic forgetting. I will say forgetting is a lot less catastrophic than it used to be, you know, back in my day, um, five whole years ago. Catastrophic forgetting was a very severe problem in general when you did fine tuning. And these days, you know, models are a little more robust to that just because they're so much bigger and they're able to store a lot more knowledge.

• 27:18Speaker 00

  And also because we have better techniques for managing catastrophic forgetting. To give you one example, there was a paper we worked on last year as a lab. It's called Laura learns less and forgets less. It's using Laura, you know, this parameter efficient fine tuning technique where you don't train all your parameters. People used to think of this as like a compromise. You're doing this for the sake of efficiency because you can't afford to do full fine tuning. But you're going to get worse results. What we found a lot over the past year has been actually Laura is a regularization technique as much as anything else. It learns less and forgets less. It's a way to to reduce overfitting and a way to reduce forgetting. Now there's a trade off there. Sometimes it'll, you know, learn too little and,

• 27:59Speaker 00

  you know, forget or learn too much and forget too much. Sometimes it'll learn too little and forget too little. There's kind of a balance. But Laura has actually been a really interesting technique just to manage learning and forgetting, or at least to give you more opportunities to explore the trade off between the two. Deeply surprising to us, but it actually, one of the most common scenarios I see Laura is for customers who come to me and they say, I'm overfitting and I don't know what to do about it. And use Laura. It sounds counterintuitive, but it works great.

• 28:27Craig S. Smith?

  And ju- just for listeners, so we understand the the process. You take a model in in the old days or the the the current paradigm, you you take, let's say, an open source model. Uh, and you want to fine tune it. So you get a bunch of uh, labeled data. And you uh, fine tune the model on that labeled data and and then you release the model in the wild and and then the reasoning models are allowed at test time to to run further uh, on on the data or explore the data. Maybe you can help me um,

• 29:11Craig S. Smith?

  with with my vocabulary there.

• 29:13Speaker 00

  I I would, the way I would describe it to to folks who are a little bit familiar, at least, is it's automated chain of thought. You're kind of letting the model figure out its own chain of thought rather than guiding it through it. But you're kind of saying to the model, it's totally okay for you to spend some time thinking and exploring and considering and doing chain of thought. Get back to me when you're ready. Whereas, you know, in the more traditional paradigm, we basically ask the model for the answer right away. And hope that just doing one pass through the model is enough for it to figure things out.

• 29:40Craig S. Smith?

  Yeah, uh, but in that case, uh, the improvement that happens at test at inference time uh, is temporary. And uh, what strikes me about Tao is that uh, during tuning time, you're you're you're doing something similar in that you're uh, you're gathering uh, data from from inference time. And using it to tune the the the model of of.

• 30:15Craig S. Smith?

  Fine tune the model of using reinforcement learning. Uh, but the those changes like supervised reinforcement learning, it they they uh. They change the weights of the underlying model. So you don't lose that improvement. Is that is that right?

• 30:34Speaker 00

  That's right. And I think there's I'd add a little more nuance to this. Which is that, you know, how did OpenAI train O one in the first place? Or how did DeepSeek train R one? Well, what they did is they they let the model think kind of they did the same thing of kind of like tune time compute, if you want to call it that, where the model was given the chance to think and spend a lot of time coming to the right answer. And what they did is they had a way to check whether the answer was correct or incorrect. They threw out all the cases where the model thought for a long time and the answer was incorrect. And that became their new training data. And so they trained a model that knew how to kind of reason and do its own automated chain of thought. I think one of the big differences here is

• 31:16Speaker 00

  for our customer tasks, we have no idea how to determine what answer is right versus what answer is wrong. For DeepSeek or for OpenAI, they're focused on domains like math and coding where you can actually check the answer. There's a math equation. It leads to the right answer. There's a proof. You can check whether it makes sense. There's a, you know, there's a program and you can check whether it executes and has the right behavior. Whereas for a document question answering task or, you know, a summarization task, there is no right and wrong. And so instead, we're looking for better and worse. That's kind of an if you want to look for a key difference here. The other part is we're not trying to train the model to actually spit out its whole thinking trace. We're just trying to train a model to get right and wrong answers.

• 31:59Speaker 00

  But we're trying to generalize this to basically any domain that a customer might want to address. Which is, you know, they include a lot of hard, really fuzzy problems, like, you know, answering questions about health, where there's no one right answer necessarily, or no one right way to to give an answer. And so we have to, we have to have something a little more general to help with that. The other piece is that we expect the user to bring their own prompts. And if they bring their own prompts, we can kind of automatically specialize the model for a domain. Whereas in the DeepSeek case, they kind of have to generate a lot of math problems. And keep generating new math problems over and over and over again to get more examples. We're kind of relying on the user to bring their own prompts because, you know, who are we to say what their problem is? We should just let them describe it to us using the prompts.

• 32:42Speaker 00

  And we found it's very easy for users to get lots of prompts. It's just hard to get a lot of responses.

• 32:48Craig S. Smith?

  Yeah. In this process, before I get on to what Tal can do in terms of increasing the power of, of open source models. What is the, can you talk about the reinforcement learning process? And how that's done? You mentioned DBR. DBRM, is that what it's called? Deep Databricks.

• 33:19Craig S. Smith?

  Reward model, I'm sorry. Yeah. How exactly does that work?

• 33:24Speaker 00

  So we're using, you know, there's a bit we're probably not going to say about this because there was a lot of like very detailed algorithmic work and, you know, a little bit of innovation that went on there. But the general idea is that we're, we're generating a lot of data. We're figuring out which examples are better and worse using, or which outputs are better and worse using DPRM. And then we're applying relatively standard RLHF techniques on this. So it's nothing that will blow your mind or nothing that will surprise you. A lot of the work is in just getting the details so that this whole process works well and doesn't completely fall apart. And so honestly, the most interesting parts are the experiments where this went wildly off the rails and we had to kind of spend some time getting the hyperparameters exactly right. Getting the order of the steps exactly right, figuring out how many outputs to generate.

• 34:07Speaker 00

  All the ugly stuff that again, you know, for my fan club on Twitter or my anti-fan club on Twitter, you know, not novel enough.

• 34:16Speaker 00

  Nah, the novelty is overrated. Working systems are what matter.

• 34:20Craig S. Smith?

  Yeah. So, so the, the,

• 34:25Craig S. Smith?

  The DBRM is Databricks flavor of of a reward.

• 34:34Speaker 00

  Exactly. We collected a ton of data on tasks that we thought were going to be most relevant to our customers. A lot of stuff involving documents or structured data to unstructured data or vice versa. Writing tasks, kind of some math tasks, but we really kind of focused our data on the things that we thought would be reflective of what our customers were doing. And we feel pretty good that everybody's trying to understand the data they have in Databricks, and that's documents for the most part, a little bit of SQL.

• 35:01Craig S. Smith?

  Yeah, the in that process, you you use, I mean, there's the RLHF

• 35:11Craig S. Smith?

  part that that provides data, but on the reinforcement learning of tuning, that is autonomous.

• 35:23Speaker 00

  That's correct.

• 35:24Craig S. Smith?

  Using other L L M's to judge the outputs to mark whether it's to to reward or not reward, is that right?

• 35:33Speaker 00

  So that's where, yeah, that's where DBRM is most helpful. Is that DBRM kind of gives us our signal of what we should reward and what we shouldn't reward. If you imagine that game playing environment, in some sense, DBRM tells you what your score is at the game. It helps you understand which strategies led to higher scores and which strategies led to lower scores. And so that's kind of, if there's kind of one key piece to this whole thing, it's the work we put into making DBRM really good and really specific to enterprise tasks.

• 36:01Craig S. Smith?

  Yeah. And so with this, you're able to bring like a llama model up close to the level of oh, one or oh, three.

• 36:15Craig S. Smith?

  Uh, O three mini or R one uh, without increasing uh, uh, the inference cost of a, of a, of llama that's not fine tuned. Is that right?

• 36:28Speaker 00

  So I'll be really careful how I answer this, you know, as a good scientist, you have to, you know, I have to be a little bit pedantic here. I'm going to sound like a lawyer for a moment. Um, for all my friends in the scientific community, I want to represent this properly. The, the thing that I can say with certainty is at least for the tasks that we tried and for the models that we tried with the data sets we tried, because everything in science is contingent on those things. If you have bad data, nothing good's going to happen. Um, and you know, we can't, we're never going to claim that something universally works on every task, but for what we tried, which we hope is representative of what our customers do, we saw really big improvements in performance by using Tau. And not only were they big objectively, but they were actually big compared to doing supervised fine tuning. This was the part that got me most excited and

• 37:13Speaker 00

  was a bit of a surprise for us. If you, we, we used the exact, we had these, for every one of our tasks, we had a set of, you know, several thousand inputs and several thousand labeled outputs. So we could do supervised fine tuning. That's the traditional way of solving these tasks. We tried doing Tau where we just dropped the outputs and instead use Tau on the exact same inputs. And it in fact led to better results across the board, which is crazy, right? We're getting rid of labels. We're throwing out information and it leads to better results. And, you know, again, I don't want to promise this is universally true. It's something we saw for these training sets on these tasks. My hypothesis here, and it's just a hypothesis, is even for really good labeled data, data that, you know, is used widely in the academic

• 37:56Speaker 00

  community, that's quite popular, that our team has looked at, or our team has done our best work to try to create, it's still really hard to create good labels.

• 38:05Craig S. Smith?

  Yeah.

• 38:06Speaker 00

  And so DBRM in some sense is, it allows us to put all of our effort into building this one really, really good reward model. And that may be easier actually than trying to create several sets of several thousand good labels. You know, I don't think this is necessarily, we didn't invent magic. I don't think that in general, having no labels is better than having labels. Like, obviously, the more information you have, the better you should be able to do. I just think getting good labels is really hard. That was our takeaway. Like, wow, even for these world-class benchmarks, the labels may not be perfect. And so DBRM kind of shines in comparison. And so if it's hard for my team and for world-class academic researchers to create good labels, it's not a fair fight for our customers to have to do that.

• 38:51Speaker 00

  They're busy, they have other things going on. They're not sitting around getting PhDs in machine learning and spending their whole day trying to build the best labels. So let's just, you know, not only should we reduce the work for them, but it's probably leading to better results if they don't even try. And we instead kind of put a lot of work into DBRM to make their lives easier. So it's been this really, I don't know, it was really rewarding for me, no pun intended, to kind of see that maybe not only can we make customers' lives easier, but in fact, it might be better to do that. Usually you would think of this as a compromise. Now, the comparisons to O1 and R1 and GPT-4O, I think that varies a lot by the task. It's also a bit of an unfair comparison in a sense because, you know, OpenAI has to make O1 good at everything.

• 39:35Speaker 00

  And Tao is trying to make a model good at one thing. But that's kind of where the unfair advantage is for anyone who has some interesting inputs and domain knowledge. You just have to make the model good at your one thing. OpenAI is spending a huge amount of effort to try to make a model good at everything. And that's where you can kind of compete with the closed models by building something yourself and where we see a lot of opportunity for kind of custom fine-tuning of open models like Llama.

• 40:00Craig S. Smith?

  Yeah, and I guess the question is, you can get that improvement through supervised fine-tuning if you have enough label data, right? It's, it's just the, what is the cost of doing it that way versus the cost of doing it the Tao way? And the Tao way, since you're generating prompts or queries through inference time, and if you have a big user base, you're going to get a ton of that. It's just a cheaper way of doing it, is that right?

• 40:40Speaker 00

  Yeah, cheaper, but I, you know, the word cheaper is doing a lot of work there because it's cheaper in terms of cost, but it's cheaper in terms of time and effort and frustration. I don't know, Craig, have you ever tried to label data?

• 40:52Craig S. Smith?

  No, I've watched it done on a small, you know, on images, but yeah.

• 40:58Speaker 00

  I'd love to like, this is actually something I try to do with my team periodically, just so they remember. Like whenever we say, oh, just have the customers label data, I want them to kind of feel the pain. Understand how frustrating this is. Like whenever we say, oh, we're going to go and buy label data, just remember. Try being the user of the rubric and the specification you created or the product you created. It's really frustrating. Like getting my team to do this for more than like 20 minutes before they start checking their email and checking Slack. Is really hard. You start seeing people drop off the zoom call. Oh, gotta go. It's lunchtime. Got a meeting. Sorry. And now we're going to ask our wonderful Databricks customers to go do that. So it's really kind of.

• 41:40Speaker 00

  You know, it's cheaper, but I think that underplay is like it's, is the problem going to get solved with AI or is the problem not going to get solved with AI? And I want to give our customers at least more at bats to try to, to try to solve problems with AI. And this is a key enabler for that, I hope.

• 41:58Craig S. Smith?

  Yeah. And, and so the process is you, you start with a bunch of queries that if you, if you've been using the model. In, in some application, you're getting off your customers, but otherwise you, you have, have people interact with the model to develop the queries. Then you use those queries.

• 42:28Craig S. Smith?

  Then you go through a reinforcement learning with human feedback. Phase where you score the, the, the, the, the answers to those queries it's had right, whether it's good or bad, or which you prefer.

• 42:47Craig S. Smith?

  And then you take that data and use reinforcement learning to

• 42:57Craig S. Smith?

  to replicate those high scoring behaviors.

• 43:00Speaker 00

  And the question becomes, can you identify that? And if you can, you can steer the model toward doing more of that and less of the bad stuff on your task. And that's really, it's another intuition for fundamentally what we're doing here. It's, you know, easier said than done, identifying the good ones is, you know, if you knew how to do that, you probably wouldn't be generating the bad ones in the first place. But DBRM has been really helpful for doing that on enterprise tasks. And so I've kind of, it again, surprisingly well in based on what I expected. But does that, I don't know if that, is that intuition kind of a little bit helpful, kind of different way of looking at it?

• 43:34Craig S. Smith?

  And I, I guess this is the question on supervised fine tuning, because in this case, so you've, you've gone through this process, you improved the model, you, you put it back out for inference. People interact and it generates more queries or prompts. And then you can pull that in, go through the same process and tune the model further. And it, at least on that specific domain, it has sort of continual improvement. Is that right?

• 44:12Speaker 00

  That's correct. You're kind of seeing more of the set of possible inputs your users might provide. And that just gives Tal more data to work off of. So it's kind of, it, it's a bit magical that you just, you get improvement without intervention. That was what we were going for is how can I train and then improve without really needing to do anything? And just kind of let the process work itself out. Now, obviously this will encounter limits, you know, you, there, there's only so much you can figure out without giving more clarity on what task you're trying to solve. But I think for many of our customers, it certainly, the result showed it's good enough to get you, you know, an LLM that will like, you know, that will do a lot better than the base Llama model at your task and will start to rival the closed models.

• 44:56Speaker 00

  And for a lot of our customers, that's, that's winning. Like that's what winning looks like relatively easy improvements in your model that's something you control, you own is relatively cheap to serve. And then if you're really getting a lot of uptake, that's where you can go in and do the more detailed kind of hardcore sort of stuff that, you know, you can, you can start to label data or get thumbs up, thumbs down data from users or kind of do more advanced stuff to make this even more powerful. But just getting to the point where your biggest problem is making it even more powerful, that's success. That's a huge amount of progress from where I think a lot of folks are today with AI.

• 45:31Craig S. Smith?

  Yeah. And on supervised fine tuning, if you had some magical way to generate the supervised, the label data. But you could do the same thing. You could keep fine tuning the model over and over and get it stronger and stronger. It's just takes more compute, more man hours.

• 45:54Speaker 00

  Yeah, it's just effort. It's, you know, the person who's building this AI system at an enterprise is also trying to do their day job. And also trying to, you know, they're either trying to do their day job, or if they're lucky enough that this is their day job. They're trying to prove to their business leaders that AI has a place in their organization. Cause I think that's still something we all have to prove with this new technology. And so just like, yeah, let's get out of people's way. Let's make sure their time is being spent carefully.

• 46:21Craig S. Smith?

  Yeah. And what about, this is on narrow tasks. What about on multi tasks, multitask L L Ms? I mean, it's, you can apply it, given you have enough prompts of in your enterprise. You, you can apply it to more than just one task, right?

• 46:47Speaker 00

  Yeah. And we, we saw really good results with this, where we kind of put together a prompt set geared toward a bunch of different enterprise tasks. You know, some documents, some question answering, some reasoning, some, you know, chat sort of things. And we got general improvement across that family of tasks, which made us confident, you know, as people move toward agents. You're going to probably want one L L M that can handle many different kinds of tasks for you. You can use the same model in many parts of an agentic workflow. And this gives us some hope that you can actually train something that's not just specialized to one task, but can do many tasks. I'm still not going to claim it can do every task or anywhere close to it. That would be, you know, I think that would be a little bit of hubris at this point.

• 47:29Speaker 00

  But, you know, I, I do feel good that at least we have results indicating someone can, can hopefully have an L M that can both do agentic routing and can also, you know, generate sequel queries and can also, let's say, you know, solve customer service requests. And if you have good, you know, prompt sets for each of those, you can get an L M that's going to do a pretty good job at those. And, you know, that, again, these were surprises. Things worked a little better than we expected, which is, you know, it's the happy part of science. Most of the time, things work much worse than you expect.

• 48:00Craig S. Smith?

  Yeah. And where, where are you going with this? I mean, what's next on your plate? Are you, are you working on developing Tao

• 48:11Craig S. Smith?

  to, to, to, to broaden it or make it more autonomous or?

• 48:17Craig S. Smith?

  Yeah.

• 48:18Speaker 00

  Yeah, I want to have reasoning L M's for everyone as well. The same thing I mentioned before about how right now, if you want to train a reasoning L M, you kind of need a way to tell right from wrong and correct from incorrect, something a lot, it's a, it's a much more rigorous thing than saying better and worse. So we're thinking a lot these days about how we help our customers come into their own domain and get a model that will not just answer questions well, but actually reason. And so reasoning to help the model answer better and to also give either you or your users insight into why the model made the decision it made. I think that we all want a little more insight into how A I is behaving. And so that's really important to me. The other piece is that the same ideas behind Tao, the idea that, you know, D B R M can help you figure out which data is better and worse and how you steer the model.

• 49:02Speaker 00

  That's also really useful for a bunch of other things you might want to do with A I. Like it's a way to do evaluation without needing any labeled data either. It's kind of the same idea. You have your L M give multiple outputs and you look at which ones are better and which ones are worse. And if your L M is getting better over time. And that's a great way to tell whether you're making progress. So I think there's just a lot of hope of kind of reducing the burden on our users when they want to build, measure, customize A I systems. And just getting them to do the thing that they're good at, kind of coming back to where we started. The idea that on the web, everybody has their own website and they can make of it what they will.

• 49:36Speaker 00

  It shouldn't be that hard to build a website because the hard part is solving a problem with a website or doing something interesting with that website. And I think the same should be true about AI. We should make it really easy to build AI because the hard part should be going and solving your problem with your expertise and making a difference in the world.

• 49:52Craig S. Smith?

  Yeah, yeah, well, I'll leave it there, but I do have one question remaining. The judges, the LLMs that are acting as judges, are you using proprietary models, reasoning models to act as judges?

• 50:07Speaker 00

  So for judges in various contexts, judges show up a lot throughout Databricks. And it's actually a combination of a bunch of things. Some of them are doing prompt engineering on top of closed models. And some of them are actually variants of DBRM or kind of cousins of DBRM that were trained specifically to tell better and worse for a specific kind of judgment. Like, is the output actually factually grounded in the document? And so, you know, it's, it's an interesting landscape right now where if you were to hit the Databricks judges in our APIs, you'd actually see a mix of all sorts of different things, depending on what we found to work well. I'd love it to all be cousins of DBRM at the end of the day. But the most important part for me is, you know, solving customer issues.

• 50:48Speaker 00

  And, you know, if we can't offer something that's better than, you know, prompt engineering, a great model that somebody else has built, I will always err on the side of giving the customers the best thing we have rather than making myself feel good.

• 51:00Craig S. Smith?

  Yeah. Okay, well, let's leave it there. That's fascinating and I'll, I'll watch the progress of Tao and see how it's,

• 51:12Craig S. Smith?

  it becomes adopted or adapted. Okay, Jonathan, that was great. In business, they say you can have better, cheaper, or faster. You only get to pick two. What if you could have all three at the same time? That's exactly what Kohair, Thomson Reuters, and Specialized Bikes have. Since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development.

• 51:45Craig S. Smith?

  And AI needs where you can run any workload in a high availability, consistently high performance environment and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage, and 80% less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is a cloud built for AI and all your biggest workloads.

• 52:27Craig S. Smith?

  Right now, with zero commitment, try OCI for free. Head to oracle.com/eyeonai. Eye on AI, all run together, E Y E O N A I. That's oracle.com/eyeonai.