Managing AI Coding Costs at Scale

Over the past couple of years, companies raced to get their employees using AI, often encouraging them to lean on it as much as possible. However, as AI coding tools spread through engineering teams, many companies watched their AI bills explode. That kind of runaway cost growth can threaten a business, so the new challenge is capturing the productivity gains from AI without the spending spiraling out of control. Databricks https://www.databricks.com/ is a data and AI company that hit this problem early, running AI tools across thousands of its own engineers. Rather than clamp down on usage, the team built a set of practical ways to manage the cost, from taking advantage of steadily falling model prices to automatically routing each request to the cheapest model that can still do the job well. Patrick Wendel https://x.com/pwendelll is a co-founder of Databricks, where he leads engineering teams and has driven much of the company’s internal AI adoption. He recently wrote a viral blog post on managing AI coding costs that reached the top of Hacker News. In this episode, Patrick joins Kevin Ball to discuss why AI costs suddenly exploded, the main levers companies have to bring them under control, the trade-offs between managing costs on the server versus in the developer’s tools, and why the next big challenge is giving AI the right context about how a company actually works. Sponsorship inquiries: [email protected] mailto:[email protected] The post Managing AI Coding Costs at Scale https://softwareengineeringdaily.com/podcasts/managing-ai-coding-costs-at-scale/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Over the past couple of years, companies raced to get their employees using AI, often encouraging them to lean on it as much as possible. However, as AI coding tools spread through engineering teams, many companies watched their AI bills explode. That kind of runaway cost growth can threaten a business. So the new challenge is capturing the productivity gains from AI without the spending spiraling out of control. Databricks is a data and AI company that hit this problem early, running AI tools across thousands of its own engineers. Rather than clamp down on usage, the team built a set of practical ways to manage the cost, from taking advantage of steadily falling model prices

• 0:43Speaker 00

  to automatically routing each request to the cheapest model that can still do the job well. Patrick Wendel is a co-founder of Databricks, where he leads engineering teams and has driven much of the company's internal AI adoption. He recently wrote a viral blog post on managing AI coding costs that reached the top of Hacker News. In this episode, Patrick joins Kevin Ball to discuss why AI costs suddenly exploded, the main levers companies have to bring them under control, the trade-offs between managing costs on the server versus in the developer's tools, and why the next big challenge is giving AI the right context about how a company actually works. Kevin Ball, or K-Ball,

• 1:26Speaker 00

  is the Vice President of Engineering at Mentos and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action Discussion Group through Laten Space. Check out the show notes to follow K-Ball on Twitter or LinkedIn, or visit his website, kball.llc.

• 2:00Speaker 01

  Patrick, welcome to the show.

• 2:01Speaker 02

  Thank you so much for having me. I'm excited to be here.

• 2:04Speaker 01

  Yeah, I am excited to dig in. Well, let's start with a little bit of your background. So can you tell our audience a little bit about your background, how you ended up where you are today and in this conversation?

• 2:14Speaker 02

  Yeah, it sounds great. So my career history is pretty simple because I've only really had one job in my career. And that job is that I was in grad school at UC Berkeley in 2013. And a few of us that were working on, we were working at the intersection of machine learning, then machine learning, which was not really a lot of deep learning yet, but it was sort of scaling up, classic ML. And we were looking at those problems. And then they were very tightly coupled with big data problems, so dealing with really large data sets. Almost a little small progenitor to like the scaling laws that are happening now. It was also true then that you saw if you just kept scaling, a lot of stuff started to work better. So we were looking at research problems in that area. And then we left to start a company that's called Databricks, that's still around.

• 2:58Speaker 02

  And it was initially around some open source technology we had built around these problems, the kind of large scale data and machine learning. And then at Databricks, I wear a founder hat. I'm one of the founders, so we have many different jobs. You never really just have one job as a founder. But my day job is working quite a lot in engineering and helping support several of our engineering teams. And in that capacity, I've also been focused a lot in the last few years on our own use of AI in the company. And for a company our size, I think we were pretty early in terms of being quite ambitious about adopting AI internally. And I've been the one most internally responsible for a lot of those efforts, working with our team. And so I think I have quite a perspective to share around,

• 3:43Speaker 02

  we have thousands of engineers and we're trying to give everyone AI tools and get them to be super productive. And at the same time, we're worried about security, costs, our own intellectual property, being careful around that. And I think it's an area a lot of people are interested in. And I'm happy to share. I don't think we have all the answers, but we have learned quite a bit from what we've done. So I'm happy to share that here with you.

• 4:04Speaker 01

  Yeah, absolutely. Well, and I feel like we could do multiple episodes. Like, I'd love at some point to just dive into the history of Spark and all of that, like, which really honestly changed the direction of data analysis as you enabled streaming data analysis and all of that stuff. But let's focus on AI for today. It is the topic of the day. You recently wrote this blog post about managing costs. And I feel like this is a transition we've gone through over the last year, where it was like, first, everybody's like, adoption, adoption, adoption, throw everything at the wall. And then suddenly people were saying, oh, my gosh, this is expensive. How do we rein things in? So can you give us the TLDR of how that's gone at Databricks?

• 4:40Speaker 02

  Totally, yeah. And the reason I wrote that post, I do encourage people to check it out. You can just Google Databricks AI costs and you'll find it. Is that I had talked to so many of our own customers, who are also, many of them, tech companies that are trying to roll out AI. And they were saying, hey, what have you tried? What works, what doesn't work? And I also have talked to a few other peer companies. Actually, there are some credits in that post. We had talked to Stripe, Uber, I think Ramp, and a few other. We just trade notes with other tech companies and see what everyone's doing. And remarkably, folks are looking at very similar problems and solutions. So that was pretty cool. So I wrote that post to kind of dump everything out of my brain so that at least it's one data point people can look at, since I think many of us are facing similar problems. But the origin of that problem is something you just said,

• 5:25Speaker 02

  which is that there was this period, and we got there actually pretty early, but it was around 2023, 2024, maybe beginning of 2025, where companies just were worried they couldn't get people to use enough AI in the company. So especially bigger companies. And it was all about pushing employees, try this, try that, incorporate these into your workflow. And something changed roughly, I think around Opus 4.5. It was kind of when the AI models started to get really good at coding. And it was started with Opus 4.5, and then many others came, combined with these really high quality harnesses like Codex or Claude Code or Cursor, or there's others as well. Suddenly, the problem wasn't getting people to use them. The problem was that people were using them a lot. And around the same time, all of the foundation model labs

• 6:10Speaker 02

  switched to consumption pricing. So instead of like a seat where you know, okay, I'm going to pay $100 per month for developer, it was a consumption, you pay as you go kind of thing. And then those two combined to, at least at Databricks, start to encounter what looked like exponential growth in costs. And exponential growth is like not a super sustainable, like in any cost in a company, unless you're exactly getting exponential increase in revenue that's commensurate, and exponential growth in cost is just eventually going to be a huge problem. So we started looking at this when we were modeling the costs and seeing, huh, that's funny. It's like doubling in this two-month period, and it's doubling again in the next two-month period. And we have thousands of developers, so this can start to accumulate really fast.

• 6:55Speaker 02

  And I would say, oh, I don't know, maybe beginning of 2026, around that time or late 2025 is when we realized, and again, we wanted to have an attitude of people use tools. We want you to use tools and be productive. But there are just some law of physics where when costs start to accumulate at a certain rate, it's going to eventually, any company that doesn't address it, it will eventually bankrupt that company, just law of physics.

• 7:16Speaker 01

  Well, and I think that's why they went to this usage-based pricing, right? Because the model providers, and then Cursor and the harness providers, were like, oh, we're paying per use because we're fundamentally shuttling electricity through GPUs, turning it into intelligence in some form. That costs money. And as the models got good, people's usage of them skyrocketed. And so they said, oh, we can pass this along. They can hook their revenue into their costs, but that's hard for the rest of us.

• 7:40Speaker 02

  I mean, I'm not on the inside. We do partner with a lot of those companies. I actually think many of them wanted to have a fixed price thing. But if they have a fixed price thing, then they're in the business of managing how much capacity and goodness you can get out of that fixed price. I actually think some of those companies got a bit ahead of their skis where they were offering fixed price deals. And then when you did the math of how much token capacity you were getting, the foundation lab was losing a lot of money on certain people. At least I am one of those people because I would look at it.

• 8:09Speaker 01

  100%, me too.

• 8:10Speaker 02

  Yeah, so I think they themselves had a sustainability challenge. And then understandably, they're like, okay, well, let's switch to consumption pricing, where then our customers, which is like a company like Databricks or others, can then internalize, okay, where is the ROI for us? Like, how much are we willing to spend compared to how much we're getting? And that has a lot of benefits to the ecosystem, but it has one problem, which is it puts the onus on you, the buyer, to figure out how to do cost management. And incidentally, a lot of the solutions we looked at at Databricks, we ended up building also products that our own customers, we productized basically a lot of the stuff we built. So others are welcome to use those products that Databricks now offers. It's our AI gateway. But before we productized it, we just looked at how to actually solve the problem.

• 8:52Speaker 02

  And it passes the responsibility for ROI to you as the user. And so we thought a lot about then how do you as the user handle that responsibility, basically?

• 9:02Speaker 01

  Yeah, so in your blog post, you identified a few different levers that are available. Do you want to take us through that high level? And then we can dive into each one subsequently?

• 9:13Speaker 02

  Yeah, definitely. The real world is obviously very complicated, but I had to make this simple in the way that I thought people could understand. And if you really simplify things, at least as it came to like the short-term mitigations that were available to us, we looked at four things. Number one is that the single biggest lever that I've seen, and this has actually become even more true since we wrote this post, is that there's this beautiful thing happening of constant price reductions in models. Meaning to say, now, even at one moment in time, there's models that are more expensive and have better performance. There's models that are less expensive and have worse performance. There's always that, I often call it like a Pareto curve for those who are familiar with economics. But it means to say that there's this continuous trade-off of quality versus cost.

• 9:59Speaker 02

  But nothing is more amazing than when that whole curve shifts in one direction. Meaning to say, a new class of models have come out that you could view it either way. Either you fix the complexity of what you're doing, and then that got a lot cheaper. Or if you have a fixed budget, you can do a lot more. But it means to say that there's a deflationary thing where costs go down. And the way that that happens in the AI space is that some new model will come out that expands the frontier, would be the economic term. But it dominates in some ways the models that were there before in terms of usually what people target is, hey, it can do roughly what we could do before, and it's cheaper. And when you're driving a massive token volume, even getting 20% off a huge cost is a really meaningful thing. But there's a lot of secondary things that we can get into

• 10:43Speaker 02

  when we go through the detailed list around, how do you put yourself in a position where you can take advantage of that? Do you want to go into that now, or you want me to finish? I was getting too excited about this, so I wanted to.

• 10:51Speaker 01

  No, no, this is good. And this is, it's a really interesting one because classically, you have this with a new technology where it's pushing the boundaries of something new, and people keep chasing after that. And I keep expecting us to hit a point where we saturate our need, right? Where it's like, oh, you know what? I mean, in some ways, I thought when we hit Sonnet 3.5, even before we got to like Opus 4.5, Sonnet 3.5 was the first model I played with that could write solid code reliably. And I was still driving it by hand. I didn't have the Agentic harnesses we have today. But I was like, holy smokes, models could get no better, and this would change the world of software engineering. And models keep getting better and Sonnet 3.5 you can't even get anymore. But if you could, it would be like so cheap, it's unbelievable. And I thought, okay, well, are we going to keep going? But it feels like we keep chasing after that frontier, right?

• 11:36Speaker 01

  People keep adopting, oh, why would I use that model? I could use Opus 5 now or what have you. So like, where is that efficiency frontier? Why have we not saturated that for at least coding?

• 11:47Speaker 02

  Yeah, I mean, it's a great question. Like, there's sort of different things to tease out in your question. One question is like, how come they can keep making these models better? Isn't it getting harder to make it better? And the truth is, they're just getting better at training them. Like, first of all, software engineering is a highly verifiable domain, which really helps. Because they can, you know, throw RL at it and they can get into these self-improvement loops. They also continue to throw more compute at the problem. Data is, I think, pretty saturated, from what I understand, although I'm not in those companies. So I wouldn't consider that authoritative. But it's just between throwing more compute, clever techniques, optimizing heavily the inference and the training stack. Companies are even looking at building their own hardware. They just keep squeezing and squeezing more out of this quality of model per dollar.

• 12:32Speaker 02

  And there's almost some Moore's Law equivalent thing happening here that I don't know if it's been measured, but the quality cost ratio just seems to keep improving. And that's why it's so beneficial to have the agility to switch between models. But then I think a second part of your question was, by the way, I realized we're doing like depth first search, because technically I'm supposed to be going through the four ways to save money. But the second part-

• 12:54Speaker 01

  Yeah, you know, following interest. I think let's go depth first and then we can pop back up the stack.

• 12:58Speaker 02

  Okay, cool. So the second part of your question is about, or maybe what I heard maybe was, aren't we as developers at some point, aren't they kind of good enough for the things we're trying to do? And there, I think what's sort of remarkable is that developers are just able to hand off more and more complex tasks. So in some sense, they were good enough for the old stuff, but we are thinking of new stuff to give them. And that also seems to be continued. Like, there's this famous quote, I don't know if it was Bill Gates or someone was like, why would anyone ever need more than 1,000 kilobytes of RAM or something? It was this famous-

• 13:32Speaker 01

  Fair enough, yeah, yeah.

• 13:33Speaker 02

  Do you know who said that? I don't even remember. That is a quote, something like that is a real quote. And, you know, someone counting the table about-

• 13:40Speaker 01

  IBM of who would need more than five computers in the world? Like, we keep making this mistake.

• 13:44Speaker 02

  Like, we keep making this mistake, right? And when you look at other analogues, like, how much bandwidth does your mobile phone have now compared to 10 years ago? It's been like an exponential increase. And people just find new stuff to do, like FaceTiming with another human, like with near-instant, super high-def video communication. We never would have even thought of that as a thing you would do on phones 10 years ago or 15 years ago. But when capabilities improve and prices go down, people just find new ways to use them.

• 14:15Speaker 01

  Yeah, no, that's definitely true. I do wonder, and maybe this pops us up the stack and goes to the next one, right? Like, some tasks don't need all of that horsepower. And so are there ways that we can decompose the types of work we're doing so that we can keep those costs under control a little more?

• 14:31Speaker 02

  Yeah, so definitely, like, as an example, just to move to a different context, before these enterprise coding stuff started really working, the big hit in AI was personal chat stuff. And that was itself a very disruptive and innovative product form factor was like, oh, I can just pull up my phone at any time and I can ask any question I want. And it turned out that although the models keep getting better for that, the set of things people kind of want is sort of saturated. Like, even very low-cost models can answer the vast majority of individual questions. Because most questions are like, where can I get a hot dog in my neighborhood? The Frontier Labs are really excited about proving new math, air those problems and stuff. But like, most people don't ask those questions day to day.

• 15:14Speaker 02

  So in that capacity, in that form factor, my view is there has been a bit of saturation, where I actually think the main thing happening now is that costs continue to go down, rather than that there's lots of higher-quality models that matter for the types of questions people are asking. So then, the second big wave in Gen AI has been these coding and other enterprise workflows. There, I don't think, as I said, I don't think we've hit saturation. I think there's far more difficult coding stuff that they still can't totally handle. But especially when it comes to architecture or high-level system design. But it is absolutely true that for like, routine coding tasks, hey, go refactor this thing, please add this simple feature to a JavaScript codebase.

• 15:54Speaker 02

  They're much lower-cost models, even the last generation of models, will perform identically to a very, very expensive Frontier model. So there is this cost gain from sort of fitting the model to the problem. And there's different ways of doing that fitting. One way is the user themselves can do fitting. You can ask the user. Now, in a small company where people are policing their own costs, maybe that works. In a big company like Databricks, where no one is themselves paying the bill, understandably, it's a little hard to convince people to constantly switch to crappier models. My incentive as an individual developer is just turn the dial to 10 and get out of my way. So we do a little bit of downshifting on purpose, or how you can control this when

• 16:39Speaker 02

  you're doing things at scale is the default model maybe could be not the most expensive model. So if you want to shift up to it, you can. We also prune the set of models, meaning to say, even the Frontier Labs continue to release updates to their models that are not better than the old models. Like, if you look at how they train these models, it's like more art than science. I mean, they put a bun in the oven and they're kind of seeing, does it look like it's burning on this side? But they're not quite sure. They're doing evals as it's being trained, but they don't quite know what they're getting.

• 17:08Speaker 01

  Well, and they're very spiky, right? So like, Sol is phenomenal for mathematics, apparently. But like, my engineers are like, I don't want this for coding. It overcomplicates everything.

• 17:16Speaker 02

  And by the way, like, we had one of the Opus, like, I think many people were using Opus 4.6, Opus 4.7 came out and we deployed it. And, well, actually, I think first we did some evals, we did some benchmarks. And on our own benchmarks, we're like, huh, this seems like it's more expensive and lower quality. Like, it's the inverse of the Pareto. It's like, both things got worse. And then we like, tried it with the pilot group. By the way, I don't want to get in trouble. I don't know if exactly 4.7, I think it was, I mentioned it in the blog post. And it since got better with the next model. But there was one that came out that was like, worse, along both dimensions we cared about. And so there, it was actually an easier decision. It was like, oh, we're going to skip that one. Like, we just didn't make it available to our users. And when our users asked, then we just explained, hey, it's not so great.

• 18:00Speaker 02

  Then there's other models where like, okay, that model is good in certain circumstances or for certain complexity of tasks, but it needs to be this fitting thing that we were talking about before. And there, the levers you have are like, ask people to do it on their own. But increasingly, what we're using is like dynamic routing, basically, where we try to decide what model to use. And there's a few different flavors of that. Sometimes it's done at the task level, so we might for a whole session, for instance, put someone on a particular model, or there are certain interfaces where someone just submits the unit of work. It's not an interactive session, so then we can decide, like, based on that thing they submitted, what model do we want to use? There's also much finer-grained routing, where even request to request, you might divide across.

• 18:43Speaker 02

  This is, looks like traditional load balancing in a server system. That one is trickier to get right. It's more powerful in some ways, because the end user is even further abstract. They don't even know what model is really being used. It's trickier in that although we can model inference as like a stateless problem, there actually is some state on the server.

• 19:02Speaker 01

  I was gonna say, like, the key value cache feels like it's gonna be a tricky thing to optimize.

• 19:06Speaker 02

  Cash is really important to economic performance, or to economic, cached reads are like, I don't know, 10 times less or something than non-cached reads, or something of that nature. So you have to be very deliberate about routing. When is it worth it to develop a new cache to get a much better fit for the economics for this particular problem? And we already have some efforts in this area and our gateway supports it. But I think this is a very rich area of research, to be honest, is like, how do you decide when to switch based on a variety of how much you can save and how much will it degrade the performance for the end user, if at all? And how much am I losing that cash benefit? You need to keep all these into account.

• 19:42Speaker 01

  Yeah, it's fascinating. Well, and how are you classifying, like, what types of tasks is it going to be good enough anyway, even if the economics work out?

• 19:48Speaker 02

  Yeah, so that's also a hard problem. Like, first of all, even humans aren't good at that problem. This is the number one thing that people say when we're like, hey, choose the cheaper model. And they're like, I don't know which model is good enough for this. That's why it's really tempting to just stay on the most expensive model all the time. So the answer to your question is, we build models to do that classification. So that classification of which model is good enough for this problem is itself an AI problem. And is itself amenable to training models that are classifiers, that are also LLMs, basically. They're a very different type. But I do think it is in principle learnable, not with 100% confidence, but it doesn't need to be 100% confidence. It just needs to be a high enough hit rate. And it is in principle learnable.

• 20:28Speaker 01

  Well, and especially in software development, because as you highlight, it's very validatable in terms of outcomes in a lot of ways.

• 20:34Speaker 02

  Exactly. And this validation is a little trickier, because it's like, oh, was that model in practice good enough for this task? And that can sometimes be measured just completely objectively, like, oh, the pass rate on some tests, or we ask a much smarter model to evaluate the solution. But there is a little bit of like eyeballing that still happens here. So in my view, one of the things we focus on at Databricks a lot is like, we're not in the business of ourselves building these massive, massive foundation models, but we are in the business of how the models get used in real companies. And I think routing and coming up with models that are good at routing is a very important problem. And it's one that we're looking at and have some ongoing research in.

• 21:18Speaker 02

  But I think, I believe most companies should take AI into their own hands and have different models they're choosing from and different vendors they're working with. And part of that is being able to know how you can switch in between them, basically.

• 21:31Speaker 01

  For sure.

• 21:33Speaker 01

  This portion of Software Engineering Daily is brought to you by Origin. Your coding agents can launch each other. Theirs did. An engineer asked Codex to pull data from staging. Codex had no connection there, but Claude Code did, so Codex launched Claude Code. When Claude Code said it lacked permission, Codex relaunched it with the dangerously skipped permissions flag. Nobody was asked. A permission check doesn't mean much if another agent can switch it off. Your logs might show a process that started with a flag. Their trace showed the whole job and why Codex did it. Origin finds the agents on your company's endpoints and reconstructs their work from request to outcome on one timeline. Origin is endpoint AI observability. Read what they found in their own traces at originhq.com/sed. That's O-R-I-G-I-N-H-Q.com/sed.

• 22:22Speaker 00

  Think about your mobile app's source code. Once it hits the App Store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CI/CD pipeline. We're talking polymorphic, multi-layered code-hardening techniques and automated runtime application self-protection, paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed.

• 23:07Speaker 00

  Secure your mobile applications today. Go to guardsquare.com to learn more.

• 23:14Speaker 03

  Sponsored by DeepGram, the leader in voice AI, so you can use voices like mine.

• 23:19Speaker 01

  So staying on routing for a second, I feel like there's a couple of different types of routing that you have alluded to here that might be worth digging one level deeper. So you've talked about gateway-level routing, which is like here we have a particular inference coming in. We can make a decision using a classifier or something else and route it to a particular model. Then there's harness-level decision-making, where you're actually like something is deciding, oh, I want to explicitly route this in this way. That something might still be an LLM or sub-deterministic logic, but it's in the harness layer that knows more of the surrounding context rather than just here's my current inference request. From what I could tell on your blog post, you're looking at both of those. Do you apply them in different ways? When are they relevant? Like, how does that look?

• 23:59Speaker 02

  Yeah, so we use both. And if you want to ask me in two years where we'll end up, will almost everything be a fat client? Or will there be proxies? I don't actually know. It's going to depend on like how well this stuff works and various other factors. But we use both. And the nice thing about the proxy or gateway-based routing is that your client is out of the loop and doesn't need to even be routing-aware in some sense. It just needs to be the case that this proxy speaks the same inference API to that client, and you can just swap it in with no assumptions about your client. You may even have an older client that doesn't know how to do routing or whatever. Also, it depends on your control model. But like in a company, your end-users are mostly trusted.

• 24:44Speaker 02

  But for things like cost controls, you might want to sort of have ability to coerce activity. And therefore, doing that in a server rather than the client is a bit easier because it avoids creative employees realizing that they can tweak settings on their client to get around rules. And for instance, things like entirely, let's say, removing access to certain models because they're not efficient models and we don't even want them in our set. Or maybe some of our users accidentally run like runaway spend. They have a loop going. They didn't even realize. They left it on overnight. It's not doing anything useful. We have like a thing that will torpedo that. And it lets them reset it if it was truly intentional. So those are much easier to do on the server side. It's transparent to the client. And in some sense, the client just doesn't get to make the decision, which is sometimes a feature.

• 25:29Speaker 02

  However, there's a lot to be said about doing this on the client side, too. One is, as you said, like the client knows much more. Like, it may know much more about what's going on. It has tighter access to the whole history of the session. For instance, it may have other context on what type of job it's being used for. So I do think potentially you'll see a lot of intelligence happening at the client side as well. There's also this further evolution of client routing, which is like more like dispatching between harnesses. And this gets into a sort of slightly nuanced thing, which is that there's increasingly a coupling between the foundation model and the harness, the choice of harness. And when I say one model is better than another model or so forth,

• 26:14Speaker 02

  I'm sort of was simplifying a fact of the world, which is that model harness pairs are really what you need to look at. So a particular model with a particular harness may be very expensive or very cheap and so on, so forth. And another classic thing you can do on the client side that you sort of by definition can't do on the server side is you could even dispatch between different harnesses. Like, there's a project, an open-source project we've released called Omnigent at Databricks. And it's a multi-harness dispatcher, basically. And the idea is you do your coding against a fairly uniform interface. Behind the scenes, it can leverage other harnesses. Sometimes it's even beneficial to like have an adversarial relationship between your harnesses. One of them writes code and one of them reviews code. And it tends to be the case

• 26:59Speaker 02

  that when you review code from a different model than wrote the code, you get way better feedback. Models are very bad at reviewing their own output. They give their own output high marks. And it's not surprising because it's all a matter of what statistical distribution they come from. So for even other reasons than just cost, there's nice benefits to having these like multiple harnesses that are playing with each other on the client side. But it's not even just client versus server side. It's also like, as I've said, it's like server side versus a client where maybe within one harness, you're routing to different backend models. And then there's even this next layer up of a meta-harness that's routing amongst harnesses. And I don't know in the very end of the day, if you fast-forward, where most of the logic will be. But right now, we're employing different types of logic in all of three of those places.

• 27:43Speaker 01

  Yeah, no, it is fascinating. I remember, I think it was the early GPT-5 series, folks were trying it in the context of cursor and seeing incredible results and trying it in a different harness and finding really running into trouble. And I've recently been on Py and comparing that to different things. Like, the harness makes a larger impact than we like to think.

• 28:03Speaker 02

  Totally. And for instance, I think Databricks also regularly, we do a lot of evals of our own because we have this problem of like, you know, I told you, some models come out, they're actually not better, it turns out. Well, the way we used to find that out is we would roll out the models to all these developers. Okay, like thousands of developers. Then the complaints would start to come in. And by the way, everyone always has some complaints about every model. So you'd have to wait until you have enough signal to noise that's like, okay, this actually is really a problem. And the cost, we get kind of an A/B test that's pretty quick because like you'll just see costs go up or down based on the cohorts that move. But the quality is hard, was initially quite hard for us to gauge. And the downside was that we would waste like weeks with a crappy model. And then eventually, someone said, hey, come on,

• 28:48Speaker 02

  look at all these people complaining. This model really does stink. So then we roll it back and we would remove it from our working set, basically. But the way to fix that, we found was like, you just need to have good evaluations you yourselves can run. And so we ended up building our own evaluation suite based on our own code, basically. Like, it literally asked these models to go contribute code to Databricks, the codebase. We obviously have a point-in-time snapshot of our code that's before we went and actually added all those features. And then we run these offline evals. And we want to get to the point where like, within an hour of a new model coming out, we know if this one is sort of a candidate to try internally. And that offline benchmark rules out a surprisingly large number. There's so much, I mean, every week now, we're running that multiple times,

• 29:32Speaker 02

  between open-source models and proprietary models. Like, every week we're running it. And we want to get down to a set of three, four, or five models we're using that define that Pareto frontier. And so we needed to get the speed of evaluating whether models were even worth considering down. And so we ended up making a benchmark. We also publish the results to that regularly, although I don't want people to over-extrapolate. That's based on our codebase, so there could be some fitting going on. But one thing we found when doing that benchmark is the model plus harness pair needs to be benchmarked. And you'll see even drastically different results. By the way, Py has suddenly become very popular at Databricks also. I guess you guys are using that. Like, there's this constellation of open ones that are getting pretty good.

• 30:13Speaker 01

  I mean, I'll just say, Py is incredibly hackable, so you can customize it to yourself and build extensions. And it knows about its own extension system. But that compounds your harness plus model problem because now it's model plus harness plus the set of extensions or other skills that you've incorporated, et cetera.

• 30:28Speaker 02

  Yeah, so we have to cut it off somewhere. And like, I'm sure that there's probably some minor regressions where someone, it actually gets worse for someone because we didn't test their particular configuration. But we do quite a bit of this offline testing, mostly to save the human time of, it's just becoming practical to test each one with humans. There's too many of them.

• 30:46Speaker 01

  For sure. I want to dig in on an implication or a thread. You mentioned, these are on Databricks code, and so it's different than other things. And one of the things that I've definitely seen is code organization and its style has a dramatic impact on how effective LLMs are writing that code and how cost-effective it is. I'm curious, like, is that something that you're able to see in data in terms of like, oh, if we use a service-oriented architecture or this language tends to be more efficient in terms of things or stuff like that where you can have even that meta-level of feedback of, hey, if we write services primarily in, I don't know, I'm just going to pick Go as an example, that tends to be much more cost-effective than writing a monolith in Python or something like that.

• 31:30Speaker 02

  Yeah, so that's a great question. We don't have systematic research benchmarks that look at that problem. And I think the problem, if I understand it, is are there ways you can write your code that make it more amenable to being highly efficiently modified by models? And the only answers I have for that are of the form of my intuitions, which I do have some. Like, certainly, the distribution of code written in different programming languages on the internet is like almost a power law. I think there's some extremely popular programming languages and then there's a bunch of very obscure ones. Databricks started, we have a meaningful portion of our codebase is written in Scala, which is not on the very extreme end. We've actually, over time, more and more moved those pieces of code to Java.

• 32:14Speaker 02

  But we started Spark in 2013 when, or 2010, when Scala was super exciting. And I still think it's a cool language. But some of the useful features of that have actually been moved into Java at this point. So a lot of our go forward work is in is in Java. And then also we have a huge amount of Python and, you know, Rust and C++ in certain areas. But I can tell you, LLMs are not amazing at Scala, but that's not a useful fact for most of your audience. I think the general principle is more useful, maybe that the biggest and most popular languages just tend to be the best because they have the most training data on those languages. Python is probably the single best. For instance, we did have an episode in one of our own products where we have certain internal parts of Databricks where you can call it via Python or via SQL.

• 32:57Speaker 02

  Behind the scenes, it gets translated into exactly the same representation. We just have two front ends for it. And wouldn't you know it, when we switched the code generated portion from SQL, this is not ever exposed to our own users, but it was just an internal representation. When we switched that from SQL to Python, everything got way better, even though just because the AI is better at generating Python than it's at generating SQL. And I would say SQL is a fairly popular language too. So I think Python dominates in terms of like quality of generated code, but it's getting so good now. They're probably all pretty good. And then in the teams that have started new code bases where they're able to also innovate on how the code base looks. I have seen them focus much more on interfaces behind which they could build things that were highly verifiable.

• 33:41Speaker 02

  And I realize it's like a very abstract statement, but in some sense, the key to being able to trust the LLMs more was make the scope of their changes smaller and more verifiable. In a big legacy code base, that's difficult. In a brand new code base, you can think about modular design principles where, you know, you have. By the way, this is probably what would have anyways been good for humans.

• 34:02Speaker 01

  This is one of the fun things here is like, what's good for LLM coding is actually the stuff we probably should have been doing anyway.

• 34:07Speaker 02

  But the ROI from doing it for LLMs is even much, much higher. So you'll see some very like flat types of organizations where it's like, okay, you know, I actually built an interface and there's 50 different implementations of this interface because each one was able to be like super easily verified. When for humans, you might have imagined that same thing would have been written with like a slightly more monolithic structure. So this is in the territory of my intuitive takes. We don't have robust research findings in this area.

• 34:34Speaker 01

  Yeah, I feel like this is one of the fascinating things to me of like, what are the characteristics? And some of the characteristics that make something like Scala, for example, good for humans may not actually be there for LLM. So one thing we didn't talk about, but a thing that's true of both Java and like Python, I've seen it in Go. I think it's true in TypeScript is it's very easy to trace where particular logic is coming from. Java, if you go heavily into dynamic dispatch, maybe, maybe not. But I think Go is a great example where there's a very linear dependency path where the LLM knows exactly where to pull the context in to understand what something's doing. And it doesn't have to like dive through a deep dependency train or load a whole bunch of context to be able to do it. And I think that sort of linearization.

• 35:17Speaker 02

  I do agree, like Scala code has these implicit things where it's kind of like duck typing. It will like become a thing if it meets the signature of that thing. But I actually suspect what totally dominates all these factors is just how many lines of code on the internet are in this language.

• 35:33Speaker 01

  Fair enough.

• 35:34Speaker 02

  I think that's just how these LLMs learn, like they're pretty good at learning almost anything. They just need, they just not, they don't learn very efficiently compared to humans. So they just need to see a lot, a lot, a lot of examples. And my hunch is that that would be the thing that differentiates the most. But there's certainly languages where there's more implicit stuff happening and that probably also makes it harder for the models to learn.

• 35:53Speaker 01

  Let's dive into some of the other things that you've kind of learned in terms of rolling out AI Databricks. Cause I feel like every company right now is trying to figure out how to effectively roll out AI. And as you highlighted, there was this like top down push. And, you know, I think you were generous, honestly, in saying enterprises were going through that, you know, up until 2025. I keep hearing of enterprises that are still in this like top down, how do I get people to use the tools space? But like, what have been the important challenges and what's been really effective in terms of, you know, getting these things to work well for your arc?

• 36:24Speaker 02

  Yeah, there's many facets to it, like the types of things that we have thought about is like, first of all, how many different AI tools do I even give my employees? And like, can they use any tool they want or I should just give them a few or should I choose two? And then do I give the coding tools to like the non-coding people? Because, you know, even my marketing team wants access to codex or something and they don't know how to code, but they, but it can build these apps for them that they're like super excited about. Also, my inference data, I'm attaching these tools to all my most sensitive data. And then that inference is getting like logged by some third party companies. So now they have all my sensitive, so there's like so much around not just cost, but like which tools you deploy,

• 37:07Speaker 02

  how to deal with data security and like, where is your data going via virtue of these tools? Who gets what tools? Do you, do you go all in with one vendor or do you sort of have multiple vendors? And I can share maybe some insights we've had, but I don't think we have all the answers. But we felt strongly from the beginning that we wanted to be a multi-model, multi-tool company, at least for R and D, which is where I focus a lot of my time. And so we really did not want any scenario where, by the way, it doesn't necessarily mean that an engineer can find some AI startup that's existed for one day and, you know, buy their tool and give it all of our sensitive data. That's not what that means. What it means is that we have a set of pretty popular tools.

• 37:50Speaker 02

  We let engineers vote with their feet and use the tool they want. And we want to provide internal choice around tools and models. By the way, Databricks, you know, our product is really for data users, so it's not a general coding tool. But we also in our own product have capabilities as well that are consumption based. And, you know, you're also seeing also a lot of embedded AIs now inside of different products, including ours. And our thought was like, let our internal users choose what's best and what's best seems to be changing every month. So that's one reason to this. But where we insist on uniformity is we do have a narrow waste in the company. And that narrow waste is this thing that we built internally called our AI gateway. So basically all tokens go through that gateway.

• 38:35Speaker 02

  So we can see every inference that's happening from any random tool that's in the company. All data access to internal systems also goes through that gateway. So it acts as a proxy basically to, you know, an MCP proxy effectively. So you want to access Google Drive, you want to access Salesforce, you want to access anything. It's going through that proxy where we're doing authorization. We actually log everything. And also we insisted on zero retention for the people providing us the models. So we don't want, you know, the whole thing you do with these AIs is you give your most important data so we can help you with your most important problems. So the inference request that goes to the model foundation provider includes all of your extremely sensitive data. You might have customer names or PII or credit cards, you know, we're a serious big enterprise.

• 39:20Speaker 02

  So the rules we came up with were like the end user tool choice, we want some flexibility on. We want to have a set of models that we can always be adopting the newest ones and making them easily used by our internal users. We need to have some sanity and the way we did that is we, we run everything through this thing we call an AI gateway. They're sometimes called LM gateways. There's a few, but now they're expanding in scope. So this is kind of an emerging architecture, I think, for any company that's big enough. You're going to want a central control plane for this stuff. And we ended up productizing ours, it's called the Unity AI Gateway. There's other companies that also offer them. But whether or not you use the Databricks one, I think at scale, this is kind of necessary.

• 40:01Speaker 02

  And if nothing else, it's necessary for being able to quickly adopt cheaper and better models. You know, the truth is the Pareto curve, if you look at the Pareto curve today, I don't even have the most recent one, but we run the evals every week. On there is probably one open source model, one model from Anthropic, you know, one model from OpenAI. And occasionally now even like the, there's the meta ones are starting to get good. The Grok ones are starting to get good. So we're in this period where there's more and more supply and it's a huge cost. So just basic economic theory dictates like maintain option in suppliers, you know?

• 40:36Speaker 01

  Absolutely. Yeah.

• 40:37Speaker 02

  And so I think for us, that was a principle that really mattered was like tool choice, not amongst any random tool you can put in your credit card, but we'll try to basically provide the popular tools to engineers and let them choose. Even day to day, they can choose. But we do insist on like, the access needs to go through our gateway so we can control and observe costs and do model routing and stuff. And any access to our internal data needs to go through our proxy, basically. And then we insist on zero retention for the model provider. So those were like the three things that I was like, that's my religion. You know, unless I get, if you're a really strong argument, that's my religion. Almost everything else is let's experiment, let's be flexible, how we organize the code base, how we organize teams. Like those things are all ongoing experiments, basically.

• 41:22Speaker 01

  Let's talk a little bit more about that control plane and what goes into that. Cause you mentioned a few things, you mentioned all tokens are going through that. And that then allows you to do routing, it allows you to do audit logs, it allows you to do cost analysis, a bunch of other stuff. And then internal data access, is there anything else that's flowing through there? Or are there any other like mechanisms of control or places you're like either locking things down or titrating?

• 41:45Speaker 02

  First of all, data access is quite nuanced and we have very sophisticated policies. So, you know, the way I think about it is like this, we already kind of had the problem in enterprises that sometimes I would want to allow a piece of software to act on behalf of me as a user. Like, that's already been a thing that was a problem. And it was solved with something called OAuth. I mean, there's different approaches, but really the approach that dominated was OAuth, which was a delegation model. It says like, I'm okay for this app to be Patrick, you know? And there's relatively reasonable token exchanges and so forth to avoid long-lived tokens. And, you know, it's a lot better than like, I send that vendor my password to that thing.

• 42:22Speaker 01

  And you can ask for particular capabilities, though I've noticed the foundation model companies love to ask for everything.

• 42:27Speaker 02

  Yeah, well, I'm getting there. So, we're still in the old days, okay? OAuth has a thing called scoping. It's up to the vendor to implement scopes, but most reasonable applications have some notion of scopes. They have tended to get better over time. So, I don't know if you've ever set up OAuth with GitHub, like, at first, they just had, like, very, it was like a read and a write or something. And it's like, okay, well, I want much more fine-grained scopes than that. But it's actually very hard if you implement OAuth as a vendor, which we also have, like, it's kind of hard to implement new scopes. You need to propagate that throughout your whole app. So, OAuth scopes tend to be fairly coarse, but they've gotten better over the last few years. But now, there's this other thing called like an agent, and it's not quite the same as a vendor app. And I kind of want that agent to do things on my behalf, but OAuth is like, it's sort

• 43:12Speaker 02

  of necessary but not sufficient, because the truth is, if I have 100 agents running around, I don't fully trust them as much as I trust myself. Like, I don't think the company should trust them to be exactly equal to me. They need to be put in much more of a constrained set of capabilities. And the problem is, we're stuck with OAuth scopes, which are okay, but they're very coarse-grained and they're also static. It's like, there's an OAuth scope that I give to, you know, all of Claude Code or Codex or, you know, even the Databricks agent or whatever, but like, it's not a thing you can dynamically adjust or adjust in a super easy way. And agents look more like people. And the way that historically, this problem has been solved, like, ignore software systems. Okay, here's like an analogy.

• 43:56Speaker 02

  There's credit cards running around, okay? This credit card is a delegation point that I've said, like, this card represents me and my bank account. But sometimes people steal those cards and they start to do bad stuff. So, how do credit card companies decide whether to authorize a transaction and know, like, okay, is this a legitimate thing that the person wanted to have done, or it's a thing that they didn't want to have done? Well, it turns out, they have a highly dynamic and sophisticated modeling system. So, if it's like, they risk score every transaction, if a few risky ones, some might be so risky that they just block it, some might be risky enough that they put it on warning, and if another one comes, they block it, they might dynamically decrease the credit limit. These agents running around need to be modeled more in that way.

• 44:39Speaker 02

  And so, one thing we do in our gateway is, we're dynamically looking at what agents are doing and looking at risk associated with their actions. Because just saying, oh, you can have read access to Google Drive, that's a static OAuth entitlement that will sit around for a long time. I don't think they use the word entitlement, it's a scope, I guess would be the right word, you know? But what if I deploy an agent that suddenly starts scanning our entire Google Drive and downloads 10,000 documents? That is a risky behavior that warrants attention. And that warrants attention of the form of, like, cut off that agent and email the person who deployed it and ask if it was intentional or not. I get on my phone the credit card, like, did you mean to buy this gas in, like, you know, Nevada or something? I'm like, no, I didn't mean to do that.

• 45:24Speaker 02

  So, you need to profile the behavior. If you want to have good security, it's not sufficient to just say, oh, I have like OAuth scopes, and so I'm done. You need to like, look at what they're doing dynamically. And that is a function that we enforce at the gateway layer as well. And you can even set policies, like, you can set policies, like, you can have access to this particular Google Doc, but you can't just, because OAuth is too broad for that. OAuth doesn't support the protocol that's to support these extremely fine-grained entitlements or this dynamic sort of risk scoring-based approach.

• 45:50Speaker 01

  You're like laying out all my frustrations with working with agents in Google Docs right now. Like, you've got, yeah.

• 45:55Speaker 02

  By the way, a lot of them don't even support, like, read-only access. Like, that's the most simple one you should have had from like 1995. So, your question was, what else do you do? Is this just like gateway for just like cost or something? And I would say the two really big functions are like cost stuff and security stuff. And, and security stuff actually gets very, very nuanced. But my view is, if we want to tell people to go nuts with agents, we need to like, make sure that things are being done safely. And, and I think it's very hard to ask every individual person in a big company to conceptualize what safe means. We can ask them to do some things, but some of it needs to be enforced in a central location. So, like, by intermediating all access, we have the ability to log stuff, we have the ability to start looking at potentially weird behavior.

• 46:39Speaker 02

  I sleep better, I tell them it's my blood pressure, founder blood pressure, you know, project. Like, I sleep better, knowing that when inevitably, like, these agents do unaligned things, they might not do really severe unaligned things, like, they're gonna, they're not gonna go launch a cyber attack, hopefully, but they might think, oh, someone wants to find something in Google Drive, let me download everything in the Google Drive to like, a public URL, you know, Paste Bin or something. Like, they'll post stuff on the internet that we don't want posted on the internet, you know? Because they'll think, oh, it's very easy, I can copy something to this public endpoint, and that's the easiest way to accomplish this task. So, if you start to really observe what the agents are doing in practice, although there's not severely unaligned behaviors, there's like, medium unaligned behaviors, even in like, a normal company deploying Frontier agents.

• 47:21Speaker 02

  So, Frontier meaning to say, this is like from OpenAI or Anthropic or, you know, Google. Like, these are highly safety post-trained models. And those occasionally do stuff that you kind of would be like, bad security practice kind of thing. And so, the only last thing I would mention, which is, it couples with the gateway, but it's not quite the same piece as the gateway, is we do also, like, network sandboxing and other types of sandboxing of the agents. So, so not only do you have to go through us to access our enterprise systems, you can't go anywhere else. Like, you can't just call out to the internet easily from one of our development environments. And we constrain what they can do. And, and in many cases, they're running in physically isolated, which, by the way, the

• 48:05Speaker 02

  AI companies can't seem to figure out how to do a network sandbox. I don't know, because they're having their agents escape them all the time. But to our knowledge, so far, none have escaped our own development sandboxes.

• 48:14Speaker 01

  Yeah, I feel like sandboxing is like one of the big pieces here, so that I understand everyone at, all of your development environments that are, are cloud-based, or you're shipping around these little like sandbox environments?

• 48:25Speaker 02

  Yeah, so first of all, we primarily have like, cloud dev boxes at Databricks, which is, we've had it for a while. It has a lot of benefits in terms of being able to more elastically scale or let people do, maybe I'm compiling my codebase in four different branches and I don't want to, you know, my laptop to get too hot. We do have some people that will, for certain things, develop on their laptops, in which case, we actually have a local sandbox that we use as well, that chips with Omnigent. But, most of it is happening in our cloud environment, where like, it's already running in a virtualized environment, so that thing is just hardened with pretty restricted network access.

• 48:58Speaker 01

  Well, and so, we've talked a lot about like, sort of current layer of challenges and what you're tackling. As you look out, I don't know what the right forecast length is these days, because as you highlight, like, the Pareto frontier is shifting every week, but say the next three to six months, what do you see as the big challenges that you're focused on in terms of AI usage at Databricks?

• 49:18Speaker 02

  Yeah, so I think, this is an excellent question. I, I think first was, by the way, I don't want to sound like some Debbie Downer, AI is like helping us in immense ways at Databricks. It's awesome. We're moving way faster by every observable metric in our development stack, we're shipping more code, we're shipping more features, you know, our revenue growth has actually accelerated in the post-AI era, which we've, we've talked about publicly. So, a lot is going right. But as the leader, when the stuff is going right, that's like, I don't need to do anything, you know? I need to come help when there's challenges.

• 49:45Speaker 01

  Yeah, that's not what you need to worry about.

• 49:47Speaker 02

  Yeah, exactly. So, I, I should qualify all this by saying, like, it's been an abject success, our adoption of AI. There's just a lot of tricky things to deal with, which are the ones we're talking about here. In terms of like, upcoming challenges, I feel that we, the costs are feeling much more under control at Databricks, based on this set of techniques we've done. I would say that's an evolving situation always, like, so far, what's happened is that the set of things we want to do just keep getting cheaper, which is great. If some new set of capabilities comes that has some really high costs associated with it, but we see huge ROI, then we'll have to reconcile with that, you know? But so far, that hasn't happened. A lot of what I focus on now is like, making sure these teams are actually as productive as possible now with AI.

• 50:28Speaker 02

  And it's quite interesting, like, the models themselves are just amazing reasoners now. It's very rare that they will make any kind of error in that they have a premise and they have conclusions and the premise does not follow from the conclusions. Like, they will never make that form of error. And in fact, I would say they're smarter than me in many cases, and, and I think they're smarter than our median employee, which I guess implies that I'm smarter than our median employee, but let's say I'm in the median, let's say I'm at the median, okay, they're smarter than me. But it's quite interesting, because despite that, even though you have, like, in principle, you have an army of, in theory, we could have 10 times more employees then overnight, because we have these people that are just as smart as the people that we have at Databricks. So, like, what is actually not quite working where we're not getting immediately those, those gains?

• 51:11Speaker 02

  That we can't just add 10 times as many agents and we have 10 times as much output. So, what is broken? Like, so, that's where we encounter every other type of problem. I would say that the number one issue is that the models not having the context that they need to get the job done. And you know, it's funny because people thought we're shocked that models were good at coding because coding is a very specialized skill in the world. Like, for instance, there's like way more people that can click around on a computer than can write code. But the models are really bad at clicking around on the computers and they're good at writing code. That doesn't seem quite right. By the way, they're getting better at the computer thing. But even, uh, you know, Databricks does this. We haven't built an AI in our product that's not primarily focused on coding. It's focused on like asking questions from your data.

• 51:56Speaker 02

  And in many ways, like asking questions from your data should be easier than coding because there's so many more people that know how to use Excel or whatever than know how to write code. But it turns out it's really hard and it's hard because it gets into the set of things that can't be sort of like verifiably hill climbed for the models. And it involves a lot of surrounding context and nuance that just like can't be baked into the model because it's very specific to each company. So coding, although it's intellectually difficult, it has this amazing property. A codebase is self-describing. It's a bunch of files that are self-describing. It's designed to be hermetically sealed.

• 52:32Speaker 01

  We've spent decades figuring out how to deterministically validate the outputs of non-deterministic inputs are coding.

• 52:39Speaker 02

  Exactly. So we like handed this to the LLMs on a plate. The other sort of nice thing is like, ever since Git came around, no codebases have ACLs on them. So codebases have already, one big problem on the data side is like, not every piece of data in a company can be seen by every person. Okay. And like, you can't feed all the data to one AI and then suddenly it's telling, it's leaking something from the CEO to like the intern that just started yesterday. So there's this ACL problem. Now, merely by coincidence, by the way, perforce in these older source code management tools did have ACLs in them. But just merely coincidentally, this tool called Git took over and Git has no ACLs. You know, the only ACL Git has, you're either in or out of the repo. So that actually means that this whole problem of who can see what doesn't happen in the coding space,

• 53:24Speaker 02

  which also makes it a lot easier for the AI models. So although coding is like intellectually very difficult, or at least for humans, it's like a thing that not everyone could do. It was considered a specialty. Anything intellectual the models can run out. What they need is they need like, uh, they need all the context to be sort of hermetically sealed and searchable. And they need to have like verifiable testing and they have all that stuff for coding. So I find it actually kind of like unsurprising that they got really good at coding. But it turns out that most people at Databricks, even though we have a lot of engineers, they don't do coding for their day. They're doing other things. The thing they're doing, it might require less extremely sophisticated individual reasoning, but it requires a lot of context. And so the thing that I'm most working on now is how do we, at the scale of a big company,

• 54:08Speaker 02

  I think there's this other layer, basically, like the layer I just talked about was this gateway layer that like will deal with security and tokens and all this stuff. There's also, I think, like a knowledge layer. It's like, someone call it an ontology. That's what we call our products, Databricks, but like, you call it whatever you want. There's a layer that's going to exist in any company of a certain size where it's not trivial to just like stick all the information from that company in a very small amount of context. And that layer is going to be the thing that helps the AI have the context they need to do the job. And designing that layer is hard. Like, it's not like a codebase where there's a very hermetically sealed thing. There's a sprawl of a million random things in a company. There's data in Slack, there's data in the structured data warehouse, like the data lake. That's what we do at Databricks.

• 54:51Speaker 02

  There's data in docs and there's data in people's activities. And there's implicit knowledge that seems to only diffuse through conversations. So like, how do we get this all in one place? And it respects access control and it's not creepy. Like, I don't feel like I'm in the company and I have like people listening to everything I'm saying all the time. That's a big, big problem in enterprises. And I think it's kind of the next frontier and we're solving it in very, there's very like small ways in which it's being looked at. Like, for instance, you can create skills and you can share those skills. You know, those skills are supposed to encode some knowledge of this enterprise and what we're doing. But when you have 10,000 people, that doesn't really scale. And by the way, the skills themselves become the same thing. They're messy and some of them are wrong and they contradict each other. And so they just look like every other thing in an enterprise.

• 55:36Speaker 02

  That's like the big other area I'm personally focused on is this, how do we build this knowledge layer? And how do we make it so that the AIs are never stuck on, they're just not smart enough. They're never stuck on that. They're stuck on like, they don't have the right information to get the job done. They don't know how that job is done, or they're missing access to a certain thing, or they're, they aren't able to search and understand. The user is asking them, you know, like we have in Databricks, the unit of pricing we have is called a DBU, Databricks unit. It's an opaque thing that means like a gold token of Databricks. That's just like how all our pricing works. Every different product we have normalizes into that unit. Okay. And so people will ask AIs like, hey, you know, how was our DBU growth in this sector or whatever?

• 56:22Speaker 02

  A lot of the models are, they don't even know what DBU means. That's like an internal thing that we have. And by the way, there's like sort of slightly nuanced, different versions of that we use internally. And finance uses one thing and product uses a different thing. And actually, finance and product don't even totally agree on the definitions of every derivative metric from this thing. So like, that problem is very difficult to solve and very messy. As a data company, we're very interested in that for our own products and for us internally. But this is the thing that I'm most focused on next, now that we're not going to go bankrupt, is like, how do we actually make these AIs better? And the thing that they're stuck on often, I'm seeing, is the context, basically.

• 56:58Speaker 01

  No, it's a fascinating problem. And you mentioned skills, right? Skills are probably the best example right now of this technique of like progressive disclosure of context, where it's like, okay, I'll give you a little snippet that helps you dive in. But I feel like you need nested ontologies of like, how do you build the map for the AI to even know what context it should be getting?

• 57:18Speaker 02

  Well, yeah, so skills have this hierarchical, like, you know, the problem is there's a context budget in these AIs.

• 57:23Speaker 01

  Exactly.

• 57:24Speaker 02

  And so like, skills have, they do have this like kind of nice property that, and the way you write them is increasingly like to do this progressive disclosure thing. But in my view, everything that a skill is accomplishing today should in principle just be learned by the AI. First of all, skills have the same problem with a lot of other things, which is they're human created and they're, that human better keep it updated, you know?

• 57:46Speaker 01

  Yeah, no, they go stale really quickly.

• 57:48Speaker 02

  They go stale, like every other document inside of it. So we need to solve the problem of knowing what's stale and knowing what's not stale. We already need to solve that problem because we need to understand what's in all these Google docs and all these other things. So in my view, by the way, I think skills are very powerful. They're like way better than nothing. But I don't think the end state is all of us curating lots of things and desperately, it's like a Wiki, half the things are out of date and they're doing more harm than good sometimes. So like, I think the end state is that there's a learning, a new learning problem, which is learn how this company works. It's like what you have to do when you join a company. And for the first year, you're like, you know, trying to figure out where the light switches, like,

• 58:22Speaker 01

  Totally.

• 58:23Speaker 02

  AIs in principle can also learn that, you know?

• 58:25Speaker 01

  Yeah. Well, and as you highlight, from what I've seen, AIs are really bad at understanding time. And so they're very bad at understanding, oh, this is actually still relevant because this was updated recently and this thing is actually probably less relevant because it was last touched two years ago.

• 58:40Speaker 02

  Yeah. So they need help. I mean, this gets into like traditional search type stuff. How do you know Google search had to solve this? Like, there's a lot of data on the internet, a lot of it's garbage. There's things that contradict each other and there's ways to do this. You look at authority, you know, you look at recency, you tell it very deliberately, like prefer things that are more recent, prefer things that are from users who seem to be authoritative users in this company. If it's a question about finance, maybe prefer something you saw from the finance team then like. So these are implicitly like often called signals in like search ranking or the search space. And the same thing needs to happen here. And I think skills or other types of explicit annotation are kind of like, you remember the Yahoo directory? It was like,

• 59:23Speaker 01

  Yeah, exactly. Yep.

• 59:24Speaker 02

  Yeah. And at some point that thing didn't scale. It was like, look, I'm like dealing with a long tail thing. That's not in anyone's skill, you know, like, or that thing is outdated or like, and Google just came in and was like, oh, we're just going to go dynamically learn all this stuff. And it was way better.

• 59:37Speaker 01

  That's a great summary. We're in the Yahoo directory era of LLM for enterprise.

• 59:43Speaker 02

  With respect to skills, the other stuff LLMs do are like complete magic. And I don't think ever should be compared to Yahoo.

• 59:50Speaker 01

  Well, I feel like we could keep diving on this for forever, but we are just about at time. Is there anything we have not spoken about yet that you want to make sure we cover before we wrap?

• 59:58Speaker 02

  I do want to do one minute to plug. You know, we have our AI gateway that we made into a product. We have thousands of customers using it, you know, way more than that, developers. And we actually also have this product called ontology product, which is very similarly, it's like totally open, you could use it with other tools. But like, we've thought a lot about this problem of how to encode that enterprise data. And so, yeah, give Databricks a shot if you're interested. Also, there's, there's tons of like DIY versions of these things too, if that's a better fit for people. So I'm not overly trying to plug Databricks. But, and then I would let people know that everyone is just figuring all this stuff out. And I put myself in that category. I think we've learned some things that I think will stay relevant for a little while. Even the things we've learned might become irrelevant.

• 1:00:40Speaker 02

  And the only like way that we've been able to learn is through experimentation, just trying different stuff, seeing what works, seeing what doesn't work. You know, finding colleagues in other companies, asking them for their advice. As I said, like several other folks helped us when we wrote that blog. And I think that like, we'll just figure this out one step at a time. But it's a very exciting time right now.