Building AI Systems You Can Trust

In this episode of AI + a16z, Distributional cofounder and CEO Scott Clark, and a16z partner Matt Bornstein, explore why building trust in AI systems matters more than just optimizing performance metrics. From understanding the hidden complexities of generative AI behavior to addressing the challenges of reliability and consistency, they discuss how to confidently deploy AI in production. Why is trust becoming a critical factor in enterprise AI adoption? How do traditional performance metrics fail to capture crucial behavioral nuances in generative AI systems? Scott and Matt dive into these questions, examining non-deterministic outcomes, shifting model behaviors, and the growing importance of robust testing frameworks. Among other topics, they cover: - The limitations of conventional AI evaluation methods and the need for behavioral testing. - How centralized AI platforms help enterprises manage complexity and ensure responsible AI use. - The rise of "shadow AI" and its implications for security and compliance. - Practical strategies for scaling AI confidently from prototypes to real-world applications. 00:01:00 - What is machine learning? 00:02:20 - The journey from tuning parameters to testing reliability 00:08:05 - Building production AI systems then and now 00:12:37 - Establishing trust in AI systems 00:17:18 - Centralization, platforms, and enterprise IT management 00:26:47 - Scaling AI usage in production 00:30:53 - How and why to test enterprise AI systems 00:38:10 - Cost management, tech debt, and prompt hygiene 00:41:44 - AI labs and enterprise users: Who influences who?

• 0:00Speaker 0

  After helping people optimize models for about a decade and a half, I came to a really interesting realization that it was solving the wrong problem. And foundationally it was that the thing that's holding back people getting value from these AI systems is not performance. It's not about squeezing out that last half a percent from some eval function or some performance metric. It's about being able to confidently trust these systems. And I can't tell you how many times over the decades we would help someone optimize a system and they would say, okay, but what did you break? What bad behaviors are you introducing? What lack of robustness do I now have because I've overfit this system? And we're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals.

• 0:49Speaker 0

  And that ends up masking all of these potentially undesired behaviors within the system itself.

• 1:03Speaker 1

  So can you just define for us what's machine learning, what's AI? Like as someone who's who's lived through, you know,

• 1:08Speaker 0

  the ups and downs of this market. I remember answering a similar question, I think on an A6NC podcast, maybe eight years ago. All

• 1:15Speaker 1

  right. Can maybe we can pop in the old right here. And uh

• 1:18Speaker 0

  I think the answer is somewhat the same. Like machine learning is the stuff that's now become easy, and then AI is all the fun new stuff. And then as soon as it stops becoming the cutting edge, then it just becomes, oh, that's just machine learning. It's

• 1:30Speaker 1

  not, it's not magic anymore.

• 1:31Speaker 0

  It's not magic anymore. And uh, I mean, you can go all the way back to the the old like Dartmouth conference and things like that of like, oh, spell check is AI because that's an untractable problem. But I don't think anybody would constitute that as AI today. Now, of course, with generative AI, I think one of the big differences is instead of these systems being focused on just kind of like classification or regression, trying to determine whether something exists in a set or what the next element should be. Now they're actually much more interactive. And like the types of applications, the types of uh ways that you can use these systems becoming so much larger because I mean, gen is in the gen the generative aspect of them is in the name, but um, that's fundamentally different, I think.

• 2:18Speaker 0

  And I think that's opened this whole new wave of value uh for enterprises. It starts about 10 or 15 years ago with uh my first startup, Sig opt. That company was based off of my PhD research, and it was all about how we help companies optimize AI systems. So, how do you get a little bit more performance out of a very sophisticated model? And so this is in the sort of traditional machine learning days, right? Exactly. This was all traditional first wave ML and AI. The definition of AI is obviously shifted uh over the last decade or so. But it was, yeah, tuning XGBoost models, tuning like ConvNets, tuning reinforcement learning algorithms, that sort of thing. But it was all about how do I tune all the different parameters? And now, of course, these parameters still exist with temperature and different ways that you can segment various aspects of these foundational models.

• 3:09Speaker 0

  But it was all about how do I get this peak performance. And we did this for about a decade and ended up selling the company to Intel in 2020, where I led the AI and HPC division for their supercomputing group. And after helping people optimize models for about a decade and a half, I came to a really interesting realization that it was solving the wrong problem. And foundationally, it was that the thing that's holding back people getting value from these AI systems is not performance. It's not about squeezing out that last half a percent from some eval function or some performance metric. It's about being able to confidently trust these systems. And I can't tell you how many times over the decades of working with Sig opt and all going all the way back to my PhD, we would help someone optimize a system and they would say, okay, but what did you break?

• 4:00Speaker 0

  Like what bad behaviors are you introducing? What lack of robustness do I now have because I've overfit this system? And we're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals. And that ends up masking all of these potentially undesired behaviors within the system itself. But it's really a much harder problem because instead of just having a binary output, now you have maybe more freeform text. Maybe you have an agentic system that can do all of these different things. And behavior matters more now than ever. But there aren't great systems out there to help you understand, define and test these systems. And that's really what distributional is aimed to do. It's trying to attack this problem of confidence through testing instead of just optimization through performance, like SIDOpt was.

• 4:53Speaker 1

  You were managing a pretty large team at Intel, right? And managing a pretty large customer base. Was that part of what kind of helped you see a different form of

• 5:01Speaker 0

  this problem? Definitely. That was a big jump from managing a team of about 25 when we were acquired by Intel to leading a team of about 200 by the time I left Intel. I got to see problems at a higher level. And I got to start to see some of the things that some of our customers now are frustrated with, where yes, it's great to build these models. Yes, it's great to have them be performant and things like that. But at the end of the day, if you're responsible to your customers, whether they be internal or external, you care about reliability. You care about consistency. And we kept running into problems there. And I mean, people are running this problem across uh the Fortune 500 and Global 2000 today of okay, how do I sleep at night effectively? And I couldn't find a good solution when I was at Intel. I couldn't figure out a way to outsource this.

• 5:47Speaker 0

  And so, like any good entrepreneur, if you have a problem and then you see it's a very pervasive problem, you can't see the solution, you go try to build it yourself. And it's been really exciting to do that again the second time. Again, taking all of those past learnings, all those past mistakes, I've made a ton of mistakes. It's just a naive PhD student trying to build a company. I won't force you to go through

• 6:10Speaker 1

  the mistakes that you made. Maybe, maybe that's for a follow-up podcast. Yeah,

• 6:12Speaker 0

  we'll need a we'll need a handful of podcasts uh to enumerate all

• 6:15Speaker 1

  of those. But now we get to do it again and now make all new mistakes 10 times faster. We work with a lot of founders who cycle through focus areas very quickly. We which can be great. That's a great way to sort of find product market fit. I think your persistence, you know, it is rare in a sense, where you actually spanning three companies, you've sort of worked on a similar core problem, and you're you're just really almost obsessed with finding the right way to solve it. What was it like to sort of have this light bulb moment where you're like, oh crap, I wasted, you know, X years doing the wrong form of this, or is it like, oh my God, this is so exciting, like I finally found the right angle on this problem. Or do you can you just talk a little personally what that was like for you?

• 6:59Speaker 0

  Yeah, I think it's more of the latter. And one of the joys of being persistent uh for better or worse is that it allows us to really empathize with the people who are building these systems. And we've been able to work with people doing machine learning, AI for the better half of a decade or more now. And we we get to see a lot of the same patterns, a lot of the same systems, a lot of the same mistakes being made. And we get to help them do a little bit better, do a little bit more. That's really been one of the ethos of both SIGOpt and now distributional. It's about building tools to allow these domain experts to do their expertise better, to do it more confidently, to really not have to worry about some of the things that they know they should be worried about and really focus on where they can apply their expertise to make something great.

• 7:51Speaker 0

  And it's been great to be able to work with honestly a lot of the same types of people. A lot of the people who are now in charge of building Gen AI platforms or productionizing these massive use cases are the same people who built those original machine learning systems. I think one of the main distinctions is how atomic some of these units maybe are. And so with a lot of traditional ML and AI, it was all about can I make a specific decision? Can I make a yes-no decision on whether or not I have a loan? Can I make a specific prediction? Do I know which direction the stock is going to go in the next minute or something like that? And with generative AI, it becomes much more collaborative and it's much more expansive in what it can actually do from literally just being able to have a conversation with it to some of these agentic systems that we're seeing come up, where now you have a model calling a model, calling an MCP server, calling a model, making a decision.

• 8:52Speaker 0

  And these pipelines have always existed, but the kind of end-to-end nature of them and how hands-off they can be, how dependent they can be on internal of these internal components and how some of these behaviors can propagate through this system, I think is fundamentally different from the more atomic, like unit-based, almost microservice aspect of uh traditional ML and AI systems.

• 9:16Speaker 1

  So

• 9:16Speaker 0

  I guess

• 9:17Speaker 1

  what does that mean if for like a product owner, what what do they kind of worry about when something goes to prod in kind of new AI land versus traditional machine learning land?

• 9:27Speaker 0

  Yeah. So I think some of the concerns are the same in the sense that they want these systems to be performant. They have to do better than whatever the status quo is today, whether that's a human or whether that's a traditional machine learning and AI system. But fundamentally, again, and this is the learning that I got after doing this for such a long time, focused squarely on optimization is it's more than just performance too. It's this behavior of these systems. And it's making sure that not only does it hit whatever KPI you want it to do, but it also doesn't do the bad thing. It doesn't have some sort of undesired. Behavior. And that undesired behavior could be I just don't want it to change dramatically without me knowing, or it could be that I explicitly don't want it to exhibit a specific bias or a specific type of response.

• 10:15Speaker 0

  And I think the main thing that's shifted though is just how large that output space can be. There's really three main things that make it difficult to quantify and understand the behavior of these AI systems. And one is that they're inherently non-deterministic. And so this is true of some traditional machine learning and AI systems as well. But basically the idea here is the same question can get you different answers. And this non-determinism isn't just that the exact same question can get you different answers, but it can also be a very chaotic system where like slightly different questions can get you very different answers. Another aspect of this is that they're inherently non-stationary. And so they're literally shifting underneath you. And this can be because your LLM provider decided to change their infrastructure.

• 11:02Speaker 0

  And then that changes the way that memory is accessed, which for some reason changes the types of responses, or it can be because maybe upstream of your application, somebody added more things to your vector database, or they change the retrieval prompt, or they change something else. These systems are constantly shifting underneath you when it comes to a product perspective. And this gets to this third component of the complexity of these systems, is just getting larger and larger. Again, they're no longer these atomic units where you're just getting a single yes-no answer out of them. It can be these systems where yeah, you're retrieving and then you're generating a response. And then that response is being fed into another system, being fed into another system where maybe an autonomous decision is being made.

• 11:49Speaker 0

  And so some of those issues around non-determinism and non-stationary end up just propagating through that system. If it's chaotic at the beginning and chaotic at every step along the way, these very small changes to the input that we're creating large changes to a single output can now create massive behavioral changes by the time it actually starts to affect an end user. And so it's really important. And what I think a lot of firms are running into right now is if you're only looking at that last step, if you're only looking at the system's performance as a whole, it can be very difficult to understand when, where, and why behaviors are shifting within this application upstream and be able to use that information to adapt, make changes to your application or understand exactly what's happening.

• 12:40Speaker 1

  I think we're pretty rapidly entering a world where trust in AI systems is even more important than their raw performance, right? Because these things are really good at doing a lot of different stuff. But how do how do users, how do customers actually accept that? Is that sort of part of what be of what you're talking about when you describe this problem?

• 12:57Speaker 0

  Yeah. And that trust can come in many different forms. It can be making sure it's reliable, making sure it's consistent, or even going so far as to say making sure that these kind of latent behaviors are aligned with my desires. And there's obviously a lot of great companies out there trying to solve this at the kind of AGI global level. But for individual enterprises, you want these applications to also be aligned with your business, with the values of your business with your individual goals, not just trying to squeeze out a little bit more click-through rate, a little bit more retrieval performance or whatever.

• 13:34Speaker 1

  So it's almost like the enterprise has to trust what the model or what the system is going to do in order for their customers to kind of trust them. Is that sort of a fair way to think about it?

• 13:43Speaker 0

  Definitely. And like anything in life, it's important to trust, but also to verify. And that's where testing comes in, because you do need to be able to trust these systems, but you want a mechanism to be able to consistently reliably and adaptively verify that they're doing this, both as you're making changes as the world changes underneath you, and as these models change as well.

• 14:05Speaker 1

  And this is what you're doing at uh distributional, right? Definitely.

• 14:08Speaker 0

  So distributional is an enterprise platform to allow teams to test these applications in production to make sure that they are behaving as expected. Behavior for these applications ends up being not just what it produces, but how it produces it. So it's all of these characteristics, potentially of the text itself. So not just was this a good answer, but maybe what was the toxicity of the answer? What was the reading level of the answer? What was the tone? How long was the answer? All of these just properties that can be characteristics of just the language itself. But if it's part of maybe a rag system too, it's like what was retrieved, is how often is that what's being retrieved? What were the timestamps related to these individual documents? Was it starting to ignore things that it used to cite all the time, or vice versa?

• 14:58Speaker 0

  And as we start to get into these agentic systems, and a lot of teams speak about this. I know Google's deep research team spoke about this recently. It's like, well, in the reasoning step, how long did that take? How many reasoning steps did it take? All of these are characteristics for how the model is actually behaving on the way to that end answer, on the way to that performance metric. And I definitely don't want to say the performance doesn't matter because it definitely does. And that's an aspect of behavior. You want good behavior, you want good performance, you want consistent performance, but it is fundamentally this very high level bit and it can mask all of these underlying latent behaviors that could have an effect on the system. And especially because these are inherently non-stationary and chaotic systems, you want to be able to catch those latent behaviors as quickly as possible.

• 15:51Speaker 0

  And some of them might actually turn out to be things that you want to be performance metrics, much in the same way that with traditional machine learning, fraud detection, you want to be as accurate as possible. But by detecting that you have bias, maybe you want to factor that into your performance metric over time. And this is about adapting what that desired behavior over time as you learn more and more too.

• 16:17Speaker 1

  Sounds like the first time I ever did a job interview. It's not just what you say, but how you say it. Exactly. And

• 16:22Speaker 0

  then what you do as well, too, because at the end of the day, if you have a lawyer or something, having a high score on the LSAT's great, but that doesn't tell you anything about how they're going to behave in the courtroom.

• 16:31Speaker 1

  So AI, you know, bedside manner or courtroom gravitas.

• 16:34Speaker 0

  And there's so many classic examples too from traditional ML and AI of, and you'll remember some of these, I'm sure, too, of you train a reinforcement learning algorithm to like pitch a baseball and it it's decides that just running the baseball up to the catcher is the most effective way to do it. And you're like, well, okay, I did just tell you to get the ball in the mitt. You technically solved it, but that's not great. But even in these gen AI systems, like there's this classic example that came out recently of chess, or if you just tell it to win a chess game and you give it access to board state, it's going to rewrite the board state to win. And that's an example of a behavior where, yes, if the performance metric was games one, it's got a great performance metric, but the behavior was it cheated.

• 17:21Speaker 0

  One thing that we've seen that's really interesting over the last year, year and a half is people have started to shift from kind of science project prototype land where they have a bunch of individual teams trying to roll their own stack and trying to like build everything that they need to get something out individually into starting to build more centralized platforms. And this is again very similar to what we saw in the early ML and AI days, where maybe every individual data scientist was like, I'm going to spin up Scikit Learn and I'm going to have all my data locally and I'm going to have some model that works incredibly well for rapid prototyping and exploration. But when it comes to exploiting these models and really making sure that you can do it at scale, making sure that the organization can protect itself, can really leverage all of its resources, you want to have more centralized tooling.

• 18:14Speaker 0

  And so much in the same way that we saw the in the rise of ML and AI platforms over the last decade, we're now starting to see the rise of these gen AI platforms. And this has organizational benefits from again, making sure you can scale, making sure you have like proper cost allocations, but it can also really tamp down on a lot of what we're hearing from CIOs and CTOs of kind of shadow AI. People calling models they shouldn't, feeding them things that they shouldn't, creating vulnerabilities that they shouldn't. And so this is the perfect kind of harness to start to do testing, though, as well, too. Because once you have the centralized Gen AI platform, once you have a gateway or a router that's logging all of your API requests, all of your traces, et cetera, testing can live on top of those logs on top of that data store to give you this more holistic view across all of your applications.

• 19:07Speaker 0

  How are they behaving, which ones are behaving differently and provide that behavioral analysis and testing to the end developer for free as part of that platform?

• 19:17Speaker 1

  And so is this shadow IT problem worse with language models and generative AI compared to compared to the past. I would say so because everybody's doing it. Yeah, to know what Scikit learn is and exactly. And

• 19:29Speaker 0

  it was a somewhat localized problem because like you're doing data science on your laptop versus now I'm just shipping off a secret IP to some SaaS company or something like that. And

• 19:40Speaker 1

  it's really easy to

• 19:41Speaker 0

  do it. You just

• 19:42Speaker 1

  need an API key basically.

• 19:43Speaker 0

  Exactly. I mean, it's great that all these developer tools exist and they've made it really easy. The downside is they've made it really easy for people to do things they shouldn't do either. And I think a lot of organizations are starting to realize that. So not only do you have kind of the benefit of central. Centralization and scaling and support and things like that, you're also mitigating the harm that sometimes is accidentally being done, where it's like, oops, I guess our code base is now public or whatever it may be. So if

• 20:12Speaker 1

  I'm a technology executive at a big company right now, experiencing this phenomenon of little AI projects popping up everywhere, some of them doing really well and actually becoming important parts of our business. What should I think about doing just practically speaking to kind of get my arms around this? And what should I think about building to reach kind of like this target, you know, state of like platform nirvana?

• 20:36Speaker 0

  So we see a lot of different variations of this, to be honest, across all the different companies that we're talking with or working with. And I'd say there's two things that they're trying to solve here. One is trying to make a platform be useful enough to basically draw in these people who are saying, no, don't worry, I've got it covered, like I've built my own stack. And so some of that is by providing value add services. So it's like we're going to take care of scaling for you. We're going to take care of cost optimization for you. We're going to take these things off your plate. And one of the things that they do is like, well, we'll build kind of a centralized router so you can access all of these great LLM models. And we're going to basically create a store that you can uh switch between versions and models and things like that. We can centralize logging as part of this.

• 21:23Speaker 0

  So you don't have to deal with the fact that this is producing maybe a large number of logs. And on top of that, too, you can provide testing. So now we can do this layer of making sure that you can detect and understand these underlying behaviors. And this is fundamentally different than the way a lot of people are rolling this today, because if you're using one of these kind of all in one out of the box platforms focused on developers, they allow you to get going very quickly, but maybe they're not logging everything, or maybe they have very rudimentary monitoring that is just looking at a handful of performance evals, or they have the ability to kind of look at individual inputs and outputs and do hand annotations, but it's very difficult to do that at scale. And so for these technology executives, they need to be able to come up with this way to create this kind of uniform or lowest common denominator interface so that everybody can leverage it.

• 22:14Speaker 0

  But then they also have to provide that value add. And we're seeing this from a lot of executives to entice people to move on to that platform.

• 22:22Speaker 1

  I mean, you made an interesting point about giving developers a reason to want to get onto the platform. Like if I now switch to my developer hat, it's like consume logs for me. That sounds great. I don't know where to stick logs. Like test for me, that sounds great. I don't like writing tests. Give me a store that somehow standardizes like the interface a bunch across a bunch of different LLMs. Like that sounds less good to me. It sounds like you're just like adding a layer between like the thing I actually want to use, just practically speaking, like what pieces often come first. And like, how do you actually do this if if like if this is your job?

• 22:52Speaker 0

  One of the first pieces that we see is this kind of gateway or router. And some of it is less a value add and more of uh, this is the only way we're gonna let you access these things. It comes down to some of these models need to go through a GRC process. It's like we're gonna let you use Open AI, but we're not gonna let you use hosted deep seek or something like that. And by centralizing that, they can then start to rein in some of the chaos. Here are the 30 different models that we do support and the 20 different versions of each that we do. Oh, wow.

• 23:23Speaker 1

  So you're seeing enterprises will actually support that. Because I was sort of picturing in my mind like you here are the two or three models you're allowed to use, but it sounds like you've seen one company may have 30 models available.

• 23:32Speaker 0

  People want to use the right tool for the job and different models, different versions have different trade-offs, different costs, uh, different context windows, different rate limits, all these sorts of things. And we're also starting to see more and more people wanting to fine-tune or create SLMs or use more kind of static weight models as well, too. And so it ends up being this difficult infrastructure problem of hosting against a handful of nonstationary APIs, against a handful of internal models and things like that. Creating this uniform interface ends up being extremely valuable for that developer to be able to pick and choose, to be able to A B test, et cetera. So one big complication is that sometimes the incentives are misaligned. So OpenAI obviously wants to create the best general purpose foundational models.

• 24:18Speaker 0

  But an individual business may want a model that solves a very specific problem, a very specific way very well. And a developer may just want to be able to get something stood up and integrated into their application as quickly as possible. And so this platform uh engineer, this technology executive is kind of stuck between these two things where they need to be able to provide access to all of these cool tools and all of these great abilities while also making it as easy as possible for the developer. And so we've seen kind of two approaches here. One is where they will leverage kind of a Gen AI platform from maybe a cloud provider, and there's many great examples of this from the various clouds, or build up from best in class tools. They're gonna pick the vector database that fits their needs the best.

• 25:04Speaker 0

  They're gonna pick the log store that fits them the best and the testing solution that fits them the best, et cetera. But fundamentally, they're trying to solve the same problem of fitting these two puzzle pieces together. Here's all the great research is that's happening across the world and changing every day. And here's what my developers need to move their things forward. How do I create this kind of universal kind of puzzle piece adapter in between and then provide enough value on top of that?

• 25:32Speaker 1

  And so that's a real technical problem. If I have 30, as you said, 30 models with variants and fine-tunes and all, like that's actually a real problem that needs to be solved, not just I'm gonna, I'm gonna lock this down.

• 25:41Speaker 0

  Yeah, exactly. And something that if you were just trying to get stand something up, you're just gonna pick one route, one path, et cetera. After you've built that router or gateway, though, the very next obvious thing to do is say, okay, well, this is actually there's a lot of data going through this. I should probably log that data somewhere instead of just throwing it away. And that's where they start to leverage sometimes more traditional data stores that they have in place today, where it's like, okay, we already have a way to log API calls or traces or some of this richer information. But once you have those logs, then you can start to do analytics testing, monitoring, and things like that on top of it. And so it starts to accrue this value and end up looking more like a traditional platform as well.

• 26:27Speaker 0

  And meanwhile, the developer starts to get some of these capabilities kind of off the shelf. And so they don't have to think about logging. They don't think about having to test, they don't have to think about any of this. They can focus on fine-tuning their prompts. They can focus on building these agentix systems into their current user workflows.

• 26:50Speaker 1

  I think a lot of developers, to your point, when they're trying to solve a narrow problem, kind of think they have it covered, right? They've got a handful of test cases. They just kind of play with it for a little while. You know, like what people used to call vibe check. I don't, I don't know if that's still that's still a term in use. Um what really should they be thinking about? And from an enterprise

• 27:09Speaker 0

  standpoint, what should uh people be thinking about? We believe that they should be thinking about the holistic behavior of these applications at scale. And so that's not just looking at a handful of performance checks on a small data set. That's not just throwing in your hundred favorite inputs and making sure that you always get your hundred favorite outputs. We talk to a lot of firms that are terrified to cross this AI confidence gap from I've developed something that works good in in theory. How do I actually scale it up in practice? And a lot of times what we'll talk to to uh individuals who say every single time I bring on a new user, every single time I add more data to this, it changes a little bit. And right now I'm doing that incrementally, but when I turn on the fire hose, I have no idea what's gonna happen.

• 28:01Speaker 0

  And I'm terrified about what that is. And this can create this gap where things languish in this prototype phase. They languish in this, I've got this great proof of concept, but I still am terrified to turn it on to a million users, 10 million users, whatever it may be, or turn it on to real enterprise value. And so I think the shift in mindset needs to be not just does it work the way that I want to look at it, but does it work more holistically?

• 28:29Speaker 1

  Do you have any good stories about things that have gone wrong when people don't test?

• 28:34Speaker 0

  Yeah. I mean, there's a handful of stories where people think that they're doing the right thing, think that they're, of course, this is a no regrets value add to the system. And it ends up having these weird trickle-down effects. And so RAG is obviously become very prevalent uh in a in a wide variety of industries and people use it for a lot of different things. We've spoken with different firms that they were like, okay, well, I'm just gonna continue to add more and more data to the corpus because more data is better. Like, of course. But this ends up messing up the retrieval mechanism. And so we're before it all had very recent data and it was giving very good responses because people were asking about things that had a lot of recency. Now they put their whole history into it. Now it's grabbing old stuff and pretending like 1902.

• 29:19Speaker 0

  Yeah, exactly. I wanted last quarter's earnings, and now you're giving me six quarters ago's earnings. Um, or I wanted this specific entity. And that was relatively unique to begin with. But now that you've flooded it with all these other things, I'm picking up all these things that are kind of around the edges of it as well, too. Have you seen anything released

• 29:38Speaker 1

  that upset all

• 29:39Speaker 0

  their customers or trading strategy that just

• 29:42Speaker 1

  like, you know, like

• 29:43Speaker 0

  was a bottomless pit of money? So thankfully we haven't seen anything like that quite yet. But there's definitely hallucination remains a problem. And it's one of these things where the system can convince itself that there's evidence that there isn't. Or will convince itself that it you want an answer that's different than what you actually want. And sometimes that's because it's been given information that it tries to interpolate between. Sometimes it's just trying to fill in the gaps. Like fundamentally, that's what it's uh these systems are trying to do. And that can lead to really bad behavior for users. And it can lead to another example is people using the system in the way that they've always used it, but for some reason it's starting to trigger all these guardrails.

• 30:30Speaker 0

  And it's because intermediate parts of the system have transformed it or morphed it in different ways that all of a sudden it's flipping a switch somewhere. And now they're getting this terrible user experience because they're being told they violated some policy when in reality maybe they only changed a single word in their prompt.

• 30:56Speaker 1

  Is your belief that AI systems should basically be tested atomically? Like in the same way that we had unit tests for for traditional software, you can sort of isolate each piece and make sure it's performing up to spec. Is that sort of the same idea for AI systems?

• 31:12Speaker 0

  I think you need to be able to quantify the behavior atomically, but then be able to test across that behavior more holistically, like a regression test. And so you do want to be able to quantify the characteristics of how the retrieval step is happening. You don't want to just look at the very end answer. But you also want to be able to see how changes in inputs have propagated through the system. I see. And so it's a mixture.

• 31:42Speaker 1

  And I know you've done some pretty intense math. I'm resisting the urge to ask too many math questions because I know the two of us will go go a little too far down the rabbit hole. But I know you have done some pretty sophisticated work as a team to think about the right way to approach these sorts of tests. Do you mind giving just a brief overview of why it's a hard problem and how you've addressed it?

• 32:02Speaker 0

  Testing is a is a difficult problem because I mean, just being able to quantify behavior and trying to understand what it is that's like intrinsically happening within these systems is a very difficult problem. And this is where we differ a little bit from maybe traditional approaches to LLM eval, because instead of trying to come up with a small number of strong estimators for performance, where we want to be able to conclusively say A is better than B. Instead, what we want is a large number of potentially weak estimators to be able to determine whether or not A is different than B. And you're asking a fundamentally different mathematical question at that level, because having these weak estimators that are maybe higher entropy and things like that can give you an insight into these subtle shifts in the way that the system is behaving or acting or processing information that will then have some end result in performance that you actually care about.

• 33:00Speaker 0

  But being able to correlate that, being able to go back and root cause and be able to say, my performance dropped because this shifted, because this component changed. Here's evidence of what distributions shifted. Here's evidence of the results that are no longer the same, can be an extremely powerful tool to not only give teams understanding, but allow them to actually react to this. Because one of the things we've seen is when your performance drops, that's helpful because you know something must be broken, but it kicks off this net new research process to be able to be like, okay, now I need to build again from scratch versus performance dropped because this thing is not what it used to be that gives you this foothold to be able to go fix that.

• 33:46Speaker 0

  Or maybe not all change is bad. Maybe that's actually something that's that you want to uh factor into your performance as well. I see. So you're almost saying

• 33:55Speaker 1

  it's like rather than a normal software stack where you're just kind of testing end to end what's what's happening, you're sort of saying you've got like some little lab, you know, subjects in this like glass tank, and you need to attach like all sorts of little sensors and probes to see exactly how it's behaving in order in order to kind of know what's going on.

• 34:12Speaker 0

  Exactly. Exactly. And it's it's really upping all of those sensors and probes, basically. So don't just see whether or not lab subject A versus lab subject B was able to complete the maze, but like what was their heart rate? Like what was all of these types of things that then would enable you to say.

• 34:28Speaker 1

  And so, and so are you effectively doing statistical testing to sort of understand the change, you know, in each of these sensors from one one iteration to the next, or or just again at a very high level, how does it work?

• 34:37Speaker 0

  So this gets into the name of the company. We think of all of these things as distributional. And so fundamentally it's not about having a single input be bad that you might want to trigger something, but it's about holistically how is this behavior changing in a population setting? How is one distribution of behavior today versus a distribution from yesterday? And that distribution isn't just what's the distribution of performance, but this higher dimensional distribution, this distributional fingerprint of behavior, how has that shifted? And that allows you to get these insights, which fundamentally allow you to root cause and understand your models in a way that you couldn't otherwise.

• 35:18Speaker 1

  Which is very interesting because there's a lot of talk, especially with large language models about what's what's in distribution versus what's out of distribution. And typically people are talking about what was sort of well represented in the training data versus what wasn't, right? Which is important because these models tend to do well at things that were in their training data or similar to and not so well at out of distribution things. I've seen very little actual kind of quantification of this, right? There's a few good papers about it. But what we're sort of saying is for any particular system, meaning using a certain set of models in a certain way with certain set of prompts, you can actually characterize literally what the distributions are and then and then sort of check how those change over time.

• 35:51Speaker 0

  Yeah, exactly. The distributions of the outputs, but also the entire process to getting to those outputs. And there's a lot of rich information there. And a lot of it's being masked today by only looking at individual things. It's very

• 36:04Speaker 1

  cool. I mean, it's it's really an industry-wide problem. And so what is the and what's the benefit to to an enterprise in the end of deploying distribution or something like you know, a testing solution in general?

• 36:14Speaker 0

  Yeah. So it's more confidence, which allows them to tackle harder problems, to be completely honest. We see some firms attacking the low-hanging fruit internal chatbots to like quite ask questions about HR because they're afraid to take that leap to develop the difficult problem because it's so unwieldy and there is so much risk associated with it. A lot of the most valuable use cases also have the most inherent risk. And testing and confidence is a way to understand and mitigate those operational risks, whether they be financial, reputational, or regulatory. So being able to define like what does reliable mean it starts with change. It starts with at least is today different than yesterday. Like that's a question you can ask in an unsupervised way, because you don't need to have any preference between the two.

• 37:05Speaker 0

  But then from that, you can start to say, okay, I liked this change. I didn't like this change. You can start to become more and more specific about what types of change are acceptable or not. But just by being able to see differences, then you can start to apply supervision of start to apply uh a preference for one versus the other. And I think fundamentally people are starting to kind of accrue more and more sophistication. Maybe uh originally they were just replacing kind of classic NLP models with LLMs and things like that. And they're starting to push the boundaries of what used to be possible, not just making it better. Yes, it's easier than doing LDA to just like hit an LLM or something like that. But now, especially with agents, they're starting to really see what these completely new frontier.

• 37:55Speaker 0

  But with that comes a lot of uncertainty. With that, you need kind of a flashlight to be able to say this is this is how I understand this. This is how I can really get confidence. One thing that we've seen with enterprises, especially over the last six to 12 months is as they've started to roll more and more applications into production, they start to be able to make different trade-offs and they start to accrue tech debt, to be completely honest. And that doesn't exist when you're first building, but of course, like any technology, it accrues over time. And so they're starting to try to make trade-offs between like, hey, can I use a cheaper model? Can I clean up the system prompt that I've just appended to over and over again over the last year?

• 38:42Speaker 0

  But in order to make that decision, you need to understand what the trade-offs are. What if I switch from this expensive model to this cheaper model? What if I refactor my system prompt or whatever it may be? And so again, understanding the performance impact is an aspect of that, but understanding how does this change the way that I'm tokenizing things? How does this change the way that this application is behaving can also lead to an ability to make these decisions in a more clear-eyed way? And like traditional software, you have a build. You have a test suite when you're refactoring to say whether or not you actually made it cleaner or you broke the build. And by having good behavioral test coverage, you can now ask that exact same questions with these Gen AI apps.

• 39:30Speaker 0

  Did I make it better or did I actually break the build? So

• 39:34Speaker 1

  you're saying my system prompts shouldn't be full of capital letters and exclamation points and pleading, please please do what I ask.

• 39:41Speaker 0

  Potentially.

• 39:42Speaker 1

  Whatever

• 39:43Speaker 0

  works for you, again, the right tool for the job. Maybe you can go back and make it one long capitalization, please, instead of

• 39:51Speaker 1

  a hero on Twitter who's whose name I won't say who who extract system prompts from all the all these things. And my biggest takeaway from this is that system prompt often. Reflects the organization it came from. Like it's some new form of Conway's law. You know, Conway's Law is this thing that you kind of ship your org structure as a software company. Like I think as an AI like, you know, system company or like a prompt writer, you're kind of shipping your org to. It's like the Google one is like very, you know, comprehensive and technical and dry. And then there's the startup one that's like kind of all over the place. So it's it's an interesting thing you say. Like the system and even the prompt sent to a language model kind of reflects the the organization it's coming from and kind of like the behavior they're looking for in a way. Exactly.

• 40:32Speaker 0

  And as you modify that, or as you, I mean, it's nested system prompts all the way down, right? As you modify yours that's calling theirs, like you're gonna get different behaviors. And it's it's a combinatorial mess, even just selecting a model, let alone you exploring system prompt space yourself. And again, having at least insight into the behavioral implications of your changes is incredibly important because I mean, fundamentally, once something's actually making money, once something's actually useful, you need to take a little bit more of a conservative stance to make sure you don't break things as opposed to just trying to build something as fast as possible.

• 41:09Speaker 1

  Just love the idea of every new person who, you know, it's like the engineer, then like the uh compliance person and the market, everybody's just tacking on to the prompt. This this poor confused language model probably has no idea what to do.

• 41:19Speaker 0

  And some of them are conflicting and like whatever it may be, and uh yeah, policies shift. We've we've uh talked to a lot of firms that are afraid to change that prompt because they're like, well, the GRC team wanted this line in there, and this team wanted this line in there. And what happens if we combine them? That's

• 41:36Speaker 1

  it. But with a system like yours, basically they're able to check how things change once they once they do make some of these fixes.

• 41:42Speaker 0

  That's very cool.

• 41:47Speaker 1

  Most of these AI labs are not enterprise folks, which is, you know, which is which is fine, right? You know, it's a lot of very, very, very smart researchers um who know their field very, very well. But yeah, I yeah, anyway, I'm curious. I'm curious if you see this interface. You know, because I guess some of the very big customers for you know the big labs do have more influence than you know than ordinary, you know, ordinary devs.

• 42:09Speaker 0

  I mean, I think it's gonna be uh co-evolution, where obviously some of these labs are very focused on the cutting edge of research and like pursuing AGI and things like that, but they also need to make money. And so they're gonna adapt to what their users need. And then the industry is going to adapt to the tools that are available and it's gonna be back and forth. It's gonna be like uh the finches on the Galapagos Island. And overall, we're gonna get some, I think, specialization. Certain models are going to come out, they're gonna solve specific enterprise needs is incredibly well. But in general, too, the the enterprise is going to continue to adapt to, hey, I've got this new tool. Hey, I have access to this new thing. How can I make it fit? But unfortunately, again, I feel like the platform owner within these enterprises ends up being this connector.

• 42:59Speaker 0

  And it they need to be able to be this interface to this technology while also providing it in a way that's accessible, consumable, and understandable and ultimately fits the needs of the enterprise, whether that's through testing and auditing and things like that or just scalability.

• 43:16Speaker 1

  That makes a lot of sense. And you and you're sort of saying that these enterprise platforms actually have a pretty important role to play in like the whole, the whole industry. If I'm sort of understanding, right, because the labs put out what they put out, developers love experimenting with things, some of them work, but if there's nobody connecting the puzzle pieces, as you say, like that actually could be a real problem.

• 43:33Speaker 0

  Definitely. And somebody needs to make sure that these models continue to work. These applications continue to work over time as well, too. And I've seen this with traditional ML and AI, and this happens in traditional software too, where the developer moves on and then it needs to be maintained. It needs to stay up.

• 43:52Speaker 1

  Yeah, because we don't we don't have like AI ops teams yet, right? There's no dedicated group of people that just like try to make sure the system is running. And in the way that we do with sort of traditional, you know, DevOps and things like that. Who gets paged in the middle of the night when like, you know, your your AI bot just like sold the the office building by mistake.

• 44:09Speaker 0

  Exactly. And who has to figure out what whose fault it was at the end of the day, too. That's right. Um, I think as we see the rise of these gen AI platforms, we're gonna see the rise of more AI ops, the people who have to make sure the system's working and understand when it isn't and then fix it.

• 44:26Speaker 1

  We talked about global versus local solutions to this reliability problem. What's your view? What should be solved kind of by the industry as a whole versus what what needs to, you know, this kind of like local context?

• 44:38Speaker 0

  Definitely. So I think there's aspects of this that are universal, just being able to define some of these behaviors and detect large scale changes in them, being able to set up a system that's able to give you that level of insight again at that extremely high level. But then every individual team has different behaviors that they they want or things that they don't want. And so very quickly, you need to be able to take from a system like ours an ability to detect change at a global level and then be able to go through a workflow to adapt that and build a better behavioral test coverage that becomes more and more specific and bespoke to your individual application. But fundamentally, that same global level can help many different teams within an organization or even across different organizations, get that foothold.

• 45:30Speaker 0

  And that's what we're trying to develop with our platform. But then beyond the platform itself, that workflow is about fine-tuning it and specifying it to the behaviors that you care about individually.