The Gap Between AI Spending and AI Value

It is widely reported that a gap has emerged between enterprise spending on AI and the durable value captured from that spend. Individual employees have enthusiastically adopted coding assistants and chatbots, yet those gains do not seem to be transforming businesses at an organizational level. One of the most important questions in the tech industry today is understanding why AI is not yet delivering returns that match the investment, and what separates the small number of enterprises succeeding from the many that are not. Scale AI https://scale.com/ is known for supplying the human-labeled data behind many frontier models. It now also builds AI applications and agents for large enterprises. That combination of working alongside frontier labs and inside enterprise deployments gives the company a rare view of why enterprise AI may be stalling. Emily Xue https://www.linkedin.com/in/yuan-emily-xue-3483012/ is the Head of Enterprise AI at Scale AI, and previously spent over a decade at Google. In this episode, Emily joins Kevin Ball to discuss the three layers where enterprise AI breaks down, why frontier model benchmarks miss what enterprises actually need, the data foundation problem, how the most successful companies combine internal domain expertise with outside AI specialists, and more. Sponsorship inquiries: [email protected] mailto:[email protected] The post The Gap Between AI Spending and AI Value https://softwareengineeringdaily.com/podcasts/the-gap-between-ai-spending-and-ai-value/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  It is widely reported that a gap has emerged between enterprise spending on AI and the durable value captured from that spend. Individual employees have enthusiastically adopted coding assistants and chatbots, yet those gains do not seem to be transforming businesses at an organizational level. One of the most important questions in the tech industry today is understanding why AI is not yet delivering returns that match the investment? And what separates the small number of enterprises succeeding from the many that are not? Scale AI is known for supplying the human labeled data behind many frontier models. It now also builds AI applications and agents for large enterprises.

• 0:44Speaker 00

  That combination of working alongside frontier labs and inside enterprise deployments gives the company a rare view of why enterprise AI may be stalling. Emily Hsu is the head of enterprise AI at Scale AI and previously spent over a decade at Google. In this episode, Emily joins Kevin Ball to discuss the three layers where enterprise AI breaks down, why frontier model benchmarks miss what enterprises actually need, the data foundation problem, how the most successful companies combine internal domain expertise with outside AI specialists, and more. Kevin Ball, or K-Ball, is the vice president of engineering at Mentos

• 1:27Speaker 00

  and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action Discussion Group through Laten Space. Check out the show notes to follow K-Ball on Twitter or LinkedIn, or visit his website, kball.llc.

• 1:57Speaker 01

  Emily, welcome to the show.

• 1:59Speaker 02

  Thank you so much for inviting me, Kevin. It's my pleasure to be here.

• 2:03Speaker 01

  Yeah, I'm really excited to dig in with you on this. Let's start with a bit of your background. So can you kind of give us a little bit of your career to date and how you ended up at Scale?

• 2:12Speaker 02

  Yeah, I absolutely love to share that. A lot of people are saying like my career is kind of a reverse of many people. So currently, I'm the head of enterprise AI and Scale AI. It's kind of a company, a startup company, but it's not like as small as a startup anymore that people kind of know from the perspective supplying the data to many of the frontier labs and being an active participation to the Gen AI large launch model to where we are. So it's very interesting work from both understanding the customer requirements on the enterprise side as well as driving the innovations to research for pushing enterprise AI to land with revenues. Then before that, I was actually worked for a really large company, Google,

• 2:54Speaker 02

  one of the hyperscalers and also the frontier labs, two in one. I worked there for 11 years, being very hands-on engineers. I'm also early member of Google Brain team. So I was a researcher, hands-on model training, publishing papers. And I was engineering lead for several of the cloud AI product, Vertex product, tuning, Gen AI tuning, evaluation, agent, engine with my team members. We really start with the concept, the idea and the seed of market needs for it and then put our hands down there, build up the product and bring this to the customer. So it's a really fruitful experience working with the team, put up a team to bring this to the market to see what the enterprise adoption for this platform products are.

• 3:38Speaker 02

  So that's my experience at Google for about 11 years. But actually, before I joined Google, I was a professor. I got my tenure from Vanderbilt University. So my research is kind of combination of like applying a lot of the theoretical analysis from optimization and statistical learning in understanding certain networking system, a demo network and large scale distribute system, and with the application to several domains, separate physical system, healthcare system. After I got tenure from Vanderbilt, I spent my year, sabbatical years at Google and decided to stay with the industry. That's kind of like me, yeah.

• 4:17Speaker 01

  That's awesome. Well, I'd love to dig into a few different pieces of that, but let's start also with a quick overview of Scale, because you mentioned Scale. When I was doing the research for this call, I initially was like, oh, I've heard of Scale. They do data labeling, right? But y'all do a lot more than data labeling now, don't you? Like, what is the span of what Scale is tackling?

• 4:35Speaker 02

  Absolutely. This is a great question. Secondly, so people really know about Scale, and this is also like how Scale actually, you know, incorporate as a company is really to bring data to machine learning sort of labs and the machine learning companies as a data provider, starting from providing data for image, like vision perception models. And later on, the market is actually, you know, expanding from the vision models and then getting to language models. So this is where we're actually getting to partnership with OpenAI in very early days, providing human annotations, human feedback data to make GPT where they are now. So that's a core business. But meanwhile, at Scale, and also I think the broader market also realized

• 5:18Speaker 02

  that beyond the foundation model capabilities, the landing of the value of all these models really depends on building reliable applications and agents that powers the applications. So that's actually another big chunk of business for Scale. And within that, sort of the application business, we have two business units, depending on who are the customers we are speaking to. So I'm currently leading the enterprise AI side where I'm in the enterprise business unit, where our customers are private sectors, 500 sort of enterprises. And we'll also have another business unit where it's public sector facing US and the global governments.

• 6:00Speaker 02

  But the goal is kind of like really building applications and solutions on top of leveraging the intellectual capabilities for models to serve the business needs.

• 6:11Speaker 01

  Yeah, well, and that makes a ton of sense. I think we're all trying to figure out where are the value layers in this new AI world, and it definitely looks like it keeps moving up the stack. Well, I'd love to really dive into the enterprise side. I think there's a lot of interesting things there. And, you know, I work at a relatively small startup. The adoption challenges in a startup world are very different than they are in the enterprise world. I think you've actually written a bunch about the different failure modes of enterprise AI and things sort of that. So let's maybe start with that. Like, what is different about adopting AI in a large enterprise? And what are the classic failure modes?

• 6:48Speaker 02

  Yeah, absolutely. So this is very insightful question. And there's actually many kind of directions people are thinking about it, right? So maybe what we can do is I would think about like, to start with, is it's a problem of two sides. On one side is on the machine learning AI capability side. Like, you know, people usually say, think about from sort of AI perspective. On the other side is actually from the platform system perspective. So in order to have system solution, application that truly serves the enterprise customer needs, both side has to meet with each other and be successful. And then on top of the technology layer, that is also adoption layer,

• 7:33Speaker 02

  which is the management side. It also touches the people side of the enterprise usage. So it involves education, leadership, brought in and all the stuff. So I just want to sort of maybe anchor this discussion on this like people versus technology. And within technology, there's the AI part of it, there's also the system part of it. So maybe we can start with on the AI side of it a little bit. So if you think about the AI side, the market, the community have a lot of exposure to what foundation model frontier labs are doing, right? I come from Google, actually, I'm part of the Gemini team. I've been going through like Gemini 1.0 all the way to 2.5.

• 8:14Speaker 02

  So if I look from the insider lens, when we developed Gemini, right? So if you look at it, the development team actually have very limited exposure to what enterprise needs are. And this is exactly what I feel, you know, even I'm from cloud at enterprise part of business of Google. Fundamentally, every model iteration, which we call data flywheel, you know, understanding where the model gaps are and how to contribute data, train it. The whole engineering practice is pretty much owned by the engineering team, who actually do not have the business exposure. And that is reflected as, right? So people are very benchmark driven.

• 8:56Speaker 02

  They have very focused goal, say, hey, now we have, we actually have different capability pillars, math, reasoning, code, conversational, multi-turn safety. And then if you look at that, the enterprise part is becoming a very narrow pillar in all the frontier labs, progressively into 2025. So which means the model knows a lot of the human intellectual capability and knowledge, many of them is reflected by our flagship benchmark humanity last exam. But in terms of concretely, what is the professional requirement? What is the operational workflow?

• 9:37Speaker 02

  What are the policies that is actually super critical in the enterprise domain? There's no evaluation criteria. There's no intuition knowledge within the people who actually know the model. So then from AI perspective, the fundamental capability has a gap.

• 9:56Speaker 02

  From what it be.

• 9:58Speaker 01

  Well, it sounds like not only a capability gap, but even a like goal or target gap. Like there's not even, it's not clear where it needs to evolve.

• 10:06Speaker 02

  Absolutely. What spotted on account. This is exactly. So we need to set up the North Star. And the enterprise, it's a single word, but it's not as simple as math and code. Math and code are the true kind of like, we have a lot of breakthrough in terms of intellectual capability advancement in the past. But if you think about math and code are the things that majority of the people who hands on the code truly understand, right?

• 10:34Speaker 01

  There are also, there are also domains in which it's, if not trivial, much easier to define what correct looks like. So you can build an optimization function and have an iteration loop and things like that.

• 10:46Speaker 02

  Absolutely right. This is where the reinforced learning with verifiable reward is actually bring a lot of capability advance in the last, I think last year, let's make it the time flies. Yeah.

• 10:59Speaker 01

  I know, right? A year, a year these days is like five years for.

• 11:04Speaker 02

  Exactly. I think we're starting from the early of 2025 where we're thinking about reasoning models. So now I'm thinking about this like one year and a half. Yeah. And you're absolutely right. Having a precise definition on what is correct, what's right, actually bring a lot of benefit in the post-training process with R. But then in the enterprise domain, and back to what we're discussing about. If you think about enterprise, it's a single word, but there's so many, it's, it's even hard to define what it is. There's so many verticals. Right? And then let's say healthcare. Now within healthcare to think about like, there's primary like providers, medical centers. There's insurance companies. There's pharmaceutical life science companies.

• 11:44Speaker 02

  They're just the, even with health provider, there's big like academic medical centers. There's community healthcare service. So they have very diverse needs regarding what they're gonna use AI for and what they regard as acceptable AI service are. So that's kind of like the complexity of the enterprise world. And the complexity is not very well captured and presented to the labs developers. That's one of the root cause for where the capability gaps are.

• 12:19Speaker 01

  Yeah, that makes a lot of sense. So tech layer, there's some gaps, absolutely. But let's not leave it there. Let's look at these, the other layers too.

• 12:27Speaker 02

  The other layers. Yeah. So that's kind of like starting from the foundation model, but then we also talk about the system side, right? So not just the models, when you develop application, it doesn't matter big or small solutions or, you know, reusable applications agents. You need to have it to be integrated with the enterprise context environment. You need to understand what data are, how to get yourself authenticated. When we see yourself is, is the agent of selves to get authenticated to the right data. And also to understand for all the enterprise services, their APIs, their data schemas behind it.

• 13:07Speaker 02

  What do they mean? Right? With different table names, with different value names. How do we actually understand the context? A lot of cases, people talks about context engineering. How we actually learn those tribe knowledge within the enterprise? And bring that knowledge to be part of the context into the agent. And that is actually a non-trivial problem, right? So because it's not like all the information is right there presented to you. The information.

• 13:39Speaker 01

  Absolutely. Like I'm dealing with exactly this in a startup environment where we have like, you know, a handfuls of systems and a few different databases. You look at an enterprise that's been around for decades and has hundreds of thousands of people, like this is a mammoth problem.

• 13:54Speaker 02

  Totally. And then you hit another point in terms of the pain point, what we see is when we think about enterprise and then we talk about like, this is a new era of AI waves. But then let's, let's look at in the past two decades, when we people are talking about digitalization, like everything from paper to digital form, digital asset. There's a lot of technical depth actually in that wave. Cause during that wave, the entity who actually are the information receivers are human. Human are extremely flexible and they are able to tolerate like, you know, it give me a PDF, give me a handwritten notes.

• 14:35Speaker 02

  It's not like perfectly digitizing a way that is, you know, to the high standard, but I can still get it. This is actually like the problem we get, for example, from the engagement with mail clinic, right? We think right now, all the electronic back to record system are digitized, right? It conforms a uniform standard called FIRE. But the reality is there are always like patient who are not actually within the network. They go through another system. They come to you with very complicated medical history. They're not going to give you the standardized, normalized electronic medical records. They will give you hundreds of pages of their historical medical records printed in PDF. So how people can read it?

• 15:21Speaker 02

  But then in order to actually ingest the data from different modality with the technical depths from the last two decades, fragmented, received, processed by human being in the past. Now you have to plug in an agent that is powered by foundation model, large language models. Then the gap is actually emerged from there.

• 15:44Speaker 01

  Yeah, yeah, absolutely. Like every shortcoming that we all have, that a human has just said, oh, you know, that's not perfect, but I'll paper over it. Now we have to go back and revisit and say, well, can an agent paper over it? Maybe not. What do we do with that?

• 15:58Speaker 02

  Absolutely. That's kind of like one of the things we, we, we experience. How we actually integrate an application, an agent into enterprise environment? Really address all these gaps from fragmented information, tribe knowledge. So all the things become the barrier, actually prevent accelerated deployment of AI solutions in enterprise.

• 16:23Speaker 01

  You're building agents that can write code, summarize documents and automate workflows, but they're missing one thing. Awareness of the world around them. X Weather combines enterprise-grade weather intelligence with agent-ready APIs, natural language capabilities and an MCP server built for tools like Claude, Codex, Copilot and modern IDEs. So your agents can adapt workflows, automate responses and make better decisions based on real world conditions. Backed by Vaisala, whose instruments fly on NASA missions to Mars, X Weather delivers trusted data and unique insights that go beyond conditions to actual impact. From real-time lightning strikes to road surface forecasts. Start with 15,000 free API calls every month and pay only for what you use as you grow.

• 17:03Speaker 01

  Your full weather stack for developers by developers. Start building for free today at XWeather.com.

• 17:12Speaker 01

  Your customer lives in the real world and it's messy. Intermittent connections, mid-onboarding drop-offs, edge cases on devices you've never tested. Mobile apps reflect reality in a way no other surface does. Yet from an engineering perspective, they're the hardest to understand. It's common for mobile engineers to see green backend dashboards, normal error rates, no crashes. But inevitably somewhere, there's a frustrated user watching your app spin. After a few seconds, they'll lose patience, close the app and turn their attention somewhere else. They may never come back. The worst part? Most observability tools never see any of it. Bitdrift, on the other hand, captures 100% of mobile data, unsampled and in real time. So it's immediately queryable by engineers and AI agents.

• 17:54Speaker 01

  It's mobile observability built for the real world. Try Bitdrift today. bitdrift.io/signup. This episode of Software Engineering Daily is brought to you by Warp Build. AI is writing more code than ever, which means GitHub Actions is running more than ever. Your GitHub Actions bill is now a function of how much AI code you generate. And every engineer knows the feeling. You push a commit and then you wait. Warp Build makes GitHub Actions twice as fast at half the cost with a one-line change to your workflow. Linux, macOS and Windows runners in Warp Build's cloud or your own. Enterprise-ready, SOC 2 Type 2 attested and trusted by teams like Sky from Comcast, Bitcoin and Brain Trust AI. Get started with $50 in free credits at warpbuild.com/sed.

• 18:39Speaker 01

  Well, and this kind of then ties into this last piece, which is the human layer. Like what needs to adapt in terms of how we're engaging in work?

• 18:47Speaker 02

  Absolutely. So this is like at the lower layer of the technical issues. But the solution to the technical problems really lies in the human's hands. And this is where, from organization perspective, the leadership commitment, the leadership, because all these motions, all these AI motions are, it's not going to be successful if there's not a clear incentive on what is the benefit, what is the return, is the ROI. If the leadership does not recognize the value of the ROI, and then when we actually get into all these barriers,

• 19:27Speaker 02

  then people could easily sort of, will not have a clear actual success criteria. People could easily pause at impressive demo and not pushing forward because there's no incentive to show the fundamental return and the impact, economical impact to the solution, right? So when there is, it's just drop it in the middle of nowhere and then not get into productionization. So from this perspective, the most important thing is the alignment at a high level, on the leadership level. What is the, where aligned on the North Dot successful metric? What is the expected output of it?

• 20:09Speaker 02

  And then driven by that, we would have a clear technical solutions, technical roadmap to say now, in order to achieve this high level North Dot success metric, what other things we need to do? What is the path? Now, what's the low-hanging fruit we need to do in the first step to get into the system? Foundation to be developed? And what's the next step to get to the user engagement? How we actually collect feedback continuously from the usage to improve the system from the agents? So those are the things that actually defining the driving force to be successful.

• 20:49Speaker 01

  Yeah. Having leadership agreeing on where to go definitely seems important. Another thing that I'm pondering about, like one of the things that I've seen and I've heard a lot of people struggling with is, at least in the coding space, which is the one that I, and I think much of our audience are most familiar with. These tools are really focused on individual productivity?

• 21:08Speaker 02

  Yes.

• 21:09Speaker 01

  But if you just drop a bunch of individual productivity improvements into an organization, without rethinking about your organization, you often end up just bottlenecked on the next layer that didn't speed up the same amount. And you don't see nearly the level of organizational improvement that you expect based on the individual. How is that showing up in these large enterprises in terms of like change management or reorganization or rethinking of how we organize ourselves?

• 21:35Speaker 02

  Yeah, this is such a great point. They're just a kind of really resonate what we see in the field as well. Like if you go to a lot of the enterprises and you ask them the AI adoptions, the answer could really go two directions depending on, you know, how people think about AI adoptions. If we say the people who are using cloud code, you know, GPT and you're using all this individual AI tools, your adoption is amazingly high. Everybody's using it, right? But they're not using it in the context of like exactly what you captured in contributing in a coordinated enterprise workflow. They just use it to improve individual productivities.

• 22:15Speaker 02

  And then to do that, there's actually two major problems, kind of very related. The first one is there could be a lot of IP leakage. Cause when you interact with an individual tool, all the queries you have, all the questions you have, there are a lot of enterprise confidential information. And even it's not proprietary data, there's also the way you ask question and then ask follow-up questions, review a lot of the judgment and the implicit knowledge within people's mind. And those information is actually leaked into the frontier labs. So that's one side of the problem.

• 22:59Speaker 02

  But on the other side of it is actually, these are very valuable enterprise digital asset is the IPs. And when people are actually getting used to just, you know, interacting with the individual productivity tools, all this digital asset, all this implicit knowledge are actually not captured and shared across the organization. And this is exactly the point you're talking about, right? In many cases, when you work on a project and then when you have experience and the context, the skills, right? All this expert knowledge from enterprise perspective, this information is to be consolidated across multiple team members

• 23:44Speaker 02

  and then into a coherent knowledge base. So that the next generation of the people who are working on the project will be inheriting the knowledge, the experience be accumulated from the previous conversation. So those are the things is actually missing by building this.

• 24:03Speaker 01

  So that kind of brings us to this, the flip side. We've talked about like where this breaks down, where the gaps are in different ways. What does success look like? I know you all recently published this report and you said, I think it was 6% of large enterprises are really right now succeeding at integrating AI and seeing the benefits. So like, what are the, the patterns? What are the ways that they get around these gaps? Like, how, how does that end up looking?

• 24:27Speaker 02

  Yeah. So when we have the 6% sort of the reports, the first thing we think about is how we define success, right? So like in, in this domain, defining success is actually not easy as well. Because as we said, like if you ask them about AI adoptions, they could say some shallow adoptions and that's not success, right? So the first thing what we did is we really think about what is the success criterias are. And then for that, we basically have a pretty rigorous way to say that. It's kind of like how people among all the projects people have been engaged in, how many of them actually go from pilot into production? And then meaningfully integrated with the workflow.

• 25:11Speaker 02

  And by meaningfully integrated with the current enterprise workflow, that is measured by very concrete usage metrics. So those are kind of like the things we're looking at, how we think about the success in a very concise way. We actually look at the characteristics, sort of to see what are the characteristics this 6% winners actually share. We are not able to attribute to say, hey, these are the factors that are causal factors. If you did this, you'll be successful. We're kind of like doing machine learning. We try to capture what are the shared attributes about all this 6%.

• 25:50Speaker 01

  Have you ever been able to make a true causal argument in a human system? Like that's, that's very, very hard. But yeah, I think that is useful, right? The pattern matching is what we're all looking for is like, okay, what are the patterns that I can try this? And maybe that will help move my organization closer.

• 26:05Speaker 02

  Yes, that's exactly some of the sentiment that like, you know, what we're doing. So there are a lot of findings in the reports, but the three things kind of really strongly reflected from all the statistics and the study we have. The first one actually is no surprising is the data foundation. And this is actually, we've talked about how data is fragmented, multimodal data, connecting data, actually to understand the data access policy. What are the governance? So how we actually collected feedbacks from the field? All this data infrastructure foundation is actually a prior, kind of like very strongly related with the success of AI sort of deployment success. No surprise, right?

• 26:48Speaker 01

  It's not surprising, but I feel like this is a thing that comes up and there's memes that have gone around, right? Of like, oh, we have this huge data mess. We could solve that. Or we could just sprinkle AI on top and make, hopefully it will work. And it all comes back to, no, you actually have to solve your data mess.

• 27:03Speaker 02

  It's a chicken egg problem, right? If you don't have the data, the AI will not do anything. But then when the AI is actually getting to the enterprise in the right way, then AI could be helping you with like, for example, one of the key technology we're actually providing to the market is they say entity resolution, right? So you have different tables, the keys, the, the way you name, you know, corporate entities are different in different tables, right? In the past, this is a very traditional machine learning problem, have been there for a long time. But with now with the large language model, with a lot of priorly trained word knowledge, they can do this job really better. So now if you bring AI to solve the problem, you have a much better normalized, standardized linked data asset.

• 27:48Speaker 02

  Then the next layer of AI agent is able to build on top of it and then discover and unleash more capabilities of solutions on top of it. So it's an iterative process, but you need to have the foundation together first. Yeah. Yeah. So that's one of the first thing we noticed. The second one, as we also chatted about, is the human part of it. Usually if there's early stage investment in terms of senior leadership sponsorship, and then they are committed in change of management. They provide resources in employee engagement and training. Cause personal usage versus enterprise workflow AI, a lot of cases have different steps, different UIs.

• 28:32Speaker 02

  If they actually would get into the code development, product code development motion and then envision when AI is co-pilot with, work with them in a workflow environment, then the success of adoption is actually much increased. Versus initially, we don't actually imagine what the change of management look like when the AI agents being deployed. So if the enterprise front load all the organization work, that is actually something we noticed among the 6% winners.

• 29:08Speaker 01

  No, I think that's, I think that's really important because, you know, I talked to a lot of people in a lot of different environments here. And it feels like one of the most common failure modes I hear about is essentially treating this as a technology and sort of magic adoption thing. Everybody must use AI. It's a magic technology. It will make everything better. Just do it. You figure it out. Versus the successful mode of like, hey, this is something new and tricky and we're going to work together and have change management and co-develop and really refine how we're working with this.

• 29:41Speaker 02

  Absolutely, and this is actually a great segue to the third point we actually discovered, which is like on the market right now, if you talk to enterprise customers, right? There's always these questions about buy versus build. And many cases, if they ask you, do you have a product that we can buy immediately? Right? And on the other side, if you get deeper with them, they're thinking, oh, we have a pretty strong engineering team. Can we build it together, right? Built internally, but the answer to the question is actually not that simple, as you pointed out. Right? So it's really, we need to bring the expertise from both sides. If you just get a product, even with like a customization consulting team to get to the enterprise side,

• 30:26Speaker 02

  we still face the problem how to integrate the product with the workflow, with the enterprise context. Right? And then how the product is actually going to be used? What's the C U J look like? What are the touch point for customization? So that's kind of like when on the other side, if they think, hey, we have the internal team, let's just build it. It's really going to fit very closely to our internal usage pattern, the workflow pattern. Then what is missing is the internal team. Of course, it usually is very strong engineering team with many years of seasoned experience. They usually don't have the broader understanding about the A I technologies.

• 31:11Speaker 02

  That actually, one of the benefit when I work very closely with the frontier labs, we actually have a best practice in terms of how we do evaluation of the foundation models. What are the best patents in developing agent around on top of these models? And what are the typical failure modes as well? So when we actually see the six percent winners, they strategically combine the internal expertise, which is that domain knowledge in terms of when this is useful, how to use it, and what outcome is actually going to change the workflow and decisions? With the specialized partners who actually bring in the expertise regarding

• 31:55Speaker 02

  A I models, A I application development and system development. So when the expertise from both sides are combined together, this actually creates the motion and the foundation to be successful.

• 32:08Speaker 01

  Yeah. So one of the things you mentioned there that I'd love to dig into, because I think you have done a fair amount of work in this space is around evaluation. Right? And I think one of the things that I definitely see hiring engineers and looking for folks to work on A I products and things around this is one of the biggest gaps is people who can grapple with non-determinism and those who struggle with it. Right? And the fact that in a non-deterministic environment, you can't do unit tests. You have to have some other way of validating quality and understanding what's within your sort of trained scope or the training corpus or the nice fit of the model and what isn't and all that sort of thing. So how do you all approach evaluation? And I particularly like, I think there's a lot of people thinking about evals at the prompt level.

• 32:51Speaker 01

  How do you think about evals at an agentic or workflow level?

• 32:55Speaker 02

  Wow, this is my favorite topic and then it's a list of questions that worth the time to dive in a bit. So in a very simplistic term, right, or think about evaluation, there are two things about evaluation. The first one is what needs to be evaluated? Right? And then, as you quite point out, like in the foundation model development, what we evaluate is the model, whether the model is going to work, right? And then that's the subject to be on. And then in order to evaluate, there are two things surrounding it we need to do. One is how the model behave and the expected input. This is actually super important.

• 33:36Speaker 02

  When we do model development, you test it under a very broad spectrum of queries. And the queries, of course, will be sliced into like different pillars, some are coding, some are conversation, and some are the worst real for risk assessment. And then the last part of it is when the output, doesn't matter from the model agent is produced, how we're actually going to tell is it good or is actually less desired? And in the undeterministic case, as like people recognize, there's another aspect of the robustness. Can you actually reliably, robustly to give the consistent output? When either there's you repeat your task or you have small perturbation in the input.

• 34:21Speaker 02

  So these are kind of the three ingredient of it in the context of agent evaluation. So the first one is actually the agent self, also the environment you're actually operating in. Right? So it's not just like, you know, what you develop the agent intelligence self and build on top of the model. It is also what is the backend environment? What are the tools that this agency is connecting to? What are the service A P I's that the agent has the access to? And more importantly, what is the backend state? How many database you are actually connected to? What are the file systems? What are the knowledge base that enterprise?

• 35:03Speaker 02

  So this actually defines the environment during evaluation. Right? Because if you change that, the evaluation result could change and that's the robustness question we need to get into. So people need to understand what are the environment, what is the entity we are evaluating? And then the second part of it is, as I said, like when you, when it's kind of software, you know, testing that you need to define your test cases to cover the happy path, the typical, the edge cases. So we need to strategically define what is the set of tasks that cover different dimensions of the capability and the functionality of the expected future, expected behavior of your agent.

• 35:48Speaker 02

  Right? So just give you like an example, let's say we are doing patient safety triage agent. Right? And then the goal of the agency is going to look at a report to say, is this an event I should report? And then what is the policy term that I need to ground my decision to? So this is the agent task. And now, when you get into the evaluation part, the high level mindset is actually very similar to what software unit testing is doing. Then you need to think about how do you design your unit test to cover different aspect of it? Right? Then you need to think about there's the positive case, you know, the event is, is true.

• 36:32Speaker 02

  But then you also need to cover different policies, like, you know, and the worst scenario. Because if you look at the event safety guidance, there could be one thousand, well, that's a little bit too high, like one hundred different criterias. Then you need to make sure your query actually hit the different categories of the events. So that you don't have some missing edge cases where things you don't have the confidence to speak about. And of course, you also need to think about the negative case and the hard negative case. And to think about what if there's ambiguity where you're missing evidence? What if there are terms that is not very well specified in the guidance?

• 37:16Speaker 02

  So you need to think about all these things, cases, and those are the evaluation data set curation.

• 37:22Speaker 01

  I totally agree. I want to dig in a little bit more about the environmental piece. Because I think looking at building out these types of eval frameworks in different environments, one of the biggest challenges that I have encountered is how do you get a realistic, but also like reproducible data environment, right? You don't necessarily want to run this against your product environment, or maybe you do, maybe that's like, right? But like, what does that whole setup look like?

• 37:47Speaker 02

  I think this is the open question, to be very honest. And this is actually a research direction we're working very hard. And then we actually have some result that to be shared with everybody coming up. And hopefully we'll talk to you again when it's out, you know? But you're actually asking a real hard question. The problem right now is if you look at across the industry at this moment, there's very limited, in a conservative, but if not now, evaluation environment that is backed up by real data. Because the real data from enterprise, it involves a lot of the privacy, legal usage concerns. But if you don't use real data, then all the dependencies, the very complex dependencies within the data

• 38:32Speaker 02

  will not be truthfully reflected in the environment you are developing. Right?

• 38:39Speaker 01

  Yes, exactly. This is exactly the problem I feel like we're all grappling with right now.

• 38:43Speaker 02

  We are all grappling with, yeah. The solution I can share a little bit of like how people approach it. Yes, there are data vendors and we're one of them, right? So we're actually looking at data in not actively being used, but it's similar, you know, in nature, in similar semantic domain, usage domain, but from the past expired data. And we do D I D, we do sort of anonymization over the data so that the sensitive information will not be leaked, but the semantic relationship will still be there. So there are some lot of research in that domain. People are looking for proxies, basically.

• 39:18Speaker 01

  Yeah.

• 39:18Speaker 02

  Surrogate and proxies to build a similar environment. But I agree with you, this seem to real is always a gap. A simulation to real environment is always a gap in the machine learning research deployment challenges. Yeah.

• 39:32Speaker 01

  I am curious, actually. So you mentioned this is an area that you all are providing. Like, what are the services that Scale is providing here in the enterprise space? Because we've talked about some of the needs and the gaps, and you've sort of alluded to, you know, we do co-development, we're working with people, you provide some of these like simulated data environments. Like, what actually does the, if I was reading it correctly, the Gen AI platform or the enterprise platform of Scale provide for folks?

• 39:58Speaker 02

  Yeah, absolutely. So at very, very high level, we do three things, right? We provide platform, which we call Skill Generative Platform. That's the foundation allowing all the system integration work, all the trustworthy agent execution, and then allowing the agent execution, all the traces to be logged and then for evaluation for auditing purposes. That's the platform we're providing, which provide the system infrastructure foundation to build a reliable AI solutions. So that's one pillar. Another pillar is actually our full deployment motion. So this is like, we're not just giving you a platform. What we do is we actually have a team, it's a very well-constructed team,

• 40:41Speaker 02

  which have strategists, purely people coming from management consulting like McKinsey, BCG. They would work with the customer to understand their business north metric, and then look at the use cases that allows us to produce the impact for outcome to meet that north star metrics. And meanwhile, we have the technical people who works very closely in the initial scoping process to say, what is technically feasible within the timeframe we have? And this we code define, we discover the use case, code define the use case to be worked on. Then we get into the delivery motion, where the full deployment people, the people with applied

• 41:23Speaker 02

  and as well as system background work together to really wire all the fibers, code all the AI solutions, perform the evaluations, understand the worst gaps are, and he'll climb on the quality. So that's kind of the full deployment service motion we provide to enterprises.

• 41:41Speaker 01

  Yeah, it's really interesting because I feel like this is a pattern that is starting to, like, if you look at our industry as a whole, we're evolving from essentially deploying software that was relatively repetitive, but in the enterprises we're integrating then, trying to find ways to integrate into their processes. And we're moving to a world where we're deploying a set of capabilities, which may, you know, has software, but really then is the details are implemented either by the company that it's sold to or by forward deployed engineers. Like, that's probably the most rapid growing software field right now is forward deployed engineers.

• 42:19Speaker 02

  Exactly. And then actually, just on top of what you're saying, here's another dimension. When AI capability is being deployed, it's fundamental, it's a learnable system, right? So it's not just like a one-shot thing. What a good full deployment engineer team like, you know, Skill AI, we're not just saying, here's the solution. We actually think more long-term. As we previously alluded to, how we actually help you to collect all the digital asset that reflected from the usage pattern through the traces? How would actually help you to identify the quality gaps? Then we talk about the evaluations, right? Where things were. And then provide you the research innovation, which was the last piece I was gonna talk about,

• 43:05Speaker 02

  to improve it progressively over time so that the agent becomes smarter over the time of people using it, because it would capture humans' knowledge, human skills, and then use it in the future sort of, it would incorporate it into the memory of the agents and then help its usage moving forward.

• 43:28Speaker 01

  That's interesting. Can we talk more about that? Because I think there's like, coming back to our co-development, there's like a co-evolution that needs to happen there as well, right? Because the agents become more capable, but then what people are trying to do with them evolves and kind of what that looks like. So how do you think about setting that up?

• 43:43Speaker 02

  Yes, I think that's actually a very essential piece when we get agent into the enterprise environment. Because it's a living thing. And don't think like in the previous era, when we say service, you know, a SaaS kind of era, you connect it, but what you get are just data into it. But then once the agent, agents actually serve as the co-pilot that works with you, then agent would actually learn from like, you know, Chris talked about, it will learn your institutional best practices as your institutional knowledge base, right? In a collaborative environment, let's say we have a group of people who are working on a project in like three months. Then when we actually work together on the same agentic system,

• 44:28Speaker 02

  then agent is able to actually accumulate our usage questions we had, working solutions that fills, would accumulate this project-oriented information. And then over time, it would make our project execution not more productive. And meanwhile, you know, on an individual basis, you could also remember what you did yesterday, what is your usage pattern, this, and so forth. So the agent would have components built in to learn from the interactions and become smarter and interact with people on the project's execution, workflow execution in a smoother way. It's actually one of the differentiations that would bring to the customer and enterprise.

• 45:13Speaker 01

  Fascinating. So how do you think about the evolution of human in the loop here? And what can get fully absorbed and automated versus where you have people checking in versus where you have this very co-pilot-esque approach?

• 45:29Speaker 02

  Absolutely. I think there are two dimensions of things. The first one is, you know, when AI gets into the enterprise, it plays a very active role from any aspect of it. But there's one thing is, who's going to be liable for the output, for the decision, right? Somebody needs to be hold responsible for it. You can't say this is AI did it. But in my humble opinion, I think the humans is going to play a long-lasting role in being responsible for the AI decisions and what the work from the AI is. Technically, how that's gonna happen, I'll get to that in a minute.

• 46:11Speaker 02

  But in the long run, humans stays. Human will be liable for it. What AI does would be helping AI in making decisions in various ways. But towards the end of the day, we need to have the people who are be responsible for the AI decisions in the enterprise environment. So that's kind of like a very long-term, sort of like the foundation of humans, stay human in the loop. But with that being said, there's also a tactical direction, like, you know what human does, right? So I believe that part is actually what we see is going to have a path towards initially when the trust between, like, when we talk about evaluation, right?

• 46:56Speaker 02

  There's offline evaluation, there's also online evaluation. Initially, before an agent is being used in the production environment for a long time to gain sufficient trust, there's always going to be a human in the loop component, where human will be triage. Ambiguous cases, human will be validating the high-stake operations, you know, where the system state got permeated. So right now, a lot of the right, almost in all the applications, if you see something is being written, and then there's always asked you to confirm, right? So all this risky, high-stake operations, human needs to confirm that. And then for even for the insight derived by the AI agents,

• 47:41Speaker 02

  human needs to review it to make sure it really aligns with the business policy and the business standard, right? But over the time, when the trust is built, and then the product is matured in a way that we are going to significantly reduce the cognitive load of human being. To a point, some of the very confident answers, confident operations from the agents should be just automated. But that is a tactic roadmap to initially, we have a higher human oversight to pay to get reliable trustworthy deployment. But this cost is going to reduce gradually when the mutual trust is established.

• 48:27Speaker 01

  Yeah. Well, and that fits with what we were talking about in terms of our lack of a truly high-fidelity eval environment. Is you gotta put it into the actual production environment and see how it behaves. And while you're doing that sort of live action evaluation, you don't want it changing key data or making big decisions. I think there's something interesting on the sort of long-term vision there. So you highlighted liability and responsibility, which I think is true. Like, that's an important piece, right? I think it's really important that whatever, and we're already seeing this in the coding world, right? People will be like, oh, the agent made this decision. Well, who, who said it was okay? Who committed it? Like, you, you still made a decision there, even if your decision was to delegate your thinking.

• 49:09Speaker 01

  But I feel like, like in that framing, it feels very burdensome, right? It's like, the agent's doing the work, but you have the liability and the responsibility. What's the positive side of that? What is the, what we're going to be doing as we automate more and more in terms of like this direction, decision-making, exploration? Like, I think there's something on that side of the world as well.

• 49:31Speaker 02

  Yeah, absolutely. So to your point, I feel like people are still trying to figure out what is the cognitive interface between AI and human, right? Fundamentally, AI is actually expanding humans' intelligence and capability, which leads to higher productivity as well as a higher quality decisions, right? But then to your point is where, what is the interface? Where it actually helps instead of asking me all these trivial questions right now? Like everybody's in cloud code or like yes, confirm is confirm. After a while, I got fatigue, right?

• 50:13Speaker 02

  And then not going to be making the right decisions anymore. So this is actually also from a research perspective. Actually, people are actively looking into in a sense that when the AI knows what is the point where we truly need human input, right? When he's sure about what's doing, when the AI is not sure about what he's doing. And that kind of like calibration capability is something that people start to look very much deeper into. And they still kind of like open research at this moment.

• 50:49Speaker 01

  Yeah, there's something interesting around that in the I remember digging in at some point, you know, large language models, your default interface doesn't actually expose the confidence at all.

• 51:00Speaker 02

  Exactly, exactly.

• 51:01Speaker 01

  But you can, especially if you're doing an open model, right? You can look at what was the distribution of logits on there and see like which one, you know, this was very sure versus this was actually pretty even spread, could have ended up in a bunch of places. So there, there's something fascinating at exposing that, you know?

• 51:18Speaker 02

  Exactly. There, there are research about like how the logits and then, but the logits at token level. And then now you think about like a long sequence, right?

• 51:27Speaker 01

  Right. Yeah. What does this look like at an agent level or a task level or something? Yeah, yeah, yeah.

• 51:31Speaker 02

  Exactly. And it's calibrated, right? So if the agent said sixty percent confident and that actually empirically calibrate with the statistics on that. So this is open research at this moment. And then people are still thinking of like talking about sometimes the agents are overconfident on many things. Yeah, that's actually a really fascinating area. People should really spend more time looking to one, especially while we're getting deeper into productionization and the human agent collaboration.

• 52:00Speaker 01

  Yeah, yeah, I've been thinking about this a lot with regards to some of these like entity extraction and context extraction and things like that. Like, can we add a confidence layer, right? Okay, we are eighty percent certain this is all one entity, but we need some more evidence. How could we validate things like that?

• 52:17Speaker 02

  Yes, yes, completely.

• 52:20Speaker 01

  Awesome. Well, so we're getting close to the end of our time together. Is there anything we have not talked about yet that you think would be useful or important for us to discuss?

• 52:29Speaker 02

  Yeah, this is a really fruitful and productive conversation. Really enjoyed it. Yeah, I think we covered everything. Yeah. What do you think?

• 52:38Speaker 01

  Yeah, I mean, I think we covered a lot of ground.

• 52:40Speaker 02

  A lot of many, many ground. Yeah.

• 52:42Speaker 01

  Awesome. Well, I think let's maybe do one thing back. If you were to look, you just did all this research looking at the different failure modes inside of the enterprise and what these six percent companies are doing really well. What would you say is if someone was working inside of one of those enterprises, but was not maybe in a leadership role? Or maybe they are in a leadership role, but you know, it's software engineer listening to this or a line manager or something like that. Like, what's the leverage point you would push them towards to increase their chance of success in terms of enterprise adoption?

• 53:15Speaker 02

  Yeah. If there's a single point, it actually, I think, is boiled into get the right pilot use case. And there's a lot of things within the right, right? The right thing is to be truly valuable. People have the incentive to make it work. It's not a toy use case, right? And then with that, then you could actually organizing, you could even like a software engineer, they could convince their leadership because it aligns with their business metric, right? So there's a lot of things within the right part. The second part is the right use case needs to be ready in a sense that you actually have the right data. You have the right resources and asset to make it correct.

• 53:59Speaker 02

  And the third one on the right part is actually you can think about whether the technology and expertise within the team outside is actually would enable this to successfully execute it and landing room. So it is basically find the sweet spot to start with and the success would compound from the first successful pilot to production launch.