#247 Barr Moses: Why Reliable Data is Key to Building Good AI Systems

This episode is sponsored by Netsuite by Oracle, the number one cloud financial system, streamlining accounting, financial management, inventory, HR, and more. NetSuite is offering a one-of-a-kind flexible financing program. Head to https://netsuite.com/EYEONAI to know more. In this episode of Eye on AI, Craig Smith sits down with Barr Moses, Co-Founder & CEO of Monte Carlo, the pioneer of data and AI observability. Together, they explore the hidden force behind every great AI system: reliable, trustworthy data. With AI adoption soaring across industries, companies now face a critical question: Can we trust the data feeding our models? Barr unpacks why data quality is more important than ever, how observability helps detect and resolve data issues, and why clean data—not access to GPT or Claude—is the real competitive moat in AI today. What You'll Learn in This Episode: Why access to AI models is no longer a competitive advantage How Monte Carlo helps teams monitor complex data estates in real-time The dangers of "data hallucinations" and how to prevent them Real-world examples of data failures and their impact on AI outputs The difference between data observability and explainability Why legacy methods of data review no longer work in an AI-first world Stay Updated: Craig Smith on X:https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Intro (01:08) How Monte Carlo Fixed Broken Data (03:08) What Is Data & AI Observability? (05:00) Structured vs Unstructured Data Monitoring (08:48) How Monte Carlo Integrates Across Data Stacks (13:35) Why Clean Data Is the New Competitive Advantage (16:57) How Monte Carlo Uses AI Internally (19:20) 4 Failure Points: Data, Systems, Code, Models (23:08) Can Observability Detect Bias in Data? (26:15) Why Data Quality Needs a Modern Definition (29:22) Explosion of Data Tools & Monte Carlo's 50+ Integrations (33:18) Data Observability vs Explainability (36:18) Human Evaluation vs Automated Monitoring (39:23) What Monte Carlo Looks Like for Users (46:03) How Fast Can You Deploy Monte Carlo? (51:56) Why Manual Data Checks No Longer Work (53:26) The Future of AI Depends on Trustworthy Data

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The way in which the data estate has changed in the last 10 years has been pretty radical. The way in which people manage and store their data has changed dramatically. However, the way in which we manage that estate hasn't changed almost at all. And that's really where Monte Carlo, that's the reason that we were created, is to help organizations that now need to manage this very complex system. Think of it almost as like a car manufacturing, right? You have all these different pieces that you need to put together. How do you know that the car is safe to drive? To put this in simple words, most organizations just build the car, drive it at 120 miles per hour, and then they learn that like something, you know, the door, you know, flies out or something, right? That's really how we've been treating reliability of data.

• 0:44Speaker 00

  What Monte Carlo does is connect to your systems across your data state and learn what your data looks like, help you first and foremost detect issues. We can talk about what sort of issues can come up and what does that look like, but first and foremost, allow data and AI teams to be the first to know about data and AI issues. And then the second is help them triage and resolve those issues.

• 1:07Craig S. Smith?

  What does the future hold for business? Ask nine experts and get 10 answers. Bull market, bear market, rates rising or falling, inflation going up or down? Can somebody please invent a crystal ball? Until then, over 40,000 enterprises have future-proofed their business with NetSuite by Oracle, the number one cloud ERP, bringing accounting, financial management, inventory, HR into one fluid platform. With one unified business management suite, there's one source of truth, giving you the visibility and control you need to make quick decisions.

• 1:51Craig S. Smith?

  With real-time insights and forecasting, you're peering into the future with actionable data. If I were a larger organization, this is the product I'd use. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities. Speaking of opportunities, download the CFO's Guide to AI and Machine Learning at netsuite.com/ionai. That's NetSuite.

• 2:28Craig S. Smith?

  netsuite.com/eyeonai E Y E O N A I all run together to get the CFO's guide to AI and machine learning. The guide is free to you at netsuite.com/eyeonai. netsuite.com/eyeonai.

• 2:53Speaker 00

  Thanks for having me, Craig. I'm excited to chat with you. My name is Bar Moses. I'm the co-founder and CEO of a company called Monte Carlo. We're the creator and leader of a category called data and AI observability. We started the company about five years ago. We're honored to partner and serve more than 400 enterprises for various industries. So these think about companies in gaming, media, retail, fintech, technology, financial services, et cetera. Each of them have lots of data teams, whether it's data analysts, data engineers, machine learning engineers, AI teams. And they're all working to create data products.

• 3:35Speaker 00

  A data product can be a report that an executive is looking at. It can be a specific dashboard they're using to drive decisions. It can be an internal product or it can be an external product that a customer is using. Or it can be a generative AI application. In each of those instances, data teams are responsible to make sure that those data products are reliable and based on trusted, accurate data. That's what Monte Carlo does. We help organizations use data by making sure that that data is of high quality and high trust.

• 4:07Craig S. Smith?

  Yeah, I had a guy on not too long ago and they had a product called Data Refinery. And I know that there are a lot of other companies doing similar things because the data that we're talking about for training or fine tuning AI models is largely unstructured. And organizations have just tons of unstructured data in various forms. And these guys take that unstructured data and then

• 4:45Craig S. Smith?

  Transform it into something that's machine readable. Is that, do you guys do that as well?

• 4:52Speaker 00

  We, um, uh, no, so we don't do that. What we would do in that situation, um, let me give, take a little bit of step back first. You know, I think in the last sort of decade or so, most of the work was done with structured data. And so monitoring the, or observing the reliability of structured data is something that is critical. And we do that very well. And, you know, just as an example for the things that you want to look at, you want to make sure that the data, um, you know, that the field level is accurate, you want to make sure that the data arrives on time, et cetera. I can go into more details into what that means. I think now there's sort of a rise with unstructured data, uh, to your point.

• 5:33Speaker 00

  I think companies are sort of enterprises are earlier in their journey for that. And one of the, you know, important things that folks are doing right now is trying to understand how to monitor and observe the quality of unstructured data. That's a very difficult problem. One of the things where we sort of, um, where we can help is it is common for, we've seen instances where our customers have, structured, unstructured data, perhaps like in the example that you suggested. And then used observability to make sure that that data is accurate. So maybe let me give a practical example to bring this to life a little bit. You know, we work with a Fortune 500 insurance company and in insurance companies, there's a lot of, support tickets or support conversations that happen.

• 6:17Speaker 00

  And so, you know, support conversations with, um, uh, end users and support,

• 6:24Speaker 00

  service teams are, uh, recorded, um, and those are basically unstructured data. And so, um, what this company did was, uh, used L L Ms to actually, score or grade those conversations on a scale from zero to 10, where 10 was, you know, the sentiment was very positive of the conversation. And zero, the sentiment was very negative. And so that allowed them to have a view of, you know, here are the positive conversations, here are the negative conversations, sort of do an analysis of that. Monte Carlo or where data and AI observability helps in this case is actually observing the output of the LLMs. So imagine that you've got a score of thirteen for a conversation. What does that mean? Or a score of negative two.

• 7:09Speaker 00

  How do you interpret that? When the scale is zero to ten, those outlier values don't mean much. And so actually having observability to make sure that the outputs make sense. In this case, how you can start to unpack what does it mean to monitor unstructured data? Does that example sort of make sense? Help it make it real?

• 7:28Craig S. Smith?

  And first of all, why Monte Carlo is did you use Monte Carlo Tree Search or something like that?

• 7:36Speaker 00

  Great question. We actually do not. It goes back, it traces back to sort of the origins of starting the company. You know, we're data people and like to work with, you know, proud ourselves to be working with the best data teams in the world. And so we were looking for a name that would resonate with data teams and, you know, be perhaps a bit catchy. And, you know, I think we have some Monte Carlo fans at our company in terms of, you know, the Formula One racing. Yeah. And so those are some of the fun origins, if you will. So it's both, you know, something that data team, data and AI teams know and find familiar,

• 8:21Speaker 00

  but also sort of a nod to the team.

• 8:24Craig S. Smith?

  Yeah. The, and so this, it's a, it's a platform. Is it a SaaS?

• 8:33Craig S. Smith?

  Is it a service or is it a tool that people subscribe and then can use? How does it work?

• 8:42Speaker 00

  Good question. So by and large, it's a SaaS platform. And, you know, the most common way in which organizations work with Monte Carlo is.

• 8:52Speaker 00

  Observing their data state. And so what do I mean by that? You know, the way that organizations today process, ingest, generate, create data is through a plethora of systems. And if you think about, you know, folks migrating to the cloud, you know, a very common stack, if you will, can include, you know, sort of upsource systems, or sorry, upstream source systems. You might have a data lakehouse, a warehouse, like Snowflake, Databricks, AWS, Azure, GCP. You probably have orchestration solutions like DBT, Airflow. And then you probably have, you know,

• 9:37Speaker 00

  either an AI model or you might have a BI solution like Looker or Tableau.

• 9:48Speaker 00

  So that you can imagine sort of this pretty complex data state. Now, you know, the whole premise of Monte Carlo is, you know, the way in which the data state has changed in the last 10 years has been pretty radical. The way in which people manage and store their data has changed dramatically. Like, you know, those solutions that I just mentioned really grew in popularity in the last five to 10 years. However, the way in which we manage that estate hasn't changed almost at all. And that's really where Monte Carlo, that's the reason that we were created, is to help organizations that now need to manage this very complex system. Think of it almost as like a like a car manufacturing, right?

• 10:34Speaker 00

  You have all these different pieces that you need to put together. How do you know that the car is safe to drive? You know, most, to put this in simple words, you know, most organizations just build the car, drive it at 120 mile per hour, and then they learn that like something, you know, the door, you know, flies out or something, right? That's really how we've been treating reliability of data. What Monte Carlo does is connect to your systems across your data state, learn what your data looks like, help you first and foremost detect issues. We can talk about what sort of issues can come up and what does that look like? But first and foremost, allow data and AI teams to be the first to know about data and AI issues.

• 11:17Speaker 00

  And then the second is help them triage and resolve those issues. Because the reality is that even if you're sort of inundated with

• 11:26Speaker 00

  alerts about things breaking, if you don't have a good strategy for how to do root cause, for how to do triage and resolution, your team is going to be wasting a lot of time on that. And so we think about sort of, you know, both of these things as two sides of the coin. Both of them are very hard to do well. And yeah, that, does that make sense in terms of what we do? How do they go into more depth?

• 11:49Craig S. Smith?

  Yeah, no, that's, and do you use? And is this primarily looking at data for AI models? Or is it looking at any enterprise data?

• 12:04Speaker 00

  Yeah, great question. So, you know, I think before sort of this, this AI craze, if you will, most organizations were really thinking about, you know, their, their data estate and not so much about AI. I think today you can think about one versus the other. And so I think any efforts to distinguish between the two are probably a mistake. And I'll explain why. I think, you know, it's, it's, if you think about this like two years ago or so, I think there was really sort of this, and there still is sort of competition in terms of, you know, what's going to be the best, latest and greatest LLM, right? And what's going to be the, the most important sort of foundation model?

• 12:45Speaker 00

  And the reality, and you're seeing this, you know, with OpenAI and Meta and Anthropic and, and, you know, with DeepSeek, we sort of appended the whole thing. And, you know, for, for folks who don't know, DeepSeek is a, is a company that's sort of allowed to, to run these models at significantly lower costs, like orders of magnitude lower and, and basically sort of, you know, put into question the whole reason of, you know, how, you know, how, how to build these things at scale. And I think what, what we are seeing is that Every single company has access to the latest and greatest model. And so, you know, every single organization can, um, you know, uh, use OpenAI

• 13:28Speaker 00

  or Anthropic or any other solution. And so, in a sense, that doesn't actually, um, that doesn't actually present itself as a competitive advantage. Because if everyone can do it, like, everyone has access to, you know, the, you know, the, whatever the best 2,000 PhDs can, can create. Then, and then so the question is, what is, what is my competitive advantage? Or what is my moat? And I think what we are seeing in working with, with companies is that the moat is really the data that's powering

• 14:01Craig S. Smith?

  Yeah.

• 14:02Speaker 00

  those models. Because, um, and I'll just give a particular example. Um, Intuit is a company that has a financial, sort of AI-driven financial assistant, which basically, you know, takes, is based on RAG pipelines. And, you know, takes information about, you know, um, what kind of car Craig has and where does he live? And all this information about Craig. And then surfaces back to you, um, recommendations, tax advice, recommendations. And so, if, you know, the, the quality of that AI product is only as good as the quality of the data that it relies on. And so if I have the wrong information about you, Craig, I might be surfacing the wrong information.

• 14:48Speaker 00

  And that is terrible. Uh, it can talk about, it can talk about some disasters that companies have had uh, by relying on, on wrong data. So, um, actually, I think United Healthcare in 2024, um, denied some claims, uh, based on wrong data.

• 15:03Craig S. Smith?

  Yeah.

• 15:04Speaker 00

  And so, you know, there, there's significant sort of societal and, and, um, brand, if you will, implications of forgetting the data wrong. And so I think what we're seeing is that a lot of organizations are now rushing to get their data right. Sort of be AI-ready, if you will. So that the data that's powering their AI applications can be relied and reliable and trusted. We really sort of more than, it's actually, you know, just, I'll share another anecdote. We recently did a survey with data leaders and we found that A hundred percent of them feel pressure to build AI applications.

• 15:46Speaker 00

  And ninety, I think ninety-six or ninety-seven percent of them actually do. Which is great, you know, I guess folks are up for the for the challenge. However, only two-thirds of them actually think their their data is AI ready or that their data can be trusted. And so there's this tension where people really want to build with AI, have are putting the resources behind that, but don't have the foundational data or the systems to serve that data in a way that makes their AI products reliable and trusted.

• 16:22Speaker 00

  And because of that, I think we're seeing more and more companies working hard to make sure that their data is actually accurate and reliable. Can talk a little bit about the reasons for why data goes wrong. I think that's like one of the key things that folks are trying to wrap their mind around. Because it's actually not, it's not straightforward to try to understand how to observe your data and AI estate. Let me just pause there.

• 16:51Craig S. Smith?

  Yeah, well, but yeah, I want to hear that. Is your platform or your system, is it a generative AI platform? I mean, do you have a conversational interface that that then, you know, looks at the data and can answer? Or is it is it more of an IT solution than that?

• 17:15Speaker 00

  Yeah, I mean, I would say so. It's that's a great question. So we definitely have AI features, if you will. So as an example, you know, we, when we start working with a data team, the first thing that they want to do is understand what their data looks like and understand what they should monitor. Um, and you know what they should monitor could be, for example, like, let's say if we're working with um the sports team or um um baseball team and they're collecting information about um uh you know, player stats, for example. And so they want to set up monitors about, you know, um the speed at which the ball is thrown,

• 17:58Speaker 00

  right, or um any sort of statistics about the players, as an example. And so actually, like, coming up with the list of monitors for that is quite hard. And so what we do is we use AI to come up with a recommendation for which monitors you should have. Um, and so we actually analyze sort of the data, profile the data, and suggest that, make, make recommendations, basically. That's one example. Another example is, um, you know, we allow, um, uh, we allow data teams to basically write in natural language what sort of monitors they want. Um, we use AI to help them create those. We also use AI to help them, um, basically understand correlations between, uh, different, um, things that go wrong.

• 18:42Speaker 00

  So what I mean by that, if someone has sort of a, you know, detects sort of a data issue or, or an alert, they want to understand the root cause. The root cause can be actually a, um, an amalgamation of, of a few different things. So we think of the, the things that can go wrong as the data, um, the systems, the code, and the models. So the data you sort of mentioned, you know, we have both structured data and unstructured data. And, you know, an issue in the data could be, um, like, just a problematic data source that sent a bunch of nulls, for example. Um, systems, you know, it's, it's your model APIs, your orchestrations, we sort of talked about, about that before, your, and, and the problem can be like, just, you know, a job.

• 19:26Speaker 00

  Failed as an example. So that's data systems code. Code is basically sort of, you know, what engineers write to transform the data. And you could have an issue with the code that happens all the time. You know, if someone makes a change of the code, has a mistake in the code. And then the fourth is is the models. And so, you know, you could have a outdated version of a model. The model might not be served on time. Or there might be inconsistency in the results of the model. You could have the same prompt sharing a different result each time. Or maybe the prompt is poorly written and doesn't have all the contextual information to provide a good answer.

• 20:07Speaker 00

  And so, you know, when when there's a data issue, the root cause can be traced to a failure in the data, the systems, the code, or the models. And oftentimes, it's actually all of the above together at once. And when a data team needs to decipher what, you know, what went wrong and why, that's where observability can help. And so we use AI to help explain what went wrong and why. So we help correlate between the issues. You know, there's an orchestration job that failed. There's also schema change that happened at the same time. There's a pull request that's related to those. And there's a problem with a third-party data source.

• 20:50Speaker 00

  And those three things have resulted in this model failure or in this report with the wrong data. So we use AI for that as well.

• 21:00Craig S. Smith?

  Yeah, you mentioned monitors a few times. What I'm not a data person, so what do you mean by monitors?

• 21:07Speaker 00

  Yeah, great question. So monitors can be monitors of all of, you know, there's a few types of monitors. Maybe the most basic type of monitor could be, you know, I receive data from Craig every morning at 8:00 AM. And I didn't receive data this morning. I don't know why. You know, in that sort of a real life example, this might be, I receive information from LinkedIn every morning at 8:00 AM. And this morning at 8:00 AM, I didn't get information. The API failed. So you can set up a monitor and automatically tells you, let me know if the data did not arrive at 8:00. Another example of a monitor can be, um, you know, I typically get from LinkedIn at 8 a.m.

• 21:49Speaker 00

  A thousand rows in my table, a thousand rows of, of data. And if I'm getting only 20 rows, or if I'm getting 20,000 rows, I need to know because something's weird. And so you might set up a monitor that tells you, you know, maybe the volume of the data here is abnormally low or abnormally high. Um, and I'll just give a final example, you know, let's say I'm tracking, um, you know, I'm tracking, uh, you know, what kind of toppings I have in a pizza. Um, and suddenly I get, um, chocolate. What sort of a topping is that?

• 22:30Speaker 00

  And so, you know, I might want to set up a monitor that says pizza toppings can only include these five values. And if I get anything else like that, then, um, please let me know that must be in the air. I cannot have chocolate on my pizza. That would be a disaster. So those, those types of mon, does that make sense? Happy to give more examples.

• 22:49Craig S. Smith?

  Yeah. No, does it, does, um, does data observability also go into, uh, measuring the bias of the data set in different ways?

• 23:02Speaker 00

  That's a wonderful question. You know, that's something that we're asking ourselves. We haven't built that, um, at, at Monte Carlo today. That's not, um, it's definitely something, uh, that I think is going to be very, very important. Uh, and for, for a variety of reasons, um, we haven't built it to date, but perhaps in the future.

• 23:23Craig S. Smith?

  Yeah. Uh, so, uh, it's a SaaS house offering and enterprise, uh, connects to your platform and then the, does the platform automatically scan your systems and identify data sources? Or do you have to, uh, list them?

• 23:42Speaker 00

  Great question. So we would typically want or would have would have the data team point us to their data warehouse or their data lake, their data orchestration.

• 23:54Speaker 00

  And sort of, you know, the various solutions in their data state. And those can be many and varied. And, you know, we would, we would typically want the data or AI team to direct us to that. And we would make the connection. The other thing that we would typically recommend is to start with a particular use case that you know matters. So for example, you know, you don't want to monitor. Or I'll explain why historically data and AI teams have done a pretty good job of hoarding data. And so there's a lot of stuff out there.

• 24:29Speaker 00

  Now, you know, whenever you put monitoring or something on observability or something, there's a risk of alert fatigue. And so if there's some data set that I really don't care about, perhaps I don't need monitor for it. But if there's a data set, you know, that's being used every single day. By the product teams or by my customers, something that's externally facing. You know, for example, we work with regulated industries. Where, you know, you think about like banks or trading platforms.

• 24:59Speaker 00

  You can't afford a stock number to be wrong. For, you know, for a very long time, or even for a short amount of time. And so that is data you need to monitor very, very carefully.

• 25:13Speaker 00

  You know, on the other hand, if there's data that you're using maybe once a quarter, maybe it's okay. You know, that the frequency of monitoring is is less and the severity of the monitoring. So, you know, we like to work with data teams on understanding what's the most important data that you have? What's the data that's really, you know, you're, you're relying on it? It's mission critical to your business. Customers are using that data. You know, this could be financial data, it could be marketing data. It can be customer platform data. All this, you know, more and more organizations are identifying these critical use cases. And so we recommend, you know, to start with those or to have a strong sense of visibility and monitoring into those for sure to to start out with.

• 25:56Craig S. Smith?

  Yeah. And you've written a book about data. What's the focus of the book? Is it, is it this observability question or is it broader than that?

• 26:09Speaker 00

  Yeah, it's a good question. So yeah, we wrote a book about data quality and data observability. I think there's sort of an interplay between data quality and data observability. I think, you know, historically, the problem of data quality has been around for a very long time. It's been around 20, 30 years. I do think the meaning of it has changed dramatically today because 20, 30 years ago, you really could get away with bad data. Very few people, very few organizations were actually using data. And if they were using data, it was very intermittently in small pockets of the organization. You know, maybe just to, you know, report numbers to the street once a quarter, for example. But today's world, if you think about the types of companies, you know,

• 26:54Speaker 00

  just take an example, Pilot is a company that has, sort of manages truck stops and fuel information and needs real time on information on, you know, where are trucks? What are the, you know, what's the fuel data, et cetera? And so in those instances, you can't afford to look at the, look at the data, the quality of the data only once a quarter. And so more and more companies are using data in a way that's very different than what it was five to 10 years ago. And so the definition of data quality needs to change as well. And so we really think of data observability as the next evolution of data quality. Data quality often stops at detection of issues.

• 27:36Speaker 00

  Data observability helps you take it from detection to resolution and triage remediation of issues.

• 27:43Craig S. Smith?

  Data is fascinating because since I've been paying attention, which is only since about 2017, there's just so much data that's being created in every organization. As you said, people didn't, didn't think about it a lot before because it wasn't considered that valuable. But now that you have AI that feeds on data, people are starting to store it and analyze it. And how, how many different kinds of storage, you know, S3 buckets? And, you know, relational databases and these, all these, is, is that

• 28:30Craig S. Smith?

  sort of surface expanding with?

• 28:33Craig S. Smith?

  Increasing numbers of data storage solutions and things like that. And can you connect to all of them?

• 28:45Speaker 00

  Yeah, you're actually spot on. So one of the things that we've seen happen in recent years is this explosion of these technologies. And so oftentimes, companies will have multiple from each of those categories that you mentioned.

• 29:01Speaker 00

  For different purposes, you know, one can be for analytical data, one can be for real time data, one can be for data science and AI use cases. So there's actually, you know, legitimate reasons for why you would want to have different ways to store and process data. Now, as there's been this explosion of these solutions, it's really hard to make sure that the data is accurate and reliable in each one of those. And so you're spot on. You know, the idea of data and AI observability, or what Monte Carlo does, is connect to each of those and help teams make sure that the data in each of those is accurate, reliable. And by that, I mean, all of the data is there. The data is actually the right data. It's arriving on time.

• 29:47Speaker 00

  The volume of the data is what you expected. The connections between the data are the right data. Its output is the right output. There's sort of a lot that goes into making sure the data is trusted.

• 30:00Speaker 00

  Sort of a variety of different techniques to make sure it's trusted. But you're right, it's a lot harder for organizations to do it now that data is spread across a very different number of integrations. So, you know, Monte Carlo, we have over 50 or so integrations that we connect to, where our team would deploy Monte Carlo, within 24 hours or so, would start having, seeing lineage. So lineage means that we automatically parse the connections between the tables and the fields to understand, you know, which, how is data flowing? And once we have that lineage, which is basically like a sort of think of it as like a graph of connections

• 30:41Speaker 00

  or like a tree of connections, then we can start understanding, okay, if there's an issue with this particular data set, what are all the different areas that are impacted by this data set? And if there's a problem here, what is the upstream root cause of those?

• 31:02Speaker 00

  And so the first thing we do is sort of have that picture of lineage. And then after that, work with the teams to help make sure that there's the, you know, the accurate sort of monitors on that data.

• 31:17Craig S. Smith?

  Do you, have you expanded into, so this is kind of DevOps, right?

• 31:25Speaker 00

  Great question. It's actually, it's, you're spot on. It's very much, we did not invent this. You know, try not to reinvent the wheel. We're taking what works for DevOps, which is exactly right. Observability for software engineering. And that's super well-understood concept. We've applied it to data and AI teams. So you're totally right. It's all of these concepts are concepts that are in DevOps. We are adapting them for data and AI. And our belief is that just like every single engineering team in the world has something to make sure that their software solutions and applications are reliable and trusted. There's a concept of uptime, right? Like is my application up 99.99% of the time?

• 32:07Speaker 00

  We believe that in the same way, every single data and AI team needs to make sure that their data products, data and AI products are reliable, are trusted, are powered by the right data, et cetera.

• 32:20Craig S. Smith?

  Does observe, is observability, I mean, I imagine it is related to explainability and there's, you know, a lot of work on explainability because a lot of enterprises.

• 32:34Craig S. Smith?

  Are are prohibited for regulatory reasons or whatever from using AI if they can't explain its outputs. Or the the logic behind the outputs. Do you do you move into that space at all?

• 32:48Speaker 00

  That's a wonderful question because explainability is so important. And when I think about most data leaders, data and AI leaders that I speak with. Their number one, you know, sort of mission for the for the year is often something like modernize our architecture. And make data more easily accessible to more people in the organization. And that's very hard to do if the data is not explainable. And so, you know, I I hear that.

• 33:17Speaker 00

  Come up a lot, sort of the question of of explainability. And that's something that a lot of data teams are spending time on. We specifically don't don't focus on explainability. I think it's it's something that our platform can certainly help. With explainability in terms of understanding, again, what are the, you know. If there's a particular table that you're looking at, what are all the different sources of that table? What is this table feeding?

• 33:43Speaker 00

  And so we can we can help with explainability, but it's not our focus. I will say that driving explainability of data when the data is wrong is probably the worst thing you can do. So and that is a that is sort of a mistake that we do see. Folks do is, you know, they'll try to drive adoption of data, but when the data is inaccurate, people will. Default to not using it or not trusting it. And then there's. You know, the brand of the data and AI team is tarnished.

• 34:15Speaker 00

  Right? Because they're basically supplying data to users who don't trust the data. And there's finger pointing and they'll complain and say, hey, the data looks wrong and. You know, you know, my my dashboard is blank, or, you know, the model output doesn't make sense. And in all of these cases, the finger is always pointed towards the the data or AI team. And so that is something that's that's quite destructive in an organization. So we will we will typically help with first making sure that the data. Can be used, is trusted, is reliable, and then also. You know, organizations will take that a step further and and make sure that it's it's explainable, if that makes sense.

• 34:53Craig S. Smith?

  Yeah.

• 34:55Craig S. Smith?

  Are you?

• 34:59Craig S. Smith?

  You know, you created this category of data observability.

• 35:06Craig S. Smith?

  How do you see the principles of data reliability and trust evolving? Uh, for to to address the the challenges of, uh, you know, a probabilistic technology like generative AI where you have hallucinations and you're trying to maintain

• 35:29Craig S. Smith?

  transparency and in the inputs so that

• 35:35Craig S. Smith?

  people can see if there's a problem in the outputs or or even if not explainability, at least be able to identify if there's a problem where it originated.

• 35:48Speaker 00

  Yeah, 100%. Um, and maybe this is a good a good time to talk a little bit about what folks are doing today. I think especially with you mentioned hallucinations and other things. I, you know, most organizations in that those instances are defaulting to sort of human evaluations. Um, where, you know, you can have like an end user like thumbs up or thumbs down or sort of score a certain response. Um, I think those are types of event evaluations are 100% a critical part of making sure that your model is performing as you expected and mostly that meets the user's needs and expectations, uh, if that makes sense. I I am finding that it's a necessary but not sufficient solution.

• 36:31Speaker 00

  Um, and I say that because, um, it's very hard to do things like gaining visibility into the problems that we talked about across data systems, code and models in a way that's manual and and relying on on human intervention. And especially when there's both structured and unstructured data.

• 36:54Speaker 00

  Sort of having automation to do that is a lot easier.

• 36:58Speaker 00

  It's also, you know, the, the, the advantage of having sort of automated or AI driven observability is that you can actually both monitor and troubleshoot changes across your entire AI value chain. And so oftentimes, an end user will only interact with the output, you know, of a certain prompt, certain model, but they don't actually have visibility to everything upstream. And oftentimes, the problems do occur upstream. And so having, you know, some way to monitor in a more holistic ways is very important.

• 37:35Speaker 00

  And again, you know, you sort of mentioned, I think detecting the issues or understanding that the data, that the, you know, that the output looks a little off is, is maybe only half of it. The other half that I think is going to be really important is being able to tie, you know, the quality of the output from the model back to the root cause that you need to address. And that could be because there's something wrong with the model, or it could be something wrong with the data itself. You know, it could be that your, you know, that the training data is stale. There's so many other reasons for why, for why there, there might be hallucination. You really need to have this full visibility into your data, systems, code and models to make sure that you minimize those as much as possible.

• 38:21Craig S. Smith?

  Yeah. And the, how is this presented to users? Is it a graphical interface with

• 38:34Craig S. Smith?

  charts and alerts and that sort of thing? Or is it a text interface that?

• 38:44Craig S. Smith?

  That's, uh, you know, giving insights into the data. I mean, how, how, what does it look like for someone using this SaaS product?

• 38:53Speaker 00

  Yeah, um, great question. So the most common, um, workflow, if you will, um, is for a data engineer to receive an alert, oftentimes through a solution like Slack. Um, or it can be a solution, um, like, uh, GEAR our service now, uh, or PagerDuty, where basically they're, they're receiving an alert, hey, something's wrong. So they might configure it such that if it's a, you know, SEV zero, the utmost priority or utmost criticality type of alert, they'll go directly to PagerDuty. Um, and the first sort of information that they get is what type of alert this is.

• 39:37Speaker 00

  So for example, the alert might be, you know, there's a particular data set that you expect to arrive at eight AM. It's now eight oh one and the data isn't here, for example. Or the alert that you might receive might be, you know, um, we suddenly got a feel that says chocolate for your pizza toppings.

• 39:57Craig S. Smith?

  Uh huh.

• 39:58Speaker 00

  Um, something off, come take a look at it, because this is weird. So the first interaction is actually getting that alert. And then typically what folks would do, you know, they would start to triage it. So they would click into the alert and then they would go into the platform. And then that platform, and the platform, what you can see is basically sort of an investigation page with all the various information in one place. So you can see, you know, this is what, um, you can basically double click into the specific alert. So you can see this is what the data looked like historically. And today, this is what we're detecting that's off. So for example, you know, every single day the data has arrived at eight AM. And just this morning, it doesn't. But then if you look, you know, a week ago,

• 40:41Speaker 00

  you can actually identify that, you know, once a month, the data is off, for example. Um, and so in that instance, you might say, okay, this is just seasonality. And then our model actually can adjust and learn to. That and say, once a month, there's a dip and the data arrives at nine AM, for example. Um, and so basically the idea is through the platform, you have various information, you can profile the data. You can um have different cuts of the data to understand what it looks like. And that allows you to better understand, you know, what what's sort of their their root cause. So we might surface information to you about. You know, um, uh, anything that's sort of upstream of that that may have changed. So basically correlated changes that happened around the same time.

• 41:27Speaker 00

  And so, you know, the team member might um, uh, you know, learn that there's a specific problem and then they would go ahead and fix that problem. So that's sort of the most common workflow. There's another workflow, which you mentioned sort of around dashboards. A dashboard can give you a more holistic view and it can give you that at the data product level. So what do I mean by that? You can actually construct a data product in in the in the platform and say, these are all the assets. That contribute to a particular data product, like for example, I'll use something domain based. Um, you know, this is a data product that helps our marketing team set up a campaign. Um, a particular campaign that they're running right now. And so I need to monitor this entire data product.

• 42:09Speaker 00

  And I want to look at the health of the data product overall. And so a dashboard might tell you, you know. The data has been fresh, the data has been up to date, the jobs have been running correctly. Um, you know, the SLAs are tight, sort of all the all the different things that kind of the the data product level. To give a sense of uptime of the data product, if that makes sense. So that sort of gives it a higher a higher level overview. And then for an executive, you can actually see a summarized view of that. So you can say. You know, my organization has more than a hundred data products. How many of them is are green? You know, okay, eighty percent of them are green. Great. Can we get to ninety percent of them to be green next?

• 42:52Speaker 00

  Quarter, what does that look like? And so data executives are starting to sort of manage. You know, the health of their data products and have the sort of visibility and also measure the success or the the health of their team in working through the incidents. So one of the most important measurements is time to detection. How quickly do we know about issues? And then time to resolution? How quickly have we resolved issues? And so data teams often measure themselves on that. You know, for example, it takes us, you know, an hour on average to to know about issues. And it takes us, we've improved in the last year from 20 hours down to three hours to resolve data issue.

• 43:36Speaker 00

  That operational muscle is very important for customers to build. And, you know, just starting by having a baseline is amazing for for many teams because they haven't been measuring this and they've been hit by, you know, their their customers or their CEO or whoever it is to make sure that the data is trusted and they don't have a way to communicate about how they're improving the data. So hopefully that gives you a sense of, you know, various workflows.

• 44:04Craig S. Smith?

  Yeah, so I imagine, I'm guessing there's sort of two phases to using Monte Carlo. One, you you first want to get a handle on on the health of your data. So you you identify all the data sources and you point Monte Carlo to them and then it presumably does it give a report on the on issues that it finds in the data and then you spend some time correcting those. And then the other phase would just be the monitoring phase where you've got this the

• 44:46Craig S. Smith?

  the data connected to Monte Carlo and it's just watching your your daily flow and identifying problems. Is that a fair assessment?

• 44:57Speaker 00

  Yes. Yeah.

• 45:00Craig S. Smith?

  And how long does it take to to get the to, well, obviously depends on how much data you have, but when when you're working with a large organization is is.

• 45:16Craig S. Smith?

  Ensuring that you have, that your data sets are healthy. Is that a process that takes weeks or months? Or, you know, for a Fortune 500 organization?

• 45:34Speaker 00

  Great question. Um, so, uh, you know, as a company, we've invested a lot in trying to make it as fast as possible for our customers. Um, you know, we know that, um, or I think that the days of, you know, signing up for a SaaS solution and then waiting for nine to 12 months until you see value. I don't think, I don't think that works anymore. I don't think, I don't think you can pull that off today, um, as a vendor. And so as a company, we've invested a lot in being very easy to work with. And that means being very easy to get started with. Also, you know, from a legal and commercial perspective. Um, and so our model makes very easy for organization to start. But also from a product and deployment perspective. So we invest a lot of time.

• 46:18Speaker 00

  We have a full engineering team dedicated just to making sure enterprises can get up and running very quickly. Because if you're not deployed, you can't see any value. So our goal and, you know, our, our team is incentivized only to make sure that the customer is deployed as fast as possible and can start using the solution. And so we, we invest a lot of resources in that. And so in the simplest instance, you know, for a small organization, that could be a matter of, you know, a couple of hours to just get integrated, um, and start seeing value very quickly, maybe even within a day. Right? That was sort of be on, on one end. I think for larger organizations, oftentimes, you know, they have security requirements

• 47:02Speaker 00

  and DevOps requirements and various permissions and authorizations that they need to go through. And so our job is to help shepherd them through that, to help them navigate that in the best way possible. And, you know, we typically try to do everything we can to make it as easy as possible. Uh, you know, to hold ourselves to the highest standards across all those areas and make it easy for enterprises to work with us in terms of the certifications that we have and, um, you know, our documented processes and policies to do that. Um, and so for an enterprise, you know, it's not, again, it's uncommon to, um, spend a couple hours, a couple of days to get connected.

• 47:44Speaker 00

  And then within a week or so, you can already start, um, rolling this out to your team. So again, you know, most of the, um, most of the work from a product development perspective on our end has been to reduce the amount of time on that.

• 47:59Speaker 00

  There is a component from the, the customer sort of enterprise side. And I will just mention, you know, there's also, one of the things, you know, sort of mentioned, you know, this is a new category. And so, um, it's not like data teams have been doing this for the, in the past. Uh, rather they are, um, developing a new muscle. And so there's actually a fair amount of thought that needs to go into what is our strategy? I talked a little bit about it, but what's the data that we care about? What are the integrations that we should start with? What's the data that's most important? What's our alert strategy? Who should handle the alerts? Who takes the pager duty alert? What happens when they do that? Who should they go to? Um, and so all of that stuff is, is things that we help operationalize.

• 48:43Speaker 00

  And so we have, you know, really strong customer success and, and partners that we work with to make sure that customers are, are really, really successful in that strategy. So I don't want to estimate, underestimate the importance of that part.

• 48:55Craig S. Smith?

  Yeah, uh, I mean, you were saying this is, uh, new, uh, and, and we're in this period of history where data is exploding and the importance of data is, is coming to the fore. How, what do people do without a solution like this?

• 49:13Speaker 00

  Great question. So, um, prior, you know, I want to say 10 years ago, they were just off the hook. Like, I'm just gonna be honest, nobody cared.

• 49:24Craig S. Smith?

  Right.

• 49:25Speaker 00

  Um, because the stakes weren't that high for data, you know? Um.

• 49:31Speaker 00

  At the most basic level, companies had to count the number of users that they had or the number of customers that they had. And honestly, companies still struggle to do that today. So I'm not sure that, you know, we've made a ton of progress there. But that was what was 10 years ago. Five years ago, when we started the company, most, you know, chief data and AI, or at that time was just data, officers would tell me something like, I will manually look at every single report before it hits my CEO. Yeah. And before I look at it, I have my four people on my team look at it manually. And so really, like five, six years ago, it would just be eyes on glass. Like you would basically have like, you know, how many sets of eyes could manually look at the data?

• 50:16Speaker 00

  That's what people did. I want to say a couple of years later, people started writing, you know, writing SQL code to create monitors. So basically say, please look at this data set and make sure that XYZ criteria is being fulfilled on this timeline. And so people would manually do that. You know, if you look at the biggest, you know, financial institutions today, you still deploy a lot of people who will do a lot of manual work and so will manually look at this data. I think that still exists. And I think to a certain degree, that's never going to go away. Some of it won't go away, right? You're always going to have, I think, some human loop or some intuition.

• 51:00Speaker 00

  But in today's world, every single or very many users in the organization use data. AI is being surfaced to externally.

• 51:12Speaker 00

  You know, I think that various companies are sort of deploying agents where an agent can have a conversation with a customer. And if that agent is promising that customer something, it better promise something that the company can actually deliver on. There's been instances, I think there was someone who bought a car for $1, basically, by tricking the agent into selling them. They had, I don't remember the brand of the car, but basically, you know, you have to make sure that. Your agents are not selling cars for $1, as an example. And so, you know, I just think the world in which we are today is one in which the solutions of,

• 51:56Speaker 00

  you know, manually looking at the data or manually writing tests just doesn't cut it anyway. But historically, that's what folks have been doing.

• 52:04Craig S. Smith?

  Is there anything I haven't asked that you want to talk about? I mean, I can see that this space must be expanding very rapidly. So you're positioned well, but is there anything that we haven't talked about that you think listeners should hear?

• 52:26Speaker 00

  I think we covered a ton. I mean, I think you had great questions. You know, I'll just end by saying I'm super excited for, you know, the next five years. I think if you look at like the last 10 years, the last five years, we've made a lot of progress. I think we're going to make even even more progress. And, you know, the thing, you know, where I find my passion, where I wake up every single day is, I can't imagine a world where data isn't the most important and AI is the most important part of our lives. I'm very excited about that future. And I think that future would be way better if the data was actually, the AI solutions were based on data that's reliable and trusted and accurate. And so we're really excited for that version of the world in which we want to live in.

• 53:11Craig S. Smith?

  Yeah, absolutely. Okay, Barr, well, that's fascinating. I mean, these are all these worlds that are connected to what I

• 53:24Craig S. Smith?

  spend most of my time paying attention to and that I know very little about.

• 53:30Craig S. Smith?

  Yeah, and if somebody wants to explore. Monte Carlo, what's the URL to get started with?

• 53:38Speaker 00

  Monte Carlo data.com.

• 53:41Craig S. Smith?

  Okay.

• 53:41Speaker 00

  I'll just repeat that Monte Carlo data.com. And you're also welcome to reach out to me on LinkedIn if you'd like to connect.

• 53:49Craig S. Smith?

  What does the future hold for business? Ask nine experts and get 10 answers. Bull market, bear market, rates rising or falling? Inflation going up or down? Can somebody please invent a crystal ball? Until then, over 40,000 enterprises have future proofed their business with NetSuite by Oracle, the number one cloud ERP, bringing accounting, financial management, inventory, HR, into one fluid platform. With one unified business management suite, there's one source of truth, giving you the visibility and control you need to make quick decisions.

• 54:33Craig S. Smith?

  With real time insights and forecasting, you're peering into the future with actionable data. If I were a larger organization, this is the product I'd use. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities. Speaking of opportunities, download the CFO's Guide to AI and Machine Learning at netsuite.com/eyeonai. That's netsuite, N E T S U I T E, dot com slash eye on AI.

• 55:16Craig S. Smith?

  E Y E O N A I all run together to get the CFO's Guide to AI and Machine Learning. The guide is free to you at netsuite.com/eyeonai. Netsuite.com/eyeonai.