Unlocking Enterprise Data with Knowledge Graphs and AI, with Juan Sequeda, Principal Scientist and Head of AI Lab at data.world

In this episode, Nathan sits down with Juan Sequeda, Principal Scientist and Head of AI Lab at data.world. They discuss how knowledge graphs can be your organization's "brain" for AI, integrating structured and unstructured data, benchmarking enterprise AI systems, and more. If you need an ecommerce platform, check out our sponsor Shopify: https://shopify.com/cognitive for a $1/month trial period. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. --- LINKS: Data.world: https://data.world/ SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. X/SOCIALS: @labenz (Nathan) @juansequeda (Juan) @datadotworld (data.world) @CogRev_Podcast TIMESTAMPS: (00:00:00) - Introduction to Juan Sequeda and data.world (00:01:11) - Discussion on data and generative ai (00:06:15) - Data.world's origins as an open data catalog platform ("Github for data") (00:09:35) - Using knowledge graphs and semantics to integrate and query data (00:12:52) - Main use cases for data catalogs: search/discovery, governance, data operations (00:15:00) - The process of building knowledge graphs automatically from data (00:24:29) - AI for unlocking and capturing tribal business knowledge (00:34:24) - Understanding the data landscape in enterprises (00:38:32) - The emergence of knowledge engineers and data product managers (00:40:44) - The consumer experience in data.world (00:45:36) - The importance of context in data analysis (00:46:58) - The role of AI in improving data analysis (00:48:08) - The importance of accuracy and explainability in data analysis (00:50:08) - Question cataloging in data analysis (00:51:44) - The future potential of "chat with your data" interfaces (01:18:02) - Finetuning with data vs metadata (01:29:24) - Future of enterprise data teams This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is the essence of what my organization is. It's the brain, right? Everything here is accurate. I can use this to explain things. The LLM, these foundational models, don't have that accuracy, don't have that explainability, don't know my organization. Do we expect these foundational models to know every single organization? No, because these things is private. I don't want them to know. But I want to go use them. So that's why this combination of these foundational models, large language models, with your internal brain of organization, which is your knowledge graph. That's what I see where there's a future. And I think what we need to work on is understanding best that integration point between the knowledge graph, the brain of the organization, and its foundational models.

• 0:44Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, we're talking data and how generative AI interacts with enterprise data with my guest, Juan Cicada, Principal Scientist and Head of the AI Lab at data.world. Data.world, like many established companies that we've featured on the show over time, built much of its platform, products, team, and business

• 1:29Nathan Labenz?

  before the current generative AI moment, and is now working to make sure that it's taking full advantage of this new technology paradigm for its customers. This is part of an economy-wide trend. Frontier generative AI models are now making their way into all sorts of high-value, often very challenging, contexts. From process chemistry labs to novel protein design to US federal courtrooms to the practice of medical diagnosis to enterprise data lakes, GPT-4 demonstrated enough raw power that we now have experts in literally every field working day in and day out to figure out how to make generative AI work for their organizations.

• 2:10Nathan Labenz?

  Now, it's not all instant success. From Juan's work, for example, it's clear that early benchmarks understate the complexity of real-world enterprise data and that naive "chat with your data" type implementations are not up to enterprise challenges. Even the more advanced work that Juan and team are doing with Knowledge Graphs, while it does deliver major improvement, is still at best a partial solution.

• 2:34Nathan Labenz?

  So while data.world, again, like most established technology businesses I've talked to, is not particularly worried about competition from fast-moving AI-first startups, they do see the transformative potential and they are realizing enough practical utility today that they are rolling up their sleeves and settling into the process of AI implementation and optimization.

• 2:56Nathan Labenz?

  Interestingly, zooming out, I think this creates the potential for another major phase change in the history of AI. While GPT-4 takes meaningful effort to implement and often still falls short of our dreams, we are nevertheless building foundational capacity for both last-mile distribution and customization, such that the next big model release will have the opportunity to almost immediately plug into many millions of live business processes and systems.

• 3:26Nathan Labenz?

  The electrification of America took 60 years and included significant public works projects, and all the appliances were designed with a clear understanding of exactly how they'd be supplied with electrical power. Today, in contrast, we live in an internet-mediated, software-enabled world in which updates can quickly be pushed to everyone, everywhere, all at once. Today's software application developers are building somewhat ahead of AI capability, both so that they can deliver frontier features to their users today and, more importantly, so that they're ready to flip the switch when the next big advance comes online. How many months will need to wait before GPT 4.5 or GPT 5? I don't know.

• 4:08Nathan Labenz?

  And it might be just long enough that folks start to wonder whether AI generally is underpowered and overhyped. But my expectation remains very clearly that we will continue to see additional jumps in capability and that with each future leap, considering the foundation now being laid, the deployment cycles will naturally get shorter and more disruptive. One note for listeners. Enterprise data is a complicated space and we spend some time in the first half of this conversation discussing the general state of enterprise data and data science teams today. If you're already well-versed in enterprise data science, you might find some of this a bit basic and introductory, but for most, I think the additional context will be very useful.

• 4:50Nathan Labenz?

  If you have any suggestions for how I can better handle the introduction of such advanced topics where listeners inevitably will have very different levels of prior knowledge, and I do expect this to be more and more common throughout 2024, I would love to hear your suggestions. As always, you can email us at [email protected] or you can DM me on the social media platform of your choice. And of course, we always appreciate it when you share the show with friends, starting in this case with the data scientists in your life. Now, I hope you enjoy this conversation about generative AI and enterprise data with Juan Cicada, Principal Scientist and Head of the AI Lab at data.world. Juan Cicada, Principal Scientist and Head of the AI Lab at data.world.

• 5:35Nathan Labenz?

  Welcome to the Cognitive Revolution.

• 5:37Speaker 00

  Well, thank you so much for having me. Really excited about having this conversation.

• 5:40Nathan Labenz?

  Yeah, me too. I have been interested in knowledge graphs recently. I've been kind of feeling from for a few different projects that I'm working on that more structured data, something between your kind of classic SQL database and the totally amorphous data that people are just increasingly vectorizing and hoping for the best from is probably going to be important. And I went on a little quest to see like who's working on this and what's the state of the art and came across a paper that you had published. So I'm excited to get into that with you, but I thought maybe we should just start with kind of an introduction to you and the company because I don't know how many listeners will be familiar.

• 6:23Nathan Labenz?

  You want to just kind of give us a quick intro to data.world? Like, how did the company get started? Was it an AI company at the beginning? We could take it from there.

• 6:32Speaker 00

  Yeah, happy to. So, okay, so Data World, we're an enterprise data catalog platform. So data catalog platform is essentially where an inventory for the all the data and the metadata list in the organization. So basically like your library card catalog, right? That's kind of the the the high level thought about it. Managing metadata has been something kind of since the beginning of just data management, but I would argue that it hasn't really been a focus till the last probably five, five, six years, which became kind of the the focus in the enterprise. If I zoom out and we look at what what data, the data management world has been, it's always about moving data. So ETL and ELT, storage and compute of data.

• 7:16Speaker 00

  So your data warehouses, your data lakes and what's up. And then you're using your data. You are now trying to your analytics, your dashboards, your RML and all that stuff, right? And then there's always been this, this, this thing on like on top that we haven't really kind of focused on it, which is the metadata, the data about the data. And and that is what data catalogs are focused on, on be able to kind of understand what's going across all the stuff that's being moved, all the different data sources, tables and columns that you have, keeping track of all your dashboards and how they're connected across all the different sources, and then also keeping track of your business terminology, your business glossaries and everything. So that's what a data catalog is. And so Data

• 7:56Speaker 00

  World started off in, I think, 2016, so very early on. And the focus was to be, the first phase of the company was to be a GitHub for data. So it is, we, they want, wanted to be a catalog for open data sets. And that was the first phase of the company for a couple of years. And we continue to be the world's largest open data catalog. We have over 2 million users, I think, like, two thirds of Fortune 500 folks are on Data World, the open data platform, half a million data sets. Like during COVID, all the COVID data was how some data that world was completely open. And it is continued to be open and we're actually a public benefit corporation.

• 8:39Speaker 00

  So that means that in addition to being a C corp and maximize shareholder value, we have a public benefit mission, which we evaluated on it. It's to be the world's largest and abundance data resource, advocate for open data and linked data standards and to be an archive for the world data. So those are our, it's a public benefit mission. And what we did is that also during that first phase of the company, the goal was from a technology perspective is create a platform that can scale at a web scale. And that's what we have over two million users. And so we really built a platform that we know that you could have like high scale and and then be able to go share data. So around 2019, the next phase of the company was, hey, we built this whole platform that the open data can use in it.

• 9:22Speaker 00

  Within organizations, people want to share their data and find it and so forth. So that's how we entered into the whole, what's now being called a data catalog. People are doing data marketplaces and stuff like that's all where that fits in. So that's kind of the phase of the company we're doing right now. Now, from a technology perspective, from day zero, basically, the entire platform has been architected on top of a knowledge graph architecture. Which means that we use, first of all, we're all about open standards. We're part of our public benefit corporation. So we use the open web standards of R E F, which is the metadata, the graph standard, OWL, which is for ontologies and schemas, Spark, which is for the graph query languages.

• 10:03Speaker 00

  And that's the, the architecture we use. So basically, everything that we bring into data that world, it's already turned into a graph. And, and why graph? You know what I'm saying? You, you start thinking about I'm moving data, I'm storing data, I'm using data, like there's all, all these pieces get connected across on the, all those pipelines. So you just really want to keep track of things and how they're being connected and so forth. And that's a graph. I always say that your first application over a knowledge graph is really the management of all your metadata to understand basically what is that infrastructure that I have within my organization. To understand, hey, there is this data comes from this source and then it gets moved to this thing, oh, so this data warehouse or this application, there's people who are using it and so forth. So you're keeping track of all of that.

• 10:47Speaker 00

  And then now with AI, what happens is that, uh, well, you need data, right? Data is the foundation of this. So I'd argue that we've always been a data company and, and the data is a bloodline for, for AI. So the data catalog and being able to manage your metadata, manage your semantics, manage your context. That is critical for the generative AI because that doesn't have the semantics, the context, the, the, the knowledge of your organization as it, of, of L L M, whatever it's been trained on, but doesn't have your own internal context. So we've always kind of seen this, uh, over the last year and a half when it comes out, L L M's coming out.

• 11:28Speaker 00

  For this to be applicable, uh, in the organization at a scale where you need accuracy, explainability. Knowledge graphs are critical for that because it provides all that context that the L L M's don't have. And that's what we're seeing and kind of what, uh, we've been talking about it and the, the, the paper that you mentioned originally was kind of just the evidence that we wanted to put together to say, yes, knowledge graphs are critical for this, assuming you want accurate, uh, question, uh, accurate answers for your questions.

• 11:56Nathan Labenz?

  Let me just ask you to get a little bit more concrete on a couple of things. It's awesome. You know, the two million data sets is awesome. The public benefit, um, aspect of that is, is really cool. I'd love to hear a little bit more about kind of those data sets, like what are the, what are the big attractors? You know, what are people coming to you for? You know, I imagine that again, like most people probably just don't know. And then I'm also really curious to get a little bit more concrete on. Okay. But like, what is a knowledge graph? Like, how have they, how do they typically get created? And like, what role does data.world play in helping enterprise customers do that work?

• 12:39Speaker 00

  So, so one thing is that is the open data catalog, which is open for folks, right? And then people want the, uh, the data catalog architecture for there to create their catalog internally. So I see there's usually kind of three main, let's call it applications that people could, that, that our, our customers will use data world for. One is when it comes to like search and discovery. I don't know what data I have, right? It's a typical problem like you have data scientists. We're looking for data and they don't know where to go find stuff. Your data lake has turned into a data swamp type of approach. So they need to have a way to go search and go find that data they have. So that's usually one of the applications around that. Uh, so search is something that that we're very focused on. Uh, and the same way you search things on Google today and you, if you search on Google letters, which for Austin, you get that panel on the side.

• 13:24Speaker 00

  It's, oh, Austin is the capital of Texas and, and here's all these events going on and here's the, the weather and here. Like that is what Google calls a knowledge panel. And that is results that's coming from their graph that they've built. So we do the exact same thing. So if you think about kind of what the knowledge graph inside of data world that we build automatically when we bring your data. Think about it from the, the, the concept that show up are, hey, there's a database. A database has tables, right? Tables have columns. Uh, there are dashboards, right? There are different types of dashboards, a tableau dashboard, RBI dashboard. Uh, this dashboard was built using data from this table or this query over these tables, right? So you start to see how things get related, connected around that. Uh, there was, there's this person who created these dashboard, right?

• 14:09Speaker 00

  There's this, there, then there's policies that we were kind of defining. Who can, there's a PII, there's, there, there's personal information. This table has PII data, right? So then all these things get connected. So that's kind of the graph that happens underneath. And when, when we go off and kind of connect and extract the metadata from all these sources, we build that graph automatically for you for all that, that technical metadata I'm seeing. So that's kind of that, that, that call it the digital landscape of all the, your data and that's connected with the organization. So search is one of those applications, uh, and over that big graph of, of, of your metadata. Another one is around governance. I want to know, again, let's keep track of all the policies around our data and who can use what data for what reasons?

• 14:53Speaker 00

  Who can access this data, right? But where, where, where is all the data, sensitive data and so forth? So that's all, that's the governance side. And then the third one, and we see it a lot, is for for data ops, the just operations of data, making sure that, hey, data is getting moved from different places. Again, keeping them moving is like what we call the lineage. Oh, well, this dashboard extracts data from this table. This table comes from this stuff and so forth. You have kind of that whole graph and lineage of where data is coming from, how it's living. And then you want to make sure that, hey, if I'm going to make a change somewhere, what could that affect? Or if somebody's saying, I have an issue in this particular feed of data, where could it be? I can kind of all that. So that's all the operations of data. And then, and you can be in your dashboard in Tableau, and then you can actually get

• 15:36Speaker 00

  notifications from there saying, hey, there was an issue just reported. So just be very careful with with the data that's being shown here and to be fixed and so forth. So that's the kind of a third application. So to summarize, data catalogs and our users, they use it kind of for three main things, which is for search and discovery, for data governance, and for data ops. And at the end of the day, all of this connected and we build that, we build that graph of that metadata all automatically once we have our collectors up and down.

• 16:04Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. I'm really struck by how many of those things seem like they may already be undergoing significant transformation in the AI era. You know, 2016, obviously not pre-AI, but pre-transformer, certainly pre-large language models, certainly, you know, pre any, you know, kind of general conversational interface, you know, with a with a database. I guess I'd love to hear a little bit more about how you build a knowledge graph automatically in a pre-AI era and how that might be evolving.

• 16:46Nathan Labenz?

  And then also kind of the kind and the and the and the way the data is stored. And I had Anton, who's the CTO at Chroma, one of the, you know, high-flying new vector databases on as a guest on the show not too long ago. And he made this really interesting point that the vast majority of data that goes into their databases has never been in a database before. When you were first building knowledge graphs automatically, how did you approach that? How is that changing? And how is the actual nature of the data beginning to change in light of what AI can now do with it?

• 17:21Speaker 00

  First of all, we need to think about when it comes to knowledge graphs, there is the notion of the schema of the ontology. Let's go define the semantics, the meaning of a particular domain. So everything I've been talking up to now, when it comes to data catalog, the domain there is like this technical metadata. And just just think about the map, the let's draw the bubbles and lines on the on the whiteboard. You have a table as part of a database, you have a dashboard, a dashboard takes takes data from a table, right? So that's kind of the the schema that we're creating. That that's one knowledge graph, we can create knowledge graphs about any type of topic. So for example, let's create a knowledge graph around e-commerce. Well, you have customers, you have orders, right? A customer places an order. An order consists of a set of order lines, and order line can have products, right?

• 18:06Speaker 00

  And the orders are shipped to an address, then you could have a shipping address, you can have a mailing address. So that's basically the schema you start kind of defining. You populate that coming, that that data may be coming from so many different types of sources. They can be coming from structured sources, relational databases. Some of that can be coming from unstructured, from text and so forth. They can come in from feeds, they come on JSON feeds or right from APIs or whatever. So the knowledge graphs are a means of integrating data coming from so many different types of sources. And you can model anything that you want around that. Now, what takes time always is what I call the the knowledge engineering process. It's understanding what do you mean by what is a customer, right? Understanding what you mean by what is an order or what is a an active order or so forth, right?

• 18:50Speaker 00

  And I think this is the human nature of of probably because you go ask multiple people. And they'll probably have different interests around that. So what is the correct version? What is the correct answer? So always argue that if not even humans will agree, how will the machines even know, right? The machines will definitely be able to generate suggestions. Oh, it could be this, it could be this, it could be this. But at the end, the humans need to go saying, no, we're our decision is going to be. This definition. And that's by the way, where governance comes in because you want to have an agreement on what things mean because otherwise then we can depending on the type of business you are, you can have different types of chaos. How things are being kind of slow to create uh what I'm calling creating these ontologies, be able to

• 19:32Speaker 00

  understand what the what this data actually needs. Now, what's interesting with L L M's today, like they're helping a lot of experiments we're doing is using L L M's to help us accelerate that process of like, hey, create candidate ways of modeling these things, right? And how about through a chatbot, go talk to the end users to be able to extract the stuff that's in their head to say, how, how, how do you define what a customer is? How do you define what an order is? So I think that's something that we're seeing right now for that. So then when it comes to like the vector databases, like one thing that we're seeing today is the the low hanging fruit of being able to kind of show some really cool AI applications, specifically when it comes to like chatting, chatting with your data.

• 20:17Speaker 00

  It's usually about unstructured. So it's the Texas, as you said, I mentioned, like the stuff that has never been inside of a database. So it's your documents, your PDF files, right? And then other places you have images and so forth. And that's where you want to have, that's where vector databases come in and you have to do all of this uh the the chunking and kind of all the bits pieces for the unstructured side. When it comes to structured data, the data that's in your relational databases and your data lakes that come into and have thousands of tables written, tens of thousands of columns, like that structure has meaning. And it's in in that meaning already, a user, a human being will look into that table and they're like, oh, this is what this means. Like I, you, you want to add these two columns together, but not these other ones, right?

• 21:02Speaker 00

  There, there, that is, there's, there's knowledge behind that. And I think that's harder to go do based on all the what I was talking before. It's like, but what does this mean? You gotta go talk to so many people. That's like not a scalable thing to go do. Again, L L M's today are helping us, are going to help us because now we can kind of have have these uh chatbots that can help us acquire knowledge. So the low hanging fruit has been focusing on the unstructured. But I would argue that the untapped potential here is to be able to go focus on the structured data. Because this is the data that goes into your reports, into your dashboard. Like, that's the stuff that executives and the boards are looking at those graphs to make decisions around that.

• 21:45Speaker 00

  And that's the data that's coming from your CRMs or ERP system. Like, this is all very structured data. And those types of questions that people are asking, they expect accurate answers. And they expect to explain where those answers come from, because they go trust that. Well, if I have, I'm asking questions over text, I'm like, oh, I mean, you have a little bit more freedom over there, right? And then I can point you to this larger document where it's all human would go in. So I think that, that, that's why I'm just like, arguing like the low hanging fruit is focusing on structured. And there's just so much of it. So we can be very, very busy for a long time and provide a lot of value.

• 22:25Speaker 00

  But I think where there is tremendous value, what could argue, and I would argue here, that probably even a much more higher value is if I can have a chat with my structured SQL data, because that's where your critical decision makings are happening for, for organization.

• 22:41Nathan Labenz?

  What you're describing there, I think, connects to a couple of big themes that I notice across a lot of different areas of AI today. One is just how much knowledge is kind of undocumented. Friend of mine used to call this tribal knowledge, you know, the, it's the know-how that people share by watching each other do tasks over their shoulder, perhaps, or, you know, on a screen share, perhaps more likely today. You know, or in kind of a quick chat where people ask each other questions. And that knowledge is

• 23:20Nathan Labenz?

  Because it's kind of not localized anywhere,

• 23:25Nathan Labenz?

  it is very hard to like make that available to, you know, to anyone. But in, you know, we're starting to really feel that acutely now in the, in the AI moment, where people are like, well, why isn't, you know, why aren't these agents like working? And often the answer is, you know, because they're kind of very generic and they don't have any of the context, you know, or the sort of, you know, that what, what feels familiar to you is just like totally alien to them. So it's interesting that this, this problem, I guess, has, has been felt acute, acutely enough in the data world, even before AI, that it's like, you know, you guys have built a whole platform about it. Could you kind of sketch a little bit more there? Like typical enterprise customer, I have a little bit of a sense for this.

• 24:09Nathan Labenz?

  I did a project once, I was a very junior consultant, a firm that was working with Washington Mutual, which was once the, um, the country's biggest thrift, thrift it was called. And, um, yeah, it was in that mortgage crisis moment that I was doing this project. And I do remember just the absolute total gnarliness of the, the data and just how many different columns. And, you know, there are all these columnists derived from other columns and it was kind of the data swamp situation. I only saw a corner of it in my, you know, brief engagement before the bank went or the thrift went bust.

• 24:50Nathan Labenz?

  But maybe you could give us a little bit more kind of color on like typical enterprise, where are they today? They have obviously a lot of data, but you know, are we still in a moment where it's like spread across tons of systems and people are super siloed and nobody knows where things are? In my case, I remember vividly completing an analysis and then having somebody look at it and be like, this is totally wrong. And me being like, I would have had no way of knowing this, you know? There was, I literally could not, you know, all these suffixes on these columns, like they didn't mean anything to me.

• 25:23Speaker 00

  This is exactly, yes. And here's the thing is that. The problems that we deal with today are the same problems that people have been describing ten years ago, twenty years ago, thirty years ago, and more, since the beginning of just like modern digital, just digital enterprise. Since going into warehouses and so forth, like back in the nineties or even before that. So, this is my annoyance with, with, with the entire world is like, can you imagine that the problems that we're complaining about today are literally the same problems probably complaining about thirty years ago? Like, we're kind of fucked up. If we're, we're still trying to solve the same problem. Now, things are, things are getting easier, right?

• 26:06Speaker 00

  Like now we, like the cloud has made things a little bit easier to go do, right? We have self-service to let people to go do more things, but essentially the same problem, the problems are the same. So like, let's ask ourselves like, what have we been doing for the last thirty plus years? And then I'm like, wait, but now, now AI is coming, so is this magically gonna solve this? Like, no, we got, this is where it's so important to kind of understand our history. So, let me go down a couple of examples that kind of I, I've seen across my, my careers, right? So, at the end of the day, with data, your goal is to go answer some questions. And you want to be able to answer some questions to be able to understand how the business is doing to make sure that I'm gonna make a decision that it's gonna help us make more money or save money. Right? I think you make money, you save money, you want to mitigate risk. Pretty many things you want to be able to do in an organization, right? But, so, uh, I have a question I want to go ask.

• 26:51Speaker 00

  So, I, I got the question and I need data to be able to answer this question. Where's that data? So, when one, like call it the spreadsheet approach, you basically go ask somebody who has the data. I need data about X, Y, and Z. And that person will go off and, and, and send you a spreadsheet of X, Y, and Z. And then you get in and then you, you know who excel pretty well and you have lookups or, heck, maybe you, people have an access database on their laptop. And then they'll be able to go do munching or whatever that stuff. And then they answer their question. There it is. Okay, so that's how we've always been doing things. Is that a scalable approach? Are we not? Because then tomorrow I said, you, I need, I need data, I need data about X again.

• 27:35Speaker 00

  Because I need, I need it for the new month. So they said it again and so forth, right? But I'm asking about X. I'm asking about orders. I'm asking about our latest customers like. But how did, when I asked about orders and you occurred me and you are going to go do that work? How do we know we're talking about the same thing? Do you make, do you have that same interpretation and you're actually, and you're actually delivering the correct thing that I'm expecting to go do? I don't know, right? But this is how we've been doing, this is how we've been dealing with data so for so long. Another approach is what I call the query approach. You're like, look, you're asking me for this spreadsheet of data and every month you're asking for it like. There's no, what I'm going to give, I'm going to give you access to the database directly.

• 28:17Speaker 00

  And you can just query the database. You can connect your dashboard directly to your database and here's the query that I use. That's it. Like, perfect. Now I can go ask more in real time and do things, right? But you're probably, I mean, depending how SQL savvy you are, you, you, you can go edit the queries, you can go add things. And then suddenly this query that was 15 lines, lines long. You join it with something else and then it's 30 and it's 100 and then damn, I see queries that are 10, 15 pages long. Takes 20 minutes to run and it runs and executes and that's. And the result of that query is what I'm going to use in my report presented to the CEO. That query has so much knowledge embedded into it. Do we actually know what's happening in there?

• 29:00Speaker 00

  And we're making million dollar decisions based on a supply of things here, right? And then everybody is doing this themselves, right? So if I make a change in my query, how do I know that that's not the change that you should have done too, right? Or another like, or somebody else made a change that I should have interpreted to. So that's kind of now the, the, call it the query approach. Uh, and then you can go off and say, well, no, we need to have this one standard database or data warehouse work. Where all the, all this, everything is well aligned and we're all going to go query this thing. And we go off and like, yes, let's go build the data warehouse. Uh, but what happens? That takes millions of dollars and it takes many years to go do, you get requirements. And then suddenly what happens is that you're like, oh, now you can go answer your question on this data warehouse.

• 29:44Speaker 00

  And all invested so much time and money in, you go up and you answer that. You now run the question, run your query again, answer their question. But what happens? You compare that answer to the way you've been trying to answer that question before. You think it's gonna be the same? Probably not. So then you're like, wait, I have control of this way. I get one answer and then I get the answer from this other system, which I don't control. Uh, what am I gonna trust? You basically trust your process that you control, right? So then this is kind of why you have issues like data warehouses fail and they don't fail for technical reasons. They fail for social reasons because people don't trust them and so forth.

• 30:20Speaker 00

  And then, and then the problem of that is I'm doing ETL and I gotta structure my data beforehand. And then we went off and we said, no, let's go ELT. Let's go dump our data into a lake. And then the lakes get all messy and then it turns swamps and then like I then. So you can imagine like this is the story that we've been going off over and over again. And a theme throughout all of this is keeping track of what something means. There's the meaning, there's the semantics, there's the knowledge, there's the metadata. And I think what has happened is that throughout the last thirty plus years, we've never focused on keeping track of the meaning. Uh, of, of managing the meaning.

• 30:59Speaker 00

  And now I'll, I'll argue that L L M's and all of this AI is making us realize, oh, if I want to trust that, I need to know what it means. And now what my hope and I'm starting to go see this is like, there is now a, a new focus on, we need to invest in semantics. We need to invest in meaning and knowledge. And that's where knowledge and all that stuff comes from.

• 31:24Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:28Nathan Labenz?

  One kind of extra little detour before we get into the paper and you know how, how it all works. I mean, obviously there's a lot of like data professionals these days and, and they, you know, fall into different kinds of roles. What if you could characterize, you know, in rough terms like?

• 31:46Nathan Labenz?

  Who are the people that you work with? You know, what jobs do they have? What are the activities in those jobs? How much of it? You kind of gave like a sense of like the different approaches. But if you were to actually just observe, do a time study of these folks, how much of their time are they spending on different kinds of activities? Like, how much of it is that routine query running and kind of, you know, process that they've done month after month? How much of it is data migrations? Or, you know, these kind of mega projects that may or may not ever come to fruition? How much is, you know, ad hoc analysis that, you know, where leadership or whoever has, you know, one-off questions that haven't been asked before? I have honestly, I'm probably varies a lot by context,

• 32:29Nathan Labenz?

  but I have no idea really how people are kind of spending their time. I always find that to be an interesting foundational question to then ask like, okay, well, now, well, you know, which parts of that is AI going to impact? But, you know, for starters, maybe you could just kind of characterize, what do they do? You know, what are they doing? I never know what they're doing.

• 32:48Speaker 00

  So there's, there's what I call the data producers and the data consumers. Right? So there's two types of folks in here. So the data producers are going to be more in the backend, the technical side. They're the ones who are moving the data, migrating the data, or creating, managing the data, what makes sure it exists and so forth. The other consumers, we're consumed. They're the ones who are searching for data, who want the data, who have a particular task that they want to accomplish. They have a question that they go answer. And so they, so those are the two types. So if we look at, so focus on consumers. So on the consumer side, you can have folks who are going to be your data analysts, right? Or your BI analysts. So people will build their reports and dashboards, right?

• 33:32Speaker 00

  You have your data scientists, folks who are, or you have your machine learning engineers, right? They're saying, I need to go find this data so I can go do some work to go implement. I create a model, I create a dashboard because I'm trying to go answer a question or make it a recommendation or something. So I'm consumed data to be able to accomplish that task. Now, on that side, you hear always like the eighty twenty rule, eighty percent of the time I spent cleaning my data and all that stuff. And that's a problem. And I think actually another annoyance I have is when people say, oh, I have to clean my data. It's data janitorial work. I'm like, no, that's understanding the critical meaning of your data.

• 34:11Speaker 00

  And you're just like sweeping under the rug. Like, oh, that's annoying pieces. Like, that is true. That's the, that's the essence of what the meaning of your business is. Isn't that data? You're like, oh, I just kind of write some quick code and it's just annoying to go do like. That's, that's, that's the problem is that we don't treat it as a first-class citizen. And then on the producer's side, so they're the folks who are saying, oh, we got data coming from these sources and we get the requirements. So we hear that you want to go, you need this data. So we're going to go move the data over here and now we need it to be more scalable. So we're going to the cloud. So we're going to use stuff like our data breaks. And I, I know that you guys, consumers, you want to be able to go run these types of stuff. So we're going to have the data set up in this way. So you can go do that. And but then they're, they're the ones who are actually putting the data together.

• 34:54Speaker 00

  How do we know that the requirements that they get are actually being fulfilled correctly? Especially because the requirements from the consumer side, they're connected to the business, how the business thinks. Oh, they're, I, they're talking about customers. They're, they're talking about average order values, right? They're talking about all these metrics. Are the producer, are the producers more technical side and folks? Do they understand that? Do they, is it very clear to them? Oh, I could, I could present the data this way or that way. How do I know which is the correct way? And I think that's always been a gap, that what I call the producer-consumer gap. And that producer-consumer gap is there, there's roles that are missing.

• 35:38Speaker 00

  One of those types of roles is I'm seeing this evolve now. I'm calling it the, the knowledge engineer or, or also the, the knowledge scientist. And it's really the person or role who like can work on both sides of the, both sides of those, of the aisle, right? Hey, I can be a, a, a, a non-technical people person, go understand what you're trying to go do. I can draw the models on the whiteboard. Then I can go talk to the technical folks and be, and say, okay, it's this data for this reason and so forth. So and we're starting to also see this as called the data product manager, right? You're bringing product management into data and stuff. So I think we have this gap. So there's the producers, the consumers.

• 36:20Speaker 00

  And I think one of to kind of try to try to answer your question, which I can't give you numbers because I don't know. I don't know these numbers, but I think there's a lot of just repetitiveness happening, right? We're wasting our time a lot. We don't know if this is correct. And a way that we should be able to kind of address that is by having a role, which is focused on being that bridge. And it's a role where you're bringing in one of the people, the process side. We're doing product management. It's doing this knowledge acquisition. It's things that people are doing today, but they're not doing it in a first class way. They're just like, oh, that's the second best. That's the annoying thing I have to go do. Like it's annoying to you, but it is critical. And somebody should take the ownership accountability for that.

• 37:04Nathan Labenz?

  Interesting. So that role, if I understand correctly, that would be essentially the power user of the data dot world platform. I'm thinking almost like knowledge librarian, knowledge historian.

• 37:15Speaker 00

  So that's a good point. So yeah, it's like a librarian. I would argue that for data world, you would things like data world, the data challenge. You have all three of those, right? So for example, you're bringing in all your different sources. All I need, I didn't deal with all that technical data stuff that's coming in. You want data world to say, hey, I just got this feeds of data. And I just want to know that I'm keeping track of these four, five, six hundred tables. These other thousands, I'm going to ignore them. So that's why I'm going to keep my inventory of stuff that's going on. We're moving our pipelines of data. All of that gets managed inside of data world. So you have that for your for your data engineers, right? For the new technical folks. And then you have your consumers of data who are searching for data, right? That those are the ones who will also come into data world.

• 38:01Speaker 00

  But they're looking at it from a more consumer perspective. Oh, I'm looking for data about customers and about, oh, they can now search for that. They find that data sets. And then kind of what you would want is the data, these kind of people in the middle saying, hey, I want to expose to you. I want to expose to the consumers this beautiful data product. Something that, just like your shopping experience in Amazon, right? So if you go to Amazon and you're buying a water bottle, for example. Well, there's only one water bottle, right? There's probably hundreds of water bottles. You go as a consumer, you go in, you search for something and you're like, oh, I, I, I, I, there's a rank. I have reviews, I have pictures, there's metadata, there's a lot of descriptions. And which one are you going to go trust?

• 38:44Speaker 00

  The ones that have great reviews, that has most stars, that have great pictures, that has more metadata, more descriptions. There's other ones that are like, wow, that has, that doesn't have any pictures, I'm not going to trust that, right? So that's the experience of a consumer. Now, somebody had to put all that inside of, of Amazon in this case, put inside of data world. And that's kind of on the producer's side and they want, and, and, and how to decide who goes into what. I think that's where this data product management roles are coming in. And at the end, I think a lot, you talk about the life cycle of data and stuff like we're spending money to keep this running, keep this data. It's like, you should tell me where is, what is the ROI on this? I'm like, look, we've invested all this stuff. We've created all these data sets.

• 39:28Speaker 00

  Look at all the people who are coming to consume this data. I can actually go survey the people and say, if I turn this off, what are you, what, how is this going to affect you? I, I, I should be able to know these types of stuff. And you want to be able to keep track of all these metrics and what's up of how people are using the data and how they're accessing and searching. Also inside of your catalog and data world. So I think data world is something that just connects the entire kind of landscape of there with our organization. From the technical all the way to the consumer and everything in the middle.

• 39:57Nathan Labenz?

  That's kind of another angle on one of these big themes that I keep finding in the, as, as AI impacts, you know, all sorts of different knowledge work. I have a thesis that the platforms in which people do their work are extremely well-positioned to start to kind of create long-form narratives of like what people are actually doing. And from that, you know, perhaps begin to train future AI systems on this data, this sort of data, in this case, it's like the data of how we work on data,

• 40:40Nathan Labenz?

  but in other platforms, it's the data of how we work on other things, right? It could be creative, it could be, you name it, right? Any of a million things. Are you capturing these arcs? Because I could imagine that, you know, in a catalog, you could imagine different modes of interaction where somebody maybe comes and says, oh, I have a question about what does this particular thing mean? They come, they get answered, and then they go somewhere else and execute a query. And so you only see sort of a glimpse of their work process. But I'm sure you have all these like data connectors and things too. So I wonder how often people are actually like going on a journey within the platform versus kind of flipping in and out of the platform at different parts of their journey?

• 41:17Speaker 00

  This is a great observation you're making. And so I think when you look at tools, like in general, the data catalog market, they, it is very separated from you manage all the metadata, the search and the star of find things, and then I go to another place to go like actually access to data. One of the things that makes Data World different from the entire market is that we have, we have the both of those things. You can actually search and find the data, then you can actually access the data within Data World, because we have virtualization federation. I mean, your data continues to stay where it is, right? So you got your data in Snowflake or Data Bridge, whatever, like your data stays there, but you can actually find it and access it through Data World. So that, so we actually have kind of, kind of, again, that's all I was saying, like we have a full view of everything. Now, you brought up something really interesting,

• 42:00Speaker 00

  which is something that I'm pushing people to go do, is we need to track and catalog

• 42:08Speaker 00

  Not just all this technical metadata, but it's also keep track of all the questions people are asking. Who's asking those questions? What are the and what are the answers that they're getting back? The actual, not just all, I look for this, I'm looking for this data and here's this data set, go do something with it. Because what we're gonna, what we're seeing, we're actually working on it, data at world, it's like, I want to ask a question. Uh, show me all the amount of orders that have happened in the eastern region, right? I should be able to get, oh, here's your numbers by region, by data, like you get the actual answers to your question coming from data at our world too. So those are, that's how we're the whole chatting with your data control or working on this stuff.

• 42:51Speaker 00

  Now, what happens is that we should be keeping track of the questions and answers because we should govern also. We should keep track of these questions and what is the actual kind of stamped answer, answer for some of these questions that we can't. And what happens is that if you're asking for question, you should be able to say, hey, this other group of people who have been asking very similar questions and here's the answers that have already kind of been stamped. Because these are like official things that we have to know for, I don't know, regulatory purposes, whatever. So you should know that type of information. So I think that's, that's something that we're gonna start evolving and then because we're able to go learn from tons of people are having. That's why I, I, by the way, all of that just continues to get connected inside of your big graph.

• 43:36Speaker 00

  So remember I was like, my original graph was like, you have databases and tables and columns and dashboards. Well, then I can extend that, but I have questions, I have answers, I have people. This person asked this question, this question was executed, this question using L L M to generate this query that was executed over this database. And here's the answer and, and Nathan is the steward who said, this is the official answer for this type of question. And so next time somebody asks a question like that, they can get a really, a very similar suggested question and they have a governed answer for it like. I think that's kind of where we're, where we're going, especially for in enterprise scenarios, we're gonna have accuracy and explainability. And then obviously you're gonna have questions where just like, I'm, I'm just discovering things, right? I'm trying to be creative around that stuff.

• 44:20Speaker 00

  And then I think that's kind of also, uh, when it comes to question answering, we should also think about what types of question. I think it's where, where a lot of the agents come in. And so if you think about it from a question answering perspective, it's not just give me a question and I can answer to you. It's like, okay, you're asking a question, what type of question is this? Oh, is this a factual question? Is this an objective question? All right, two different approaches, right? Is this a factual question about the knowledge? Is this a factual question about the data that I have? Do I even have the data to be able to answer this question? Or do you even have access? Authorization to be able to get answered to that? Oh, maybe not for all of it, but for this subset you can. So then like the agent should be able to go understand what you're trying to go do,

• 45:03Speaker 00

  understand your context, where you're coming from and saying, hey, I can't answer that question, but here's a subset of that question I can't answer, and I can't answer it because of the other reasons. I think that's the type of stuff that we're heading toward. That's what I get really, really excited about AI right now.

• 45:17Nathan Labenz?

  Yeah, accuracy and explainability, those are two. It's funny how many, I mean, it's not surprising in the sense that, you know, obviously, AI systems, AI models are all trained on data and they're very, you know, they're very much like directly derived from their data. Accuracy and explainability come up in, you know, all the AI systems work that I see. It's interesting that that exact same, you know, framing, you know, clearly predates it in the, in the data management work that kind of underlies now, you know, the AI that's getting built on top. When it comes to accuracy, do we have any good like benchmarks, rules of thumb, reference points for how accurate humans are?

• 46:02Nathan Labenz?

  This is one thing I find everywhere I go, you know, that people have no idea how accurate the humans are.

• 46:08Speaker 00

  All right, I love that you're asking this question. And this is really the, how we, how, how we need to have fierce conversations with each other and push back. So like all the research that we've been doing about, oh, the accuracy and so the, the first set of critics, what do they say? Oh, but 60% is inaccurate enough. Yeah, of course, I know it's inaccurate. So what is accurate enough? 100%? 99%? What is it? Do you actually know? Okay, you, you got data today to go answer a question that you made, which is a bar, a decision that there be some of them bars. Do you know how accurate that was? Are you gonna be your life on the accuracy of that?

• 46:47Speaker 00

  Go ask that question. I don't think people, I don't know. Do you know, like you, you got that, you got that dashboard or that spreadsheet from, from Alice and Bob? Do you know how that work was done? You trust exactly? Really? Do you do, do? People aren't asking those questions, right? So they're like, they make the assumption that the machine needs to be accurate and everything, but then they don't even ask, they're not being critical about their own process to go do that. I think this is, this is the opportunity right now to be very critical about kind of how to evaluate ourselves, like how are we doing this? And going back to like how, how AI is impacting all the tasks that we're, that we're doing. It's like, let's go figure out how we're solving that problem today.

• 47:29Speaker 00

  And say, well, wow, this is a very, uh, uh, I mean, what is that process and can we improve that process? And we figure out, oh, wow, there is a big bottleneck or that's something that is pretty sketchy. It's not working well. And you want to be able to bring in the machines, the AI to go help for those things. So I think it's really, really important to ask ourselves, how are we solving these problems today? And be honest about it. It's like, oh crap, that got it, but that's something that may not be doing. I mean, I've done this. I, I remember once working with a customer who were like years. Getting, getting this report and making decisions about it. And I went off and go analyzed how they got that report since they hadn't done it.

• 48:09Speaker 00

  And I'm like, well, do you know that to get this value over here, you've, there's some hardcoded values or like multiplying 0.02 or whatever. Like that's happening and they're like, well, I said, and then somebody said, oh, I remember like two years ago, we were like doing some special discount. And like, well, guess what? The person who wrote that query hardcoded that discount, whatever. And it's been there for the last two years. So all the decisions you've been making is, has that embedded in there? All your numbers have been wrong to the last two years and you've been made decisions. Now, but the honest O B S here is like, what the company still may embody and stuff like, how big of a deal is that? So I think that's also it's like, how critical are these things? And like, ah, sometimes they're like, oh, it's good enough, right?

• 48:54Speaker 00

  So I think it's also part of the culture that you have within your organization.

• 48:58Nathan Labenz?

  I've experienced, you know, some very similar things and in a non, you know, at the company that I started, we did a lot of digital marketing for a while. And, you know, we, and we both like helped our clients do it and then we did it. And we found just so many instances over time where you come to some initial conclusion and, you know, you really have to challenge yourself, especially digital marketing will punish you, you know, for being wrong on these things. You know, we really had to challenge ourselves, is this really right? You know, does it, does it check out across any, everything else we know? You know, does it, does it violate our world model?

• 49:43Nathan Labenz?

  And, you know, dig in, dig in, dig in as many layers as we could to finally often find that there was some problem. That happened at, at the Washington Mutual thing too. I remember one time where, you know, I was summing something and it was like greater than, you know, the limit that it was supposed to be governed by. And it was like, well, what's happening here? Your, your query must be wrong. And it turned out actually, no, like some other process in the bank had changed some underlying, you know, data and all of a sudden, you know, everything had kind of, the sands had shifted under us. But yeah, broadly, I find that, I don't know. I mean, if I had to guess, I would estimate that like a significant fraction of the

• 50:29Nathan Labenz?

  data analysis that people are basing business decisions on is just like fundamentally flawed to the point where, you know, they're, they're kind of detached from reality. I don't know if we could get any more, you know, rigorous in our estimation of that. But for me, it has been a recurring theme that just, you know, somebody gives you the results of a, a query or a spreadsheet or whatever and tells you it's X and it's like, I do not take that stuff at face value anymore. We were just talking a little bit too about podcast metrics, you know, oh my God, like we had one, we were looking at the the success of the podcast. And it was like, oh, you know, we did really well in this time and we did really poorly in in this

• 51:11Nathan Labenz?

  time, you know, by comparison, like, look, there's a spike and there's a trough. And it turned out that it was like, the way that the month fell with respect to the weeks and the days on which we published the podcast and you know, it was like, well, we actually had nine episodes released in this month and seven in the next month. And that's why, you know, we got way more downloads in the in the first month than the next month. It really had nothing that you the per download, the per episode downloads were the same. But even at that simple of a level, I've seen, you know, people just get like so confused by kind of aggregate measures. Do you have any, I mean, this is a little bit of a digression for our core topic, but do you have any tips or sort of habits of mind that you encourage people to mantras, maybe even that you encourage people

• 51:54Nathan Labenz?

  to adopt? Like, I always say, there's no substitute for reading the raw data. If you have not looked at actual raw records at some point in the process, obviously, can't read all the raw records. But if you have not familiarized yourself with at least some raw records, my guess is, there's a pretty good chance you're going to be wrong in the aggregate analysis. Any things like that that you preach?

• 52:16Speaker 00

  I mean, ask the why, right? Ask the why five times, right? What is the goal? What is the objective? What is the objective above the objective that you're trying to go do? Because it goes back into like, oh, we have all these ad hoc questions that we're trying to go deal with. I'm like, why are you doing that? How is that valuable for the business? Like the first, you give that to that person, what are they going to go do with that? And so for me, it's always like, are you tying the work that you're doing to specific business value? And you, and the way to tie that is that you need to know what are the objectives, corporate strategies of the quarter of the year, whatever, that needs, like, you know what those are, because you need to be able to push back saying, I'm not going to go spend time on that, because

• 53:01Speaker 00

  that is not related at all to what our quarterly goals are, unless you explain it to me. Right? So I think that's something how, so that's kind of more from where I am going to go and top, right? Get connected into the business to make sure that we're not wasting time to go do Then another kind of another approach technique that I do is what I call the the iron an iron thread. Which is like, look, to to how do I know if something is working correctly? Like, don't boil the ocean. That's another mantra. Don't boil the ocean. We love to get it. So start from something very specific, one small thing, and figure out what is the path all the way to the to the end goal? And do it really, really small. So you have to do it just you basically get that one thread through.

• 53:45Speaker 00

  And then you're like, okay, I understand all the things I need to go through. And then you say, let's go do it again. And then you add like, basically, you add another thread to it. Yeah, then that thread gets bigger, bigger. And then at some point, then you're like, well, how about let's do it for another area, something something different. And then you so you can have somebody else in a distributed way. Uh, independently go off and do another thread somewhere else. Kind of okay. Now we start getting these kind of threads all all straight. I think that's kind of an approach I always do because otherwise we kind of like get focused too much on the bottom stuff. And then we're not connecting it to the value that people are seeing because we don't even like, I need to understand what is the the the not just the output. But what is the outcome that needs to happen? Uh, and so you want to be able to kind of just be able to

• 54:30Speaker 00

  drive all that through from top to bottom, from from the executive, uh, all the way down to to to to get into the nitty gritty of all of your sleep. So all that needs to be connected. And you don't have to boil the ocean to it for everything. Just do it for one very specific thing. Go through it. That's then you understand how things flow and what's working, what's not working, where's the struggle and so forth.

• 54:51Nathan Labenz?

  Yeah, interesting. That's software development sometimes can be like that. More generally, too, where you just it's a vertical spike through all the layers of the stack. To get everything connected and, um, you know, make make one successful round trip is like a good, uh, early milestone in a lot of things.

• 55:08Speaker 00

  It ties us to AI. It's like, well, that's like all the steps you need to go do it. Then you figure out, well, where can I put AI here to automate things, make it faster, better, more productive?

• 55:19Nathan Labenz?

  So perfect transition then to the paper. We've got a lot of preamble before the actual paper that you've published, but high level, it seems like the big demonstration of the paper is that you take GPT four and you say, hey, I want you to write some queries for me to answer some questions. And You give it either the pure schema, you know, like the table definitions and you know, there's a lot of meaning often implicit there in the or you know, to some degree explicit in like the names of the tables and the names of the columns and the way that you know, database keys and you know, whatnot relate to each other, right? You can sort of, there's

• 56:01Nathan Labenz?

  an implicit graph, again, somewhat, somewhat explicit just in the schema structure and you get a certain level of performance. And then you compare that to, okay, now let's also give GPT-4 a higher level semantic knowledge graph representation of what does this data actually mean? So that perhaps not surprisingly, given all this preamble, yields quite a bit better results. What additional details about that experimental setup or about the findings do you think are most important for people to understand?

• 56:38Speaker 00

  Everybody was excited about chatting with their data. Chatting also called, call it a text to SQL has been an area in computer science and been looking into for, again, also probably 30 plus years. So there's been all these techniques happening. And now with LLMs, people are like, well, this should be much more easier to go create, to do text to SQL. And from an academic standpoint, there's all, there's these benchmarks and techniques people have been doing out and they show 95% accuracy. And I'm like, I'm skeptical about this stuff. And when you go, when you look into these benchmarks, they're all very simple. They're like, oh, there's a couple of tables and there are, and the semantics are very explicit in there. But I'm like, that's really disconnected from the enterprise.

• 57:23Speaker 00

  Because the enterprise doesn't always have that clear and it very clear and small. And then the other part is that the questions people were asking were like, oh, here's like a laundry list of questions that are like, how, why these questions? How do you come up with these? So the main research question, there's two questions. One is, we want to understand to what extent can these large language models kind of generate SQL queries that could be accurate? To what extent that can, that can be done? And then the follow up question there is, to what extent can that accuracy improve? If you actually put the knowledge graph in the middle? And the hypothesis here is that, hey, well, if you do the knowledge graph, it's going to improve, improve the accuracy. Now, we didn't know to what extent, we didn't know how much.

• 58:08Speaker 00

  The thing is that just common, just talking to folks, everybody's like, yeah, the adding semantics, context, knowledge of like that, that should help. But when, how would we do it and how much better is it? And like, people didn't know. And I'm like, we were at Snowflake Summit in June and we were having a stuff reception with a bunch of product people. And they were the ones who frankly were just challenging us, saying, you guys should just do a little benchmark around it. And I'm like, yeah, and literally got on the plane, came back and I started working on this stuff. That's, I mean, that, that's a kudos to them, kind of really challenging, pushing us to it. So that was the whole setup. So what we did was that, um, we, we have an enterprise schema using, uh, uh, in the insurance domain. Uh, so we're using an open standard called from OMG, they have a property catching schema.

• 58:51Speaker 00

  So this is a representative of an enterprise domain, an enterprise schema right there. And then the other thing was the questions we're asking. So you have an enterprise schema, and we have a set of questions, but these questions, we're actually putting them, we created this quadrant about with two kind of spectrums of complexity. One of the spectrum of complexity is on the types of, uh, easy questions or harder questions, basically. Easy questions being, give me a list of things, show me all the claims that are open, right? Those are easy questions. Harder questions are things that are going to be more about, uh, strategy. And which involves, uh, uh, really questions already, I need to do some aggregations, I need to do maths around that stuff.

• 59:33Speaker 00

  And then from a, from a technical perspective, you also have easier questions and harder questions. Easier meaning, I only need a couple of tables to answer this. And a harder question is, I need a lot more tables, eight, nine tables to need to be joined. So then you put these two spectrums together, you get this quadrant of like easy questions over easy schema. And then you get harder questions over easy schema and so forth. You get all these different quadrants. So, so that, so that, that gave us a perspective of understanding the types of questions, uh, that people can ask. And then the third thing that we looked into was let's make, let's invest in putting, in coming up with the semantics. Invest in creating the, the context, that's what we call the context layer. So it's,

• 1:00:16Speaker 00

  here is the ontology semantic layer and here's how these things map. So, very simple. Uh, there is a claim, has a claim number. Oh, there is a table called claim. Okay, perfect, that matches. Uh, but there's actually, there's, that table has, I don't know, fifteen, twenty columns. One column is called claim identifier. That is not the claim number. There is a column, there's a column called the company claim number. That is the column that has the claim number, right? So that's, so then I make these mapping, and then you would have things like values, you'd like, well, I, I want to know where, what are my, uh, the policy holders. Well, you have to, it's everything that has the role column equals PH.

• 1:01:00Speaker 00

  Agents, oh, we're role equals AG, right? That's the semantics like that, that's, that's, that's the context that you have in there. So, we made that very explicit. So, that's what we, that's what we come up with, uh, kind of, as, as a, as a inputs, kind of to the benchmark. And what we did was evaluate saying, very simple setup. On purpose, wanted to do very simple because I wanted to like get the, the bare minimum. A very, the simplest prompt that you can imagine. Here is a SQL DDL schema. Copy and paste the SQL DDL and you say, write a SQL query for the following question. And then for the knowledge graph, we said, here is the, our ontology, the ontology, that's a semantic layer, which is an open standard that we gave it, right?

• 1:01:44Speaker 00

  We understood that GPT four knows these open standards. And you copy paste and you said, write this Sparkle query for, for this question. And Sparkle is the open standard for, for knowledge graphs, for, for the queries. And we just, so also the caveat is that we put the schemas and the ontology, the semantic layer as part of the prompt. Right? So it's, so it has to fit within the, the, the amount of tokens. Uh, and you just give that to GPT four and it generated the query. And then when we started comparing all the queries that were generated, I think that, we get that three X different, differentiators. So there's three times more accurate queries if you're doing the knowledge graph versus the only do SQL. That's if you compare all the questions. And if you look them inside of the quadrants.

• 1:02:29Speaker 00

  Effectively, when you had questions that require high complex, the tables, basically, questions that require joining more than four tables, SQL query that was generated by D B D four would always fail. And I think that was an interesting point. It's like, oh, so if you have queries that require more than four tables, like that's not going to work very well. So, and that was for this, for this very simple prompt, and I think up work that we're doing right now is, uh, well, we're going to go test it with other models too. And I think what we want, what the community to go do too, and we're seeing this is like, well, yeah, let's go improve the prompting. Let's go figure out more context. Is there a way I can pass the context to SQL in some other ways? Like, this is where we need to start working as a community to go figure out. We presented an initial baseline. So

• 1:03:13Speaker 00

  going back to our original research question is to what extent? We found an extent. We know this. Uh, and now, uh, this is how we advance science is like, let's get other people build on, build on that work and let's see if we can improve the extent.

• 1:03:26Nathan Labenz?

  It definitely is, again, another major theme, just how many of the benchmarks that largely have come out of academia over the last, you know, 10 years have just been kind of blown away by the latest models. I think that's one of the really interesting indicators of just not only how fast things are moving, but how fast they're moving relative to people's expectations is that, you know, you see like 2019, 2020, even some 2021 benchmarks that are just like obsolete because they're, you know, they're totally broken by the, the latest language models. And they were built in the way that they were because, you know, that people didn't even, you know, couldn't even really conceive of at the time that an

• 1:04:09Nathan Labenz?

  AI would get good enough to like solve all this. It was like, well, gee, if we, if it solved all this, we'd have AGI. Turns out there's a little bit more room between solving these benchmarks and AGI at least. So now you've got this kind of first stake in the ground, basically, right? Here's, okay, all these old, simple schema benchmarks are increasingly obsolete. Now we gotta increase the challenge. Let's bring a real like enterprise grade challenging problem to the, to the models here and see how they do. They don't do so well. Here's one initial technique of layering on semantics that makes a huge difference. And of course, we're fully expecting that people are going to, you know, continue to

• 1:04:51Nathan Labenz?

  improve on those techniques and get better and better performance. Do you have any tips for, you know, how people, and especially for enterprises that want to like do this for themselves, for actual like practical purposes internally? What sort of guidance would you give them on constructing their own internal benchmarks or, you know, eval suites?

• 1:05:14Speaker 00

  The contribution of our work, one is the, is not just the results of the benchmark, but it's the benchmark framework itself. So what I'm telling everybody and actually working with our, our own customers doing this too, because they're like, we're working together to build these chats and these agent systems and stuff. And they want to evaluate how, how good this is. So one, they have their data, right? So they got that first check. Number two is on the questions. So I think this is really important to understand what are the questions people are asking? And goes back to the one we were talking earlier before, it's like you should, we should catalog and keep track of the questions, right?

• 1:05:56Speaker 00

  So what are the questions people are asking today? Who's asking them? Why are they asking them, right? And how are they solving those, those answers? How are they getting those answers today? Are they doing those answers today through a spreadsheet that somebody gives it to them? Or do they write a query that they get through a dashboard? Like, let's figure that out today. So then what happens is that get those questions, let's then put them into those quadrants, right? It's just an easy question, it's just a harder question, or this is a question that is just a list of things versus this is the question that actually involves aggregating all that stuff. And this is a question that requires small amount of tables or large amount. So start putting that in into this quadrant. So that is my main recommendation for folks.

• 1:06:41Speaker 00

  And then in order to avoid boiling the ocean, make sure that you're focusing the questions that are actually going to provide some value, right? And that's why it's important, who's asking those questions and why they're asking those questions? So I think that's, that's a very important thing. And then the, the third one is the context. And I think the whole point of the work that we, that we were presenting is that you should invest in the context. You need to, you need to invest in the semantics of what the stuff means because frankly, that, that's what the L Ls don't have. So you need to start investing in that. So that's where people, that's where at least our customers, they're already investing in the context because they're already have data that we also.

• 1:07:20Speaker 00

  But for other customers who are like thinking about this, I'm like, well, if you want to have a chat with your data on your, on your relational SQL databases, and if you don't have a catalog, you've got to solve that. You've got to go build, bring in a catalog. But the catalog is for the sake of understanding, keeping track of all that metadata to start building that semantic layer and everything. And then don't boil the ocean. That's where I would say then, then have that iron thread approach and tie it. Go bring in the data, you need to go answer those questions and then keep track of all that metadata. Treat metadata, treat the semantics as a first-class citizen. So again, to summarize, one, you already have your data. So great, check. Second, make a list of all the questions, put them into quadrants and figure out and prioritize them by the why behind them.

• 1:08:03Speaker 00

  And then understand how they're answering those questions today. And third, if you don't have a catalog, then uh, that's where you can start from, from that technology perspective.

• 1:08:13Nathan Labenz?

  So what do you think the prospects are for this kind of chat with your data paradigm, right? The dream would be like. We, the AIs get really good and the decision makers, you know, whether it's the middle of the night when they have their brainstorm or, you know, whenever they can, I mean, AI has a lot of advantages, right? It's like always instantly available. It, you know, you can kind of pick up where you left off. I also recite these. Here are the big challenges, presumably accuracy. It seems like we're probably not there yet. You know, you said kind of 60%. It's not good enough. I would assume that, yeah, probably not. Do you have a sense for like, what is good enough and sort of what the steps are likely to be to get there?

• 1:08:58Speaker 00

  For the, like in the benchmark, it was like 60% for all the questions, right? Now, what we really, what we're really gonna start thinking about is, given a question, I need to analyze, can I even answer this question? And I, so I think the first, the thing, this is what we're working on right now is saying, hey, give me questions. I know that I can answer this question. I know, or give some confidence, maybe that if I can answer a question or not. So then, what you're gonna do is reject the questions that you know you can't answer, and then you're gonna focus on answering the question that you know you can't answer. And I highly suspect, and this is stuff our research has shown us, that from there, we can get to the 90% of that stuff. So I want, so if I, if I, if I zoom out and I imagine, imagine the user going into

• 1:09:43Speaker 00

  your chat, you ask a question, you're, oh, you'll either get an answer with an explanation, and I'll dive into it in a second, or an explanation saying, I'm not going to answer this question for the following reason. That already gives you confidence around that. Now, when it goes into like, when I do know that I can go answer a question, what happens is, this is where the knowledge graphs come in, it's tied to accuracy and explainability. I mean, if I ask a question, I can extract, using the LLMs, extract the concepts, all the things that are being asked to that question, and I can look them up. In the graph. And if all the things that I'm asking show up in the graph. It's open in the knowledge of I'm like, oh, I know about all these things.

• 1:10:28Speaker 00

  I can then formulate a query. But if I'm asking for a question and I'm like, oh, I don't, I can't find this in the graph. I'm like, I don't know. So basically you know what you know, and you know what you don't know. So then that's how I can explain that I don't know this stuff. I'm not going to go answer the question. And then if I do know, I can create the question and then I can answer it, execute it. And here's a, here's a big distinction is that. L L M's are not answering the question. Because if the L M is answering a question, you're training L M, then it's always going to be probabilistic. Like it can't be for certain, you know, I can't have certainty behind it. L L M's are going to be an assistant to create the code.

• 1:11:12Speaker 00

  That is going to be executed over the system that's answering the question. And when it's code, that is something I can reason upon. That is something I can govern. That is something I can manage and something I can use for an explanation. So if I match it to the graph, I'm like, okay, here it is. I generate this query. This is a query over the graph and I can, and then this is where all the, the, the technical metadata catalog comes in saying, oh, this exists in the graph for this reason. It comes from this part. It goes to this table and this person was involved in the creation of this. It was authorized by this. That's why you're getting that information. So I can then not only give you the answer, I can give you an explanation and I can go as deep and granular and nitty gritty and technical that I want. Uh, or I could give it a high level.

• 1:11:55Speaker 00

  So again, also one of the things that we're working on and is to, another kind of cat and a little bit of an annoyance I have is when we talk about explanations. Explanation is to whom, right? You give a non-technical person an explanation, they probably just want an explanation in English and also know who they should go talk to if they have a question. Don't give them code, they're not gonna understand it. If you're giving an explanation to a technical person, they probably want to see the code, right? If you give them just a high level of fluffy, they're like, that's, they're called BS on that, right? So you also need to understand who are the personas you're giving explanations to. Again, the graph is just basically, I can traverse the graph as much as I can to get as deep as the level of granular that I need.

• 1:12:39Speaker 00

  So I think that's why knowledge graphs are critical for the accuracy and for the explainability.

• 1:12:44Nathan Labenz?

  So I'm kind of mapping, you know, a bunch of familiar AI application building techniques onto this problem. And one is decomposition. It sounds like, you know, you could imagine just taking the semantic layer and the schema and dumping that all into context and saying, hey, have at it. Or you could imagine kind of bringing that up and saying, okay, first thing, your first job is to just look at the semantic layer and determine if you have the information to answer the question. Then you can have a peek at the schema and actually generate the query. That decomposition, you know, takes more work. It definitely kind of makes things a little bit more brittle, depending on, you know,

• 1:13:26Nathan Labenz?

  how quick your system is to change or whatever, but it really can drive accuracy. I imagine that's a part of it. Fine tuning models, obviously also a big trend. GPT-4 fine tuning is like in, you know, limited release. I don't know if you've had a chance to fine tune that or if you've had experience fine tuning other models to get better at this, but that seems like another opportunity for improvement. And then you mentioned earlier too, like established questions and kind of canonical ways to answer them. I think they're of like these sort of skilled database type paradigms like the Eureka paper or the Voyager paper out of Nvidia where, you know, when they, when they find something that works, it's like, okay, cache that basically to a database.

• 1:14:09Nathan Labenz?

  And next time we get a similar question, you know, or a similar challenge. And, you know, in their case, it was like Minecraft type of skill building, but you know, here it's obviously database type of work.

• 1:14:21Nathan Labenz?

  Well, let's see if we've done this before, you know, and have kind of an established answer. I imagine all three of those are kind of directions you are pushing on.

• 1:14:28Speaker 00

  Yeah, I get it. The answer is always, uh, it's hybrid and it depends, right? So, uh, yeah, for sure. So I think from the science perspective, I start with one thing to go see how well that stuff is. And this one is like, let's start with the more bearable, bearable of prompt engineering to go do things. And then you have other tools that you want to kind of figure out, like, uh, uh, what, what can I use for vector databases that I can, as another tool there. It's like, also note that for the first experiment, like everything fit in the context window, right? So maybe I have my apology, my submitter is going to be so big that I can't always pass it in here. So I need to be able to kind of get parts of it out.

• 1:15:00Nathan Labenz?

  Were you working with the 8K version in this work or the 32?

• 1:15:04Speaker 00

  The 8K, yeah, because I mean, the, the, the volume wasn't that big anyways. And then when it comes to fine tuning, I think there's, there's two, two things to look into. One is, are you fine tuning with the data or fine tuning with the metadata? And I will think like fine tuning with the data is probably not going to be valuable because first of all, you got millions and billions of rows in my Stoke Lake or Danbury, my data lake house. Are you really going to extract that and put that in there and it's so expensive and then that, which gets updated all the time like that or that? And then, by the way, then in that case, your L M is trying to go answer the question, but then it's going to be probabilistic and deterministic. And you like, if you're going to go ask something, it's going to get your balance for your, your, your bank account.

• 1:15:48Speaker 00

  Better be fricking accurate all the time, right? So like, I don't, so there, there are particular scenarios where you, I mean, the accuracy is critical, critical. So, but when it comes to like fine tuning on the metadata, I think that's, that's going to be like huge opportunities. And as you said, it's like, oh, here are these patterns of questions. You're these types of questions and it gets these patterns of queries. Like that's the stuff where you want to go, go, go fine tune on that metadata there because then it should have like, I know you're going to generate the right, increase the possibility to generate the right query. Now, other thing is, when you generate a query, you're generating a code, and I can reason upon that, and I can do that in a deterministic manner.

• 1:16:26Speaker 00

  Right? Now I can actually check, I can do static analysis, and I can check if this is the correct, if the code is going to be correct or not, I mean, with a bunch of heuristics. So then I think there's another set of kind of techniques that you can go use, it's a post-processing. So it's prompt engineering, or vector databases, you're going to do fine-tuning on the metadata, not on the data, the metadata. You'll do some post-processing. So these are all basically just a bunch of tools that you'll have. And what we'll see is like, our agent framework, an agent framework, that will, for each kind of state in that agent, will use this tool, or this data will use this tool, or this has a lot of tools at its disposal.

• 1:17:08Speaker 00

  And I'd argue that a lot of the IP that will come out is on how people are structuring their agent frameworks. And how, at the end of the day, you want to have an intelligent agent who can really make, understand its environment, perceive its environment, make decisions autonomously based on their environment. And that's really going to be how sophisticated your agent framework, your state model is going to be. The current kind of RAG architecture that we all talk about, I think that's the simplest, most naive type of agent system, like,

• 1:17:45Speaker 00

  you get a question, you know, you said something, you get, you set up the vector database, which then get, add, get context back, and you send it all in, like, that's kind of one-stop shop for everything. That's going to get it broken down into so many different pieces. And that's how we're going to start developing kind of these agents. Which, by the way, this agent stuff is, another thing I'll call out there is, we've been working on agents, this is the good old-fashioned AI. People are working on this stuff for like fifty-plus years. There's so much work on agents, on planning. So hopefully we don't, we'll spend time reinventing the wheel and build on the shoulders of giants and can advance very quickly here.

• 1:18:21Nathan Labenz?

  Yeah, one thing, a tip that I've found, and that I have a project I want to ask for your advice on, is when fine-tuning, this is definitely proven true for GPT-3.5. Unclear, you know, everything has to be kind of revalidated when you move up an order of magnitude in scale. So I'm not sure yet how this will apply to GPT-4 fine-tuning, but for 3.5 at least. Fine tuning on chain of thought type data has proven extremely valuable for me in terms of getting the fine tuned model to behave how I want it to behave. It has not been good at learning facts that way. So certainly wouldn't fine tune on like raw data.

• 1:19:06Nathan Labenz?

  Even fine tuning on like the definitions of tables and stuff. I wouldn't expect that to work very well either. You're still gonna, you might do it, but you're still gonna need it in context to be like accurate from my experience. But where I've seen the most boost is in kind of explaining the way in which we want you to go about going through this problem. And having a fine tune, and it doesn't have to be huge. Again, in my experience, as few as like 100 examples can get you a huge boost at the 3.5 level, but it's just really critical to have that kind of explicit step by step that the model, because obviously we know that like chain of thought, you know, think step by step is really helpful.

• 1:19:49Nathan Labenz?

  But you have to kind of demonstrate what kind of, you know, chain of thought do you want? Or do you need, you know, to solve the problems in the particular context? So here's my challenge, and this is, this is really why I kind of got down this rabbit hole of knowledge graphs a little bit in the first place. So I'm, we're going with this company Athena. We're in the executive assistant space and we are trying to deploy AI in a million, you know, different ways to enhance our service. One of the things that we know that our clients would really like is if we had some way for the EAs to have like always on access to everything about them, right? All their preferences, all their history, whatever, anything in the background, right?

• 1:20:31Nathan Labenz?

  A lot of times right now, the EAs have to ask questions and it'd be better if we could somehow, you know, answer those questions without them having to ask. So the idea is like, can we create a long lived client profile? That we can sort of maintain and update over time. A huge challenge, though, is almost all the data is unstructured. We like interview clients when they come on board and, you know, record that call and transcribe that call and then summarize that call, but it's all text. The whole thing is text. There's no schema whatsoever. So we can then throw that stuff into it in a vector database and query against it and get, you know, kind of what did they answer at the time of the onboarding call? But then, of course, you know, things change, right?

• 1:21:15Nathan Labenz?

  So how do we add more to that database over time? And how do we sort of manage like what supersedes what? And I wonder, you know, this is pretty early stuff. I think we're like, you know, I don't know if we're on the absolute edge of tackling these problems, but there's not too much work out there on this kind of thing yet. But I wonder if you have any kind of thoughts or intuitions for us as we think about something where they're, you know, the core challenge here is there is no schema. It's all just text and we're trying to sort of be able to like dump new stuff in there, but then have it maintained in a way where we get the right information, you know, at any given point in time. And we don't have great solutions for this yet. So any pointers or suggestions would be valuable.

• 1:21:59Speaker 00

  I will argue that there's always a schema. There's always an ontology, there's always semantics, and it's implicit because you're talking about the same concepts over and over again. You'll maybe using different words that mean the same thing. So an idea here is like, take all those, all that text that you have, all that unstructured stuff and just ask GPT to say, generate an ontology or taxonomy or business glossary from these things, right? And I start, and then you start identifying what these concepts are. And I think for your particular space, right? Like, think about the domain, model the domain, like what do people do? Like, what are the, what are the roles involved? What are the tasks? Like, those stuff are probably, you probably have really generic ones that are going to

• 1:22:44Speaker 00

  go, they're going to be, they're going to be generic across all types of companies. And then I'm sure there are going to be things that are very specific, but then kind of figure out what are the high level ones that you, that are generic? It's there. I mean, you said there's no schema. There is a schema, right? There's a meaning behind everything we're talking about. So just to kind of automate it, just go get all that text and just say literally the prompt, and I do this all the time, like generate a, generate a business glossary or generate a semantic layer, generate an ontology based on the following. Texts or questions, it's gonna come up with a bunch of stuff and then you're like, oh yeah, that, that, that, that thing is related to this thing and that. And then now you have like a consistent, controlled vocabulary way and that, that's something that you, that, that you can use later on for,

• 1:23:26Speaker 00

  in your chain of thought, right? Or, or just even how you're presenting things in the user interface, or how people are just chatting or conversing with it. I think, in a way,

• 1:23:38Speaker 00

  it is anecdotal, but I kind of, when we set up these, when we created these semantic layers ontologies, what we, what I've seen is that people start changing their behavior a little bit and using the words that was, that were defined inside of that ontology, inside of that semantic layer. And then that's a human behavioral thing. It's like, oh, well, if I use these words, I actually get my answers, right? So then they start changing. Then actually, as an organization, we all start using a common vocabulary. We have that lingua franca. And it's like, at the end of the day, like, yeah, we should, we should, it, it would be great if we started to agree on things. And I think this is also a way to kind of get that, have that human behavior. So anyways,

• 1:24:20Speaker 00

  long story short, there is a schema in there. Use GPT to help you extract what that schema can look like based on all the texts that you have. And then, then reuse that and just any other techniques you're doing.

• 1:24:31Nathan Labenz?

  Yeah, interesting. So would you recommend, right now, we are kind of just chunking and dumping stuff into vector database and kind of querying against that. We're using the high technique, the hypothetical document embeddings, which is, you know, basically the translation when a user comes to our chat and says, you know, what is X about the client? We translate that to something that we think is like what the answer is likely to look like. And that, that's the hypothetical document embeddings, you're sure you're familiar. And then that seems to improve our, the accuracy of our retrieval. But we're, we're purely using this like vector database. We're not really using much structured data. Do you have a recommendation for like, what,

• 1:25:15Nathan Labenz?

  what database would you recommend? Should we go with like a Postgres with their vector thing? Or should we be thinking like graph database? Because we do have a lot of.

• 1:25:26Nathan Labenz?

  I think a lot about that AI town paper. I don't know if you saw that one, but it was the, you know, it made the rounds. It was like these little AI agents were running around and like making plans and interacting with each other. For me, one of the most interesting aspects of that paper was the way they handled memory was through a periodic sweep of all these raw observational memories into these kind of higher level synthetic like thematic, you know, or kind of more episodic memories. They would, you know, the agents would write down, I talked to one at this time, we said this and this. But then, you know, there's too much there, you know, pretty quickly for the, you know, especially with limited context, that's of course, you know, expanding, but to

• 1:26:08Nathan Labenz?

  manage that, they had to kind of synthesize it into this like summarized layer. And I thought that was really interesting. We haven't implemented anything quite like that. That would seem like a very natural thing, perhaps for a graph database of like, this was like synthesized from this or, you know, these, if I, if I'm looking at some like abstracted summarized memory or, you know, description, like what are the raw observations that that came from? Where do you think I should put that data?

• 1:26:36Speaker 00

  I think this is going to be a combination of using whatever vector database, vector similar indices, wherever, and then I think some sort of graphs. I think this is, my perception is that that's where things are going to head. And then, I mean, at the end, just use whatever graph database you want and things can change. I mean, I'm not a big from, like, use this database or whatever. You're going to have virtual knowledge graphs for things where you have all your, your, your, your enterprise data. Like, you're not going to move all your data from Snowflake or Databricks into a graph database and keep copies of that. No way. That's not going to happen. You're going to have a virtual, a virtualization layer. So it's going to, you can query as a graph and it

• 1:27:18Speaker 00

  gets translated to SQL for that. I mean, that's kind of what we, that's exactly what we did for our benchmark work. So that's something if you already have existing data in a relational database or your data lake, you're going to have a virtualization or a knowledge referencing relation layer. But if you're starting from scratch, then I think you're going to have a combination to Guess what's gonna happen? Every single structured database is going to have vector features. They didn't have it yesterday, had it today, it's gonna happen today, it's gonna happen tomorrow. So, all graph databases are having this right now. So much, all the SQL databases are gonna do that today. Are vector databases gonna have SQL interfaces and graph interfaces tomorrow? Yeah, I've been explicit about this. Vector databases, that's a feature, not a category.

• 1:28:03Speaker 00

  There'll only be one winner. Whatever vector database is, there's only one winner. Everything else, it's just gonna be, I'm just gonna use this other database and kind of have a vector feature and it's gonna be enough. That's what's gonna happen in the next year or so. And you will use that, you will, you will need to use the vector database if you have a real scale issue, like a scale need. Otherwise, all databases will be adding vector features, so.

• 1:28:27Nathan Labenz?

  Well, folks can juxtapose that against Anton's answer in an earlier episode. His big thing, which I did think was compelling, he basically said, the data that comes into Chroma has never been in a database before. And he was basically like, just the overall growth of how much data is going to go into databases is headed for like 10x because of just all the structured, unstructured stuff moving into these, these new structures. And so, like, everyone can win, you know, because the whole market is growing 10x.

• 1:28:59Speaker 00

  Well, my whole point is that you look at the Snowflakes and the Databricks and the Azure's of the world, who have already all the SQL stuff. They're all gonna say, well, we should be able to go support more unstructured, and they will. And they will do that. Now, these vector databases, they're like, oh, we do that too. But they're like, oh, but you don't offer SQL, so what, they're gonna now reinvent SQL, add SQL to that stuff?

• 1:29:20Nathan Labenz?

  No, they're not.

• 1:29:21Speaker 00

  And then, it's, it's, now for scenarios where you need best scalability and the raw and stuff for vectors, then you will want that, but that market is not gonna be that big. It's gonna be, there's gonna be one, one big player, and everybody else is gonna be behind. I mean, it's like MongoDB, right? MongoDB is the winner, and there are other kind of NoSQL databases. They're all small behind, whereas one winner there, it's MongoDB, period, right? They'll look at their stock, and so, yeah, I think that's what's happening.

• 1:29:52Nathan Labenz?

  So what other uses for AI in data are you most excited about? I'm thinking about things like, you know, as a application developer.

• 1:30:05Nathan Labenz?

  I find it still pretty tough in a lot of cases to figure out like, what are people doing as they go through my application? It's easy to like log stuff, you know, log clicks and log whatever, but to really synthesize that into a story, you know, a lot of kind of products try to do that and it's been a big struggle in my experience to get them to work well. But I feel like maybe AI can kind of come along and narrativize. I also think about things like anomaly detection or sort of, you know, determining like what's relevant or not relevant when it's not obvious by definition. But I'm very curious to hear like, you know, aside from this kind of chat with your data

• 1:30:44Nathan Labenz?

  question answering use case, what other roles do you see AI playing in the data, in the data world in the near future?

• 1:30:53Speaker 00

  Suggest descriptions about things. People have to go write these descriptions and they're like, well, at least I don't start from a blank slate anymore, right? So and that and that's just applicable coming. I either write an email about something, right? I just use it and I'll have to start from scratch, right? So I think you're applying that for for that type of metadata, just like any type of documentation. That's a big lift on that thing. You can also use it like use another one. I mean, like you anomaly detection or PII like just pass, just pass the schema to GPT and say, out of this schema, which columns do you think may have PII data, personal information, right? Or it will give you some things of like, hey, those are actually pretty good candidates. And guess what? You don't have to look at the data at all, right? And that's probably even that's something already it's a big lift around that stuff,

• 1:31:38Speaker 00

  right? So a lot of the documentations and kind of this augmentation of metadata, I think that that's one thing. Second, for for search, just in general, I think just more natural language search, it's another one. Again, that's a productivity lift, right? One that I found really exciting is on like explanations, right? So it explains code very well, which is another type of it's another type of of documentation. So a lot of this documentation you do. And one that I really kind of, I find this really interesting working with one of our customers, WPP, who's their worst largest agencies, they want ideations. So I want to kind of use the use the LLMs as a way to inspire me.

• 1:32:18Speaker 00

  So basically it's like data that world, you have your a catalog of our data, you know our data sets. Tell me what are the questions that I could be asking with the data I have. And then it's like, oh, so I start generating candidate kind of questions that you could be asking this question and so forth. And then people like, oh, that's interesting. I don't think I thought about that. And you can keep track of all these questions, start connecting them. And then, and then we've been doing these things like, oh, I have this problem I want to go solve. What data do you recommend me to go use around that? So it's like, well, you can give all this context. Here's all the context that you have. Here are the problem. GPT, come up with, oh, and come up with some interesting solutions and then make suggestions of what they could do. So it's about ideation, but I think that's again, more of the productivity.

• 1:33:02Speaker 00

  So those are things that we're seeing here. Another one is around this call, this knowledge acquisition. It's like, let's go talk to people. Hey, so what do you mean? When you mean customer here, what do you, what do you mean by active customer? Like, right now, we have to have those conversations. Well, human to human, like, what? And now I can kind of automate and scale that out, saying, hey, can you please spend a couple of minutes chatting with my bot? You're going to ask me questions and it's like, we've done this, we'll set up our own GPTs and like be a, be a psychologist, be very Socratic. And ask somebody, what are the questions that they're trying to go, what questions do they need to go ask to the data?

• 1:33:39Speaker 00

  And why is that important? They keep going in, why, why? And try to extract the knowledge. And then in one day, I could, I could talk to 20, 30 different people. And then I could use GPT again to summarize all these discussions, saying, hey, look, when we talk about active customers, here are all these different definitions of what they could be. All right. So that's really helping me again, in a productivity way, faster, understand what we mean by things. And that's the one I'm most excited about, and I hope people get excited about that, too. Because that's the one that has always been that bottleneck. The knowledge, acquiring knowledge from people's head has always been a bottleneck, because you gotta go talk to somebody.

• 1:34:18Speaker 00

  Now we actually can do that at scale with this. And that means that we can learn so much about how the business operates. Map out business operations. And we want to have operational excellence and understand basically all our business processes and like, how can we improve these things? Like, well, we don't even know where our business processes are. Well, I can go interview somebody and say, how do you do your job? And I can map that out and say, wow, look at all these manual steps that happen for this stuff. Like, that's a bottleneck. We need to go improve that so it can be more, we can improve our operations.

• 1:34:51Nathan Labenz?

  How far do you think this goes in terms of productivity lift in the short term? Meaning like, let's just say we have the technology we have and we apply it well. Are we talking like a multiple X increase in productivity? Certainly, I personally feel that in the coding use case. And what does that ultimately mean for data analysis? Like, do we end up, you know, the old bank teller thing is sort of the happy story where, you know, we introduced ATMs and then, you know, people said, oh, that's the end of the bank teller, but it's not, you know, there's been a lot more bank branches open. A classic example, is that where we're headed in data work as well?

• 1:35:35Speaker 00

  In all work, everywhere. I mean, I think we, on our podcast yesterday, we had Jeremiah Alling and he, and we were discussing this like, okay, what, who is not going to be affected in data? I mean, we came to the conclusion that only somebody who literally lives in the middle of nowhere and doesn't talk to you, who doesn't depend on anybody else. You're part of some supply chain and that supply chain is going to get affected by AI. So I think everybody is going to get affected by AI, going to be more productive and stuff. And then there's just the honest, no BS here is that, yes, we're going to have to deal with the job change is going to happen.

• 1:36:20Speaker 00

  There's going to be, there's going to be less jobs and we're probably going to have to figure out UBI. And what we were kind of just talking about is this is probably going to be on the top of the presidential campaigns in, in eight years from now. Like eight years from now, this is going to be a deal that this is going to be the top of the conversation when we're elected president. There's going to be premiums for dealing with humans. Well, at the end, everything's going to be so much automated. We're, we're doing, we're playing around with perplexity yesterday. Like we're doing, we're doing our podcast or pie AI and like, like this stuff is, it is replacing a lot of the humans. I mean, a lot of stuff that humans are doing.

• 1:36:59Speaker 00

  So we'll see is I will pay a premium knowing that I'm going to interact with a human and not with an AI. So I think. Everything is gonna get touched little, little by little. I, another thing I say publicly and hopefully enough people will take this the wrong way is like, I feel bad people who are doing coding bootcamps. Because that's just gonna be completely commoditized. It's, I mean, writing code is gonna be all fully automated. What we're gonna need are people, are our true algorithmicists, computer scientists, people who are thinking about solving the problem because

• 1:37:35Speaker 00

  you're gonna come up with the algorithm, the step by step to solve a problem and the implementation, because you're gonna break down it into small pieces and the implementation, the code for that. It's gonna be the product, it's gonna be, it's gonna be eighty, ninety percent correct, right? So. Knowing how to code, but without knowing how to think about problem solving, like think about problem solving right then. So I think there are all these jobs are gonna go, a lot of them are gonna go away. I mean, we've seen this again over a generation. I mean, when the typewriter came out, people were against it because it's like, oh. If I write it by pen, I know who wrote it. I can see their handwriting. And like, no, this is offensive because I don't know that people are gonna be writing, typing these things that we don't know who's gonna say these things.

• 1:38:21Speaker 00

  People, that was the argument back at the beginning. Yep. Well, I don't think that's an argument anymore. So, so all, all, all these arguments will come in and then we'll go, and that thing is gonna move so fast that we're gonna see so many changes in our generation.

• 1:38:35Nathan Labenz?

  So how would you anticipate that playing out for like an enterprise data team? If, obviously, you know, it can be generic, but like, let's say you're a big enterprise and you've got all these systems and you've got however many people. I don't even really have an intuition, you know, for like, what's a typical enterprise data team size and what are the role breakdowns? But how would you expect that to evolve? In terms of like headcount, you know, which roles, you know, kind of. Change or go away? How much are people then also spending on AI? Like, are they saving, you know, an order of magnitude on their costs? Or is it, you know, does the AI end up being expensive? I, what's the before and after picture as this transformation happens?

• 1:39:16Speaker 00

  The roles that are going to be, that are going to, that are going to stick around or are going to be created are the ones that are very people heavy. Are the ones that are going to, where you need a lot of thought, right? So it's like doing an, they were like a data analyst. It's like, I mean, if you're doing just the generic analyst that like, reporting on metrics that are all very well known in a particular industry. But that's going to go away because the, all the, the large language models, all the AI systems will know. All the, all the foundational metrics in every single industry like that stuff, right? So, but then it's like, people are going to be the creative thinkers. And they're like, well, what about this?

• 1:39:58Speaker 00

  Nobody's thought about like, not even the AI is able to come up with this. I'm going to use the AI to come like, so people are really, really critical thinkers who are, who people can connect the dots from like so many different places. Like, oh, this is probably something you should go figure out. Like, I think that, those are the types of those that will really, really go value. And, and then you can see that in, in the now, in the analysis of the data. And coming up, and the other one that I'm very focused on is on managing the knowledge on your business, right? That's going to get managed in your, but being able to keep track of that, understand the, yeah. The, the LLM's AI agents are going to assist me in kind of collecting all that knowledge, but really then figuring out, okay.

• 1:40:40Speaker 00

  We're going to make the, decide that the note, that the, an active customer means this for these reasons, right? Like, there should be people behind all that stuff. So I think it's all about critical thinking. Those are the roles and then other than that, I think a lot of that stuff is going to get pretty commoditized.

• 1:40:58Nathan Labenz?

  You you kind of alluded to creativity, you know, in insight, eureka moments, if you will, being one of the things that the AIs don't clearly deliver that we, you know, rely on humans for. I very much focus on that as well because it seems like that is a I think about AI a lot in terms of just threshold effects, like when AIs can do something, we can suddenly be in a very different regime from as you know, as long as they can't. And having these like insight, you know, kind of breakthrough eureka moments. It feels like a huge threshold that we have not yet passed, certainly not with any reliability. I would say it's like very rare.

• 1:41:37Speaker 00

  This is on my to do, my to read list. I've seen a couple of papers and I was like, whoa, this scientific discovery only happened because we use AI. I'm like, that is freaking amazing if that's actually happening. Does that mean that we're actually going to advance science, taking faster, taking unknowns and making them known. Like this is freaking amazing that this happens. But it's not, it didn't do it by itself, right?

• 1:41:59Nathan Labenz?

  Lot of scaffolding. Yeah, a lot of kind of framework around them is is definitely still necessary. And but yes, there are, I think we're just starting to scrape at the bottom of that threshold right now with DeepMind recently had an interesting one where they advanced the state of the art in some challenging math problems. It's funny. It's like, I don't even really understand the problems and they're, you know, the AIs advancing the state of the art. Fun search. Yeah. I think of eureka, which is an actual name of a paper out of Nvidia where they used GPT-4 to write reward functions for reinforcement learning for robot, uh, control of like a robot hand. Gabe Gomez, who did the autonomous

• 1:42:42Nathan Labenz?

  chemical reaction optimization with GPT-4. So we're starting to see these like, you know, you might call it sparks of AGI if you're so inclined. What else though, aside of eureka moments is like, that's a huge one. Is there anything else that you would say, you know, if I'm a OpenAI or a Google DeepMind or an Anthropic and I'm like, where are they falling short? What are the, what are the kind of persistent weaknesses that you would love to see us address? Do you have a thought on that?

• 1:43:13Speaker 00

  These foundational models don't know anything about my organization. These foundational models, large language models have One of the largest language model, they're experts in language. They know general knowledge, right? They don't know anything about my organization because I've never looked at it that they were trained by. The context brain of my organization. And that's brain, that brain is the knowledge graph. Now, the thing about the knowledge graph is that it's not a language thing, right? It's like, I can't do natural language, but I have the knowledge. But this knows everything about my organization because I keep tracking it, I govern it, I use AI to govern it. I use AI to create it, but like I, there is this thing that I have and this is my precious. This is the essence of what my organization is.

• 1:43:58Speaker 00

  It's the brain, right? And everything here is accurate. I can use this to explain things. The LLM, these foundational models don't have that accuracy, don't have that explainability, don't know my, don't know my organization. Do we expect these foundational models to know every single organization? No, because these things is private, I don't want them to know. But I want to go use them. So that's why this combination of these foundational models, large language models, with your internal brain of organization, which is your knowledge graph. That's what I see where there's a future. And I think what we need to find, what we need to work on is understanding best that integration point

• 1:44:38Speaker 00

  between the knowledge graph, the brain of the organization and these foundational models. The naive one is a prompt engineer, right? Use some rag architecture, we're experts. This is what's going to happen, what we're going to be working on in the next couple of years to figure out what that integration is. And then it's going to be a plug and play. You're going to have all these foundational models saying, no, I can, I can work with your knowledge graph. I can work with your brain, right? Just play and play, play in your brain here and boom, you're powering up, right? And then also, I, I don't want to be, as my company, I've invested so much in my brain. I don't want to get tied just to one, but I can probably move around. I mean, it's just like, it's gonna get like another cloud, right?

• 1:45:19Speaker 00

  But you maybe could get all bought in, right? We use all, everything is, we're an AWS shop, right? We're an Azure shop. I do everything there. That's probably where they go, but people are still multi-cloud, right? So I want to work with different foundational models. So then, what's gonna happen? So we're gonna have these integration points. I do think that organizations are going to be investing in creating their brain, which is their knowledge graph. You have all these foundational models. And then it's just gonna, well, we're gonna, we're just find better ways of plugging in plain. And that's how we're gonna, that, that's the integration point's gonna happen. And that's all they're focused on. That's how I think about it. I don't know. What do you think?

• 1:45:51Nathan Labenz?

  This is like a project that I'm, you know, maybe starting to embark on is kind of a taxonomy of weaknesses of the current systems. And I do think you're hitting on a big one there that just, I think that ultimately kind of boils down to the fact that, you know, in today's systems, there are the weights, you know, the model itself. And then there's like the stuff that you're putting into it at runtime. And that is, you know, the input, the context, the prompt, but it's limited. And there's nothing between those two. I've been really obsessed recently with state space models as something that may enable

• 1:46:31Nathan Labenz?

  a much longer and perhaps more coherent, perhaps more agentic, you know, perhaps more human-like. Although I don't, you know, I would heavily caveat the human-like because I always try to remember too, that these things are pretty alien and in critical ways. You can fine tune a transformer model for behavior, but you end up kind of narrowing that behavior typically. And you lose a lot of the generality when you do that. So you're kind of dialing it into a specific task. That's to my finding, you know, usually with, with the fine tuning that I've done so far. That you can't really do something where you're like training a transformer model like you onboard a new employee.

• 1:47:15Nathan Labenz?

  And so I'm looking for something where I want to be able to give a

• 1:47:22Nathan Labenz?

  AI system, like a big body of knowledge and say, and maybe like a bunch of history and like, here's a bunch of things that we've tried in the past that have, you know, worked and some that have failed. And I kind of want to download that knowledge into a system, not necessarily update the weights, but have that encoded in some way where it can be then used to inform, you know, the runtime tasks that I want to give it. And I think state space models are shaping up to be a good solution there.

• 1:47:51Speaker 00

  If I abstract what you just said is like, you're giving your body of knowledge, things that you know that works, that it doesn't work, right? Now, how are you representing that? Well, text or whatever, I think all these things should be structured, right? In the graph and the knowledge graph, but basically, but you're giving your knowledge, something you have, and you're going to pass that on to these other foundation models. And you're saying, hey, let's go do something together, right? So, yeah, so it's like, that's the thing is like, you have this foundational models. You have your knowledge, you want to be able to integrate, combine that. And again, how are we going to go do that? How, what are the integration points of the B? That's where we're going to go find out what's, what's, what is, what is even the best one? What does even the best mean? I don't know, we're figuring this out. And then you'll be able to go combine that, you know, and then they're, that's how you're going to deal with the privacy issues.

• 1:48:35Speaker 00

  Like you're, I'm like, so that's what I find really exciting is like, you're combining external and internal things together. And that's what's going to make it super powerful.

• 1:48:43Nathan Labenz?

  You've said a couple of times along the way, the no BS take on it. And I, I know that that's a theme of your own podcast. Do you want to tell us where people can hear more from you if they want to go deeper down the data rabbit hole with you?

• 1:48:58Speaker 00

  Yeah, so I, in addition to the other kind of side work I do at Data World is together with my co-host, Ted Gasper, who's our Chief Customer Officer. We host Catalyst Cocktails, the honest, no BS, non-salesy data podcast. We've been at it for, I think we're, we're on our fourth year now. So we started season seven, I don't know, I've lost, I think we're episode 160 or something like that. We talk about all things enterprise data management analytics, governance, all this stuff. So honest, no BS, because life is too short for BS and drama, so.

• 1:49:30Nathan Labenz?

  Cool, I love it. Well, I'll check that out. I'm sure others will too. For now, Juan Cacada, Principal Scientist, Head of the AI Lab at data.world. Thank you for being part of the cognitive revolution.

• 1:49:41Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.