GraphRAG (beyond the hype)

Seems like we are hearing a lot about GraphRAG these days, but there are lots of questions: what is it, is it hype, what is practical? One of our all time favorite podcast friends, Prashanth Rao, joins us to dig into this topic beyond the hype. Prashanth gives us a bit of background and practical use cases for GraphRAG and graph data. Sponsors: Fly.io https://fly.io/ – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun https://fly.io/speedrun/ to get started in minutes. Assembly AI https://www.assemblyai.com/?utm_source=changelog&utm_medium=podcast – Turn voice data into summaries with AssemblyAI’s leading Speech AI models. Built by AI experts, their Speech AI models include accurate speech-to-text for voice data (such as calls, virtual meetings, and podcasts), speaker detection, sentiment analysis, chapter detection, PII redaction, and more. Speakeasy https://speakeasy.com/ – Production-ready, enterprise-resilient, best-in-class SDKs crafted in minutes. Speakeasy takes care of the entire SDK workflow to save you significant time, delivering SDKs to your customers in minutes with just a few clicks! Create your first SDK for free! https://speakeasy.com/Featuring: Prashanth Rao – Website https://thedataquarry.com, GitHub https://github.com/prrao87, LinkedIn https://www.linkedin.com/in/prrao87, X https://x.com/tech_optimistChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Kùzu: A highly scalable, extremely fast, easy-to-use embeddable, open source graph database: GitHub repo https://github.com/kuzudb/kuzuThe goals and vision of Kùzu: Blog post https://blog.kuzudb.com/post/what-every-gdbms-should-do-and-vision/Kùzu YouTube channel https://www.youtube.com/@KuzuDBGraph RAG strategies with Kùzu: GitHub repo https://github.com/kuzudb/graph-ragPrashanth Rao’s blog: thedataquarry.com https://thedataquarry.comUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:35Speaker 01

  Okay, friends, I'm here with Annie Sexton over at Fly. Annie, you know, we use Fly here at Ching. We love Fly. It is such an awesome platform, and we love building on it. But for those who don't know much about Fly, what's special about building on Fly?

• 0:51Speaker 02

  Fly gives you a lot of flexibility, like a lot of flexibility on multiple fronts. And on top of that, you get, so I've talked a lot about the networking, and that's obviously one thing, but there's various data stores that we partner with that are really easy to use. Actually, one of my favorite partners is Tigress. I can't say enough good things about them when it comes to object storage. I've, I never in my life thought I would have so many opinions about object storage, but I do now. Tigress is a partner of Fly, and it's S3-compatible object storage that basically seems like it's a CDN, but is not. It's basically object storage that's globally distributed without needing to actually set up a CDN at all. It's like automatically distributed around the world.

• 1:36Speaker 02

  And it's also incredibly easy to use and set up, like creating a bucket is literally one command. So it's partners like that that I think are this sort of extra icing on top of Fly that really makes it sort of the platform that has everything that you need.

• 1:50Speaker 01

  So we use Tigress here at ChangeLog. Are they built on top of Fly? Is this one of those examples of being able to build on Fly?

• 1:58Speaker 02

  Yeah, so Tigress is built on top of Fly's infrastructure, and that's what allows it to be globally distributed. I do have a video on this, but basically the way it works is whenever, like, let's say, a user uploads an asset to a particular bucket. Well, that gets uploaded directly to the region closest to the user. Whereas with a CDN, there's sort of like a centralized place where assets need to get copied to, and then eventually they get sort of trickled out to all of the different global locations. Whereas with Tigress, the moment you upload something, it's available in that region instantly. And then it's eventually cached in all the other regions as well as it's requested. In fact, with Tigress, you don't even have to select which regions things are stored in. You just get these regions for free. And then on top of that, it is so much easier to work with.

• 2:42Speaker 02

  I feel like the way they manage permissions, the way they handle bucket creation, making things public or private is just so much simpler than other solutions. And the good news is that you don't actually need to change your code if you're already using S3. It's S3 compatible, so like whatever SDK you're using is probably just fine. And all you got to do is update the credentials. So it's super easy.

• 3:05Speaker 01

  Very cool. Thanks, Annie. So Fly has everything you need. Over 3 million applications, including ours, here at ChangeLog, multiple applications have launched on Fly. Boosted by global Anycast low balancing, zero configuration, private networking, hardware isolation, instant wireguard VPN connections, push-button deployments that scale to thousands of instances. It's all there for you right now. Deploy your app in five minutes. Go to fly.io. Again, fly.io.

• 3:48Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am CEO and founder at Prediction Guard and I'm joined as always by my co-host Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris?

• 4:05Chris Benson?

  Doing very well today, Daniel. How's it going?

• 4:07Daniel Whitenack?

  It's going, it's going extra wonderful today because if I think back over the past sort of couple of years and the way that, you know, these shows that we produce together have also impacted my day-to-day work. One of the ones that has impacted me a lot and also I think been super beneficial, just connection, is, is one, the one that we did about kind of vector databases, Beyond the Hype, and super happy to have our guest from that show back, who is now an AI engineer at Kuzu, Prashanth Rao. Great, great to have you back on the show.

• 4:49Speaker 05

  Oh, yeah. Hi, Daniel. Hi, Chris. Very good to be back on the show again.

• 4:52Daniel Whitenack?

  Yeah, I've taught a lot of like workshops over the past couple of years. I've done like trainings with our customers. I've built things with our customers. And always when it gets down to vector databases, I'm like, hey, you know, anything I say, like, you can maybe learn some things from me, but if you really want to learn about vector databases and get the right sort of intuition, you really need to go to this series of blog posts on the Data Query, which is your, which is your blog, which was the first thing that I saw, which brought you originally on the show. We talked through vector databases and trade-offs between different types of vector databases and, and all of those things. Since

• 5:35Daniel Whitenack?

  that time, of course, you've moved on to a, to a company, a data company, working, you know, in a different type of data, which we'll talk about today, but also is related to some of those things. So, yeah, how has the year been? Lots of updates and fun things it sounds like you're working on.

• 5:54Speaker 05

  Oh, absolutely. So I think I can summarize very briefly where I'm coming from since the last time we spoke, which was, I guess, just about a year ago. So as we spoke in our previous episode on vector databases, in 2023, I was working in the Royal Bank of Canada and I spent most of that year thinking about vector search and the difference between various database options there. And during that time, I was also simultaneously working on using graphs and graph databases. And I'll talk a little bit more about what they are and how they're useful. But as I was working on them, and I think I mentioned this even in the conversation we had last year, is that I'm very interested in understanding how these two worlds come together. Because I know knowledge graphs and graphs have been around for a long time in their

• 6:39Speaker 05

  own space. And there was a huge hype around vector search and vector semantic search around this time last year. So there was a lot of scope, I guess, to understand how these two systems and these two methods can work better together in a way that we can build more advanced retrieval systems. So while I was thinking about those topics, I discovered new open source embedded graph database, Kuzu, which is where I now work. And I happened to meet the CEO, Semi Salihodou, at a local meetup in Toronto. And that got me really interested in what this tool and technology is all about, because I'd been using other graph databases. And when I discovered Kuzu, the fact that it's embedded and open source, which I'll talk a little bit more about, was very attractive to start with.

• 7:22Speaker 05

  So I spent my spare time outside of work experimenting with it and engaging with the developer team, as well as working with other people in the community in terms of understanding how they're using graph databases and graph tooling. So, long story short, I realized at the end of last year that this was a great and fun opportunity for me to really go deep into the world of graphs and graph databases. And at the beginning of this year, I moved over and joined Kuzu as an AI engineer. And currently, I'm leading the developer relation efforts and I'm engaging with the growing user community of Kuzu, which is really, really fun in itself. And of course, working with use cases that are taking this whole space forward and working with the community to understand what they are using these tools for. And I should also add, I'm still very plugged into the vector database ecosystem.

• 8:07Speaker 05

  I've not left that behind. And I'm actively experimenting with some of those in my spare time as well. And I still very much enjoy using LanceDB, which is the tool I mentioned last year in our conversation as well. So again, it's really exciting to see how these different kinds of tools are coming together. And yeah, I'd love to go deeper into some of these topics in our conversation.

• 8:26Chris Benson?

  Fantastic. Yeah, the episode that we did last time. With Lance and with the with the other vector database issues, that was a great episode. And if anyone isn't familiar with that, they definitely should go back and listen to that one as well. But just to lay a little bit of foundation here, some groundwork for those who may not be intimately familiar with graphs and how are they useful and applicable and such, could you kind of start us off with just, you know, what is a graph and why should we care about it? And what's it going to do for us?

• 8:58Speaker 05

  So graphs are sometimes also known as knowledge graphs, and they're a great way to represent structured data via nodes and edges. The nodes are basically entities in the real world, like a person or a company. And the edges are relationships between these entities, like worked with or is CEO of and so on. Now, more formally, I also want to make the point here that the term knowledge graph is actually used to mean something more slightly more involved than what I just described. A knowledge graph, in theory, can be used to express data that's quite hard to tabulate and store as records. It supports some logical reasoning over graphs as well. This is kind of what you would do when you store Wikipedia data, for example. But that's outside the scope of what we should

• 9:41Speaker 05

  go into for the purposes of this discussion. For most people and developer audience, the term graph and knowledge graph can be used interchangeably. So I just use them interchangeably for this discussion.

• 9:53Chris Benson?

  It's kind of like distinguishing machine learning and AI to some degree. The edges, pun intended, don't don't don't matter that that much in terms of their actual use. I want to direct you for one second before we get past the point is you did mention about graphs and you know, briefly mentioned it in in relation to a relational database. And just because, you know, over they've been around so long, and it's where a lot of people are starting from on relational databases, you know, and you talked about the edges of the graph, you know, describing those relationships and being able to do that in some ways much better than a relational database can. Could you talk just for a moment

• 10:34Chris Benson?

  so people can kind of make the jump from where they're at into this new tool and maybe maybe think next time I'm going to leave whatever database I'm in, Postgres, you name it, and actually try a graph approach. Could you talk just for a second about what the difference there is and bring people along on that path?

• 10:51Speaker 05

  So I think they're going into the description of what a graph database is, right? So I just quickly, before I go into the database aspect of things, I just want to point out the benefit of what a graph is even useful for. Because most people need to understand what a graph is for, and then you think about where to store it.

• 11:09Chris Benson?

  Great point.

• 11:10Speaker 05

  So I think I want to highlight that the benefits of graphs become obvious when you're looking at data itself that's highly interconnected. So for example, in medicine, you can have relationships between patients and symptoms of diseases, drugs that treat those diseases, and their side effects. And these all have complex, interweaving relationships with one another. Similarly, in finance, you can have chains of direct or indirect transactions, let's say between onshore and offshore accounts. And these ultimately connect to given individuals anywhere in the world. So you could choose to model these using tables in a relational database, or you could choose to model this as a graph in a graph database. And in either case, there are pros and cons.

• 11:51Speaker 05

  But specifically, when you're analyzing patterns and you want to actually understand these complex relationships in an analytical way, using a graph database can actually be very powerful. It's a lot more intuitive, it's much easier to construct the queries that can answer these questions. And this is where the idea of a graph and how you model the data as a graph becomes very, very powerful. But going back to the idea of databases. So a graph database can be thought of as a specialized database that allows you to scalably manage and query data that's organized as a graph. And the performance aspects of these graph databases come from specialized data structures and operators that allow you to express complex joins and efficiently traverse paths in your data. Now, a lot of listeners may have heard of graph databases already, and the most popular

• 12:36Speaker 05

  graph database model is called the Property Graph Data Model, which was invented and popularized by Neo4j. And I've been a user of Neo4j myself in the past. Today, you have many other graph databases like Kuzu that also implement the Property Graph Data Model. But in general, you start off with a data model, which is more conceptual, which allows you to express how to store and query your data. And a graph database basically is the underlying system that allows you to express the graph data model of choice. And the reason it's more intuitive than a relational database for certain kinds of queries that involve connected data and the sort of interrelated data is that it allows you to express your queries in a much more concise and intuitive manner

• 13:19Speaker 05

  using the query languages that graph databases offer, as well as the performance aspects of traversing the parts in the data.

• 13:25Daniel Whitenack?

  And as, just to sort of hone this in a concrete way, are you able to give like a few examples, like there's sort of the typical personnel related things that you already mentioned, like, you know, Daniel is a node and he's CEO of Prediction Art and that's another node and there's organizations. Maybe for people that can't connect, how such a structure would kind of be relevant to data that maybe they have in their enterprise? Could you give just a couple other examples of data that could be represented as a graph in maybe in different verticals or that you've run across in your use cases with Kuzu?

• 14:08Daniel Whitenack?

  Sort of data that's kind of beyond that kind of social network type of data, I guess?

• 14:13Speaker 05

  Absolutely. I think social networks get a little excessive credit for being the original graphs that we know about. Of course, they're very powerful and of course they're used, graphs are heavily used in social networks. But yeah, I think I mentioned an example of the medicine scenario which I described, which is you have not just individual persons or patients in a network, but you also have the drugs and the different symptoms of the diseases that are being treated by those drugs. Each of these can branch out into much more complex relationships themselves. And a lot of this data that you might imagine in a healthcare scenario or a biomedical scenario, this data pre-exists in structured form in many different sources.

• 14:55Speaker 05

  You might actually have medical records of people in a relational database or a data lake somewhere. A lot of existing workflows that work with these kinds of data sets tend to just stick with the database that is already used as a primary store. And many of those happen to be relational databases, which is fine, because they're the most, I guess, efficient and convenient way to store this kind of data. When you have records of people, let's say what drugs they've been taking and what symptoms those drugs may cause and so on. So I guess the idea here is that you can actually think of that very same data that exists in a relational database as a graph and store that in a graph database. And apply your graph query logic in a way that allows you to answer specific questions

• 15:39Speaker 05

  that might have been quite complicated to answer using SQL in the relational database. So that's one example of, let's say, a healthcare scenario. I've already mentioned the financial transaction scenario, where you have a transaction graph between individuals, the merchants they've interacted with, the money transfers they've made between their accounts, how these accounts are connected. So financial institutions make heavy use of graphs to answer these kinds of questions. Traffic networks is another common use case. If you're working with city authorities and you want to understand the flow of traffic and the numbers of people moving between locations, this is something that can actually be well modeled in a graph. And the kinds of questions you can answer can also change based on whether you choose to model this as tables or as a graph.

• 16:24Speaker 05

  And there are many, many other examples. I will highlight one thing, though. The example of Wikipedia that I gave earlier, the idea of Wikipedia as a graph, I think, reinforces the idea that knowledge graphs are a term to be used, I guess, with a bit of caution, in the sense that, in general, a graph is a general representation of how nodes and entities, like nodes are connected to other nodes in the network. A knowledge graph is something that you can think of as the collection of all knowledge that is available in that domain. And in the case of Wikipedia, you can imagine that there are certain scenarios where it's quite hard to tabulate every bit of information. As an example, if you have, I don't know, the current president of the US is Joe Biden.

• 17:08Speaker 05

  So based on the current political structure and the different parties that Joe Biden represents and all the other people represented in those parties, and then the relationship between a party and a state, the relationship between the state and the country, you can imagine that this becomes a very complicated branching sort of structure. And not all of it renders very well in a table, because you can't imagine like one table that connects to another table when you Have this kind of complicated data. So there are certain scenarios where actually the data model and the way you build your graph can actually really have a big impact on the kinds of questions you can ask of your data. So that's why I tend to use the term knowledge graph in a bit more specialized way.

• 17:49Speaker 05

  In general, I would say that you can think of your tabular data or records as a graph very conveniently using the property graph model and model things like transaction networks and social networks and drug interaction networks and so on.

• 18:17Speaker 01

  What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Dylan, tell me about Universal One. This is the newest, most powerful speech AI model to date. You released this recently. Tell me more.

• 18:33Speaker 06

  So Universal One is our flagship industry-leading model for speech to text and various other speech understanding tasks. So it's about a year-long effort that really is the culmination of like the years that we've spent building infrastructure and tooling at Assembly to even train large-scale speech AI models. It was trained on about 12 and a half million hours of voice data, multilingual, super wide range of domains and sources of audio data. So it's a super robust model. We're seeing developers use it for extremely high accuracy, low cost, super fast speech to text and speech understanding tasks within their products, within automations, within workflows that they're building at their companies or within their products.

• 19:16Speaker 01

  Very cool. So Dylan, one thing I love is this playground you have. You can go there, assemblyai.com/playground, and you can just play around with all the things that is Assembly. Is this the recommended path? Is this the try before you buy experience? What can people do?

• 19:33Speaker 06

  Yeah, so our playground is a GUI experience over the API that's free. You can just go to it on our website, assemblyai.com/playground. You drop in an audio file, you can talk to the playground. And it's a way to, in a no-code environment, interact with our models, interact with our API, to see what our models and what our API can do without having to write any code. Then, once you see what the models can do and you're ready to start building with the API, you can quickly transition to the API docs, start writing code, start integrating our SDKs into your code to start leveraging our models and all our tech via our SDKs instead.

• 20:08Speaker 01

  Okay, constantly updated speech AI models at your fingertips. Well, at your API fingertips, that is. A good next step is to go to their playground. You can test out their models for free right there in the browser, or you can get started with a $50 credit at assemblyai.com-practicalai. Again, that's assemblyai.com-practicalai.

• 20:45Daniel Whitenack?

  Well, Prashant, before we get into one thing that I'm really excited to talk about on the show, which I've been telling Chris we need to talk about for a while, which is GraphRAG. But before we get there, we sort of talked a bit about graphs. I think it would be useful to kind of just give people a reminder of the kind of, what most people would refer to maybe when they're referring to a RAG workflow, an AI RAG workflow. I was at a CIO dinner last night and one of the speakers is like, "We all hear about the RAG and the raggedy rag, ragged rag, rags." And everybody is, everybody's hearing about these things, but maybe they, maybe it's worth just a quick 30-second, one-minute reminder.

• 21:29Daniel Whitenack?

  So when most people are referring to the rags or the rag workflow, what are they referring to?

• 21:37Speaker 05

  Yeah, this is a very fascinating topic. I think about rag a lot. So let me just, I think, take a step back and describe the term itself.

• 21:49Speaker 05

  So the term RAG itself is really interesting to me. As we know, it stands for Retrieval Augmented Generation. The key thing to note here is that the term RAG emerged prior to the emergence of the term LLM. And it's come as a result of generative language model improvements that came, I think, in the end of 2019, early 2020 time period. And there were two papers that came around in early 2020, one by Google and another by Facebook Research that introduced the term Retrieval Augmented Generation. Now, note that retrieval itself is not new. We know information retrieval is a field that's been around for decades. So, and we've also had systems that can do keyword-based information retrieval for decades.

• 22:32Speaker 05

  So what is new right now is the fact that generative models, generative models have become much better than they used to be. So the generation capability is what's new. So when you look at the term RAG, the term Retrieval Augmented comes before the term generation in that acronym. So I hope that makes it clear as to the fact that the generation part is what we are stressing is the novel aspect of it. Now, in 2020, the way RAG was done was to combine a sequence-to-sequence language model, which was the standard we are doing language modeling back then. And those models could generate text quite convincingly. And you combine those models with the retrieval capabilities of dense embeddings. And the Facebook Research paper that introduced the term RAG based their work on dense embeddings of Wikipedia articles.

• 23:17Speaker 05

  Now, you had the initial makings of vector embedding-based retrievals that we are taking for granted today. And later in 2020, GPT-3 was released. So you had the coinage of the term LLM or Large Language Model. And you could think of GPT-3 as one of the first large language models that really extended the capabilities of pre-existing models at that time. But what really made RAG take off, in my opinion, from 2021 and beyond, was the arrival of this whole host of new systems that we call vector databases. And they began offering specialized features to make retrieval from these dense embeddings far easier and also more scalable. And this is what led to the explosion of all those vector database companies that we discussed about last year.

• 23:59Speaker 05

  And I hope this gives some context as to the term RAG itself and how, I guess, it grew to what it has become today.

• 24:07Chris Benson?

  I guess we've been teasing this for a while on the show here. We've talked about graphs. We've talked a little bit about RAG to get people, everyone's waiting for us to ask you the question about, you know, graph RAG and get into the topic. So without further ado, if you want to dive in and kind of give us an intro to that, would love to hear that.

• 24:25Speaker 05

  Absolutely. So let's understand what we were doing with RAG and then go into graph RAG.

• 24:30Chris Benson?

  Yep.

• 24:30Speaker 05

  So the early approach to doing RAG is, which we call it naive RAG now. And in that approach, you just create chunks of your data, you embed that using an embedding model, and you store them in a vector database. So essentially you just store the chunks on the chunk embeddings in a vector database. And when you do a retrieval, you convert your query into an embedding model using the same embedding model that you used to embed the data. And this returns the most similar chunks that are similar to the query vector. So you typically return like the top K, like let's say top five or top 10, whatever number you choose. And these top K chunks can then be sent to the LLM as context to synthesize a response in natural language.

• 25:14Speaker 05

  So in a nutshell, that's kind of what you could say traditional RAG does. Now, on paper, this naive approach to doing RAG is great, but it quickly became obvious that this has limitations. The first limitation is that the dense embeddings are typically done at the sentence level. And many user queries use keywords and keyword-based search methods like BM25 can do a fair job at this, and they've been around for a long time. So towards the end of last year, you could see a lot of these vector database vendors starting to offer a combination of hybrid search methods and the term hybrid search itself becoming more popular, where you perform both keyword-based search, which is a form of sparse vector search, with dense vector search, which is a search via dense embeddings.

• 25:57Speaker 05

  And you pass the retrieved chunks from either of these approaches to a rerank a module. So you had specialized modules that do re-ranking that give you the most relevant chunks from either of these retrievals. And this is how you combine the sparse and dense vectors into what you call a hybrid search. Now, even hybrid search can have its limitations, which is, I guess, where people began exploring further options earlier this year and maybe beyond. Because neither sparse nor dense embeddings can capture explicit relationships between entities very well. And I'll demonstrate this with an example. In certain cases, you can really benefit by modeling some of these entities explicitly. So let's look at an example of professor and let's say the PhD students the professor is advising.

• 26:42Speaker 05

  So let's say you had a block of text which is talking about the students and the professor and a bunch of other things related to their work in the university. So in natural language, we understand the relationship between the professor and the student as follows. Student X worked with Professor Y. Because we know that the act of being a student of a professor means that you worked with them. But in the text itself, you may not have expressed it that way. The text may be written as so and so person X was a student of person Y. Now, if you try to search this using the query, who did X work with? But this is a very intuitive question in natural language. We humans immediately can put two and two together, that work with and student relationship are more or less semantically similar here.

• 27:27Speaker 05

  So we are able to piece together this information and know that a person was a student of someone and inherently they worked with that person. However, if you try to search for this using vector search, the dense embedding may not capture the relationship correctly, where student of isn't close enough to worked with in the vector space. So your vector search alone may not retrieve this answer because you didn't model the relationships in that explicit way. However, if you had chosen to model this as a graph, you would explicitly capture this relationship using this concept of a triple, which is person X worked with person Y. So this is the idea of where triples come in. A triple essentially is two nodes that are connected via a relationship. You have a source and a target.

• 28:10Speaker 05

  And the person X is a source, person Y is a target, and the worked with is what represents the relationship. So the very powerful idea here is that where graphs come into this whole picture and why it's relevant to RAG is that you can actually provide additional valuable context to an LLM by modeling these relationships explicitly.

• 28:31Speaker 05

  And simultaneously retrieving both from a dense embedding vector search as well as a graph traversal and then using the retrievals in combination with one another to provide additional context to the generation LLM so that you can actually include this explicit relationship in your answer. And this actually has been proven in practice from some work that's been done recently.

• 28:54Chris Benson?

  I want to ask a question. It may be a bit of a stretch, but could, as you were describing that, that failure of making those explicit, if you're doing that, could that lead to hallucination in terms of your output from the model because it's not able to make that explicit and so it's still trying to provide information and it comes up with whatever it comes up with? Is that, is that a possibility there?

• 29:14Speaker 05

  That's a very fair point and you're absolutely right. Hallucination, I think, is definitely a high possibility or like it's definitely likely in certain scenarios. You obviously cannot predict when hallucination happens, which is a big issue in general with LLMs. Now, I think the, I definitely want to expand on this a little bit. The selling point for graph rag, by the way, the process I was just talking about before is essentially what you could loosely define as graph rag where you bring a graph as part of the retrieval process. Now, the so-called, I guess, benefit of graph rag as it was, you could say marketed in the last several months by various sources is the fact that it reduces hallucinations.

• 29:54Speaker 05

  And it's important to note that whenever context is provided to an LLM for the purposes of generation, essentially you provide a prompt and prompt is what LLM uses to provide a response. In the event of providing such a prompt, it's always possible that at some point of time you will have a hallucination. It doesn't matter whether the information came from a graph or the information came from a vector retrieval, the source of the information is irrelevant. So the very act of using an LLM to generate text means that there is an inherent chance of hallucination. So I wouldn't state that the benefit of graph rag is that it eliminates or reduces hallucination. What I would state the benefit is, is that it actually increases the chance of factual accuracy in the sense that a relationship

• 30:38Speaker 05

  that was not explicitly captured in this vector embedding is now explicitly captured in the graph. And by providing both these pieces of context to the LLM, you're essentially increasing the chances of a factually correct or a more relevant response.

• 30:51Chris Benson?

  Yeah. So one of the things I'm thinking about is people might have

• 30:57Daniel Whitenack?

  Some thoughts in their mind, like I know maybe people have built some sort of naive RAG system, or maybe even implemented some advanced RAG methodologies. But most of the time, what they've had is a sort of set of documents. Like you say, they split up into chunks, they embed those, they retrieve one or more chunks, maybe even in a hybrid way or in some advanced way. But here, you're saying you're combining the vector approach and the graph approach. I'm wondering if you could break down very concretely for us, the data side. So if I have like documents, or if I have internal data, what I would need to have in place and how I would construct the data side to be ready to do graph RAG.

• 31:40Daniel Whitenack?

  And then at the time that I, let's say I receive a user query question in my chatbot, or whatever that is, what is actually kind of concretely retrieved? And how is that combined? Or how could that be combined with the prompt into the model? So just walking us through those kind of very concrete things might be helpful for people.

• 32:00Speaker 05

  Sure, that makes a lot of sense. So the two key stages in any RAG application, not just graph RAG, is the fact that you divide it into an indexing stage and a retrieval or a serving stage. So to get the data in and store it and index it is what we call as indexing stage. So this is the stage that is upfront or upstream, where you have data that already exists in different structured or unstructured sources. Now, you could apply a variety of techniques, including using LLMs itself for this stage, where you could extract the entities, or you could say named entities, from the unstructured data or structured data that already exists, and store them as entities or nodes in a

• 32:45Speaker 05

  graph, graph database. And simultaneously, you can also extract relationships from this unstructured text. There are many different methods that you could use to extract the relationships. Now, what I've noticed in recent times is a lot of these recent papers are using LLMs to help with this information extraction step. Where you actually use the LLM to extract portions of your chunks and then from those portions, you further refine it and say, okay, from this block of text, tell me what are the triples, which is nodes connected to other nodes by a relationship exist in that block of text. So, um, once all this triples are extracted, they're essentially stored in a graph database. And simultaneously, you can also have a parallel pipeline that stores

• 33:30Speaker 05

  the vector embeddings in a vector database. In some cases, you can have both the vectors as well as a graph entities sitting in the same database. Certain databases have those features. Uh, in other cases, you may not want to have them both in the same database. You may want to leverage the graph database for its strengths and the vector database for its strengths. Uh, and you may also have pre-existing workloads that already have the data in those systems. So it's perfectly valid to have your independent sources of data move the respective datas into those respective databases. For example, your graph entities would go into your graph database and the vector embeddings would go into a vector database. Downstream of this, you could do additional post-processing

• 34:12Speaker 05

  like linking the chunk reference IDs to individual entities in the graph. Uh, essentially, the, the node that represents an entity in the graph can have an ID that links it back to which chunk it is a part of, so that when you retrieve a particular chunk, uh, you can actually point to which entities are existing in that chunk. So these are a lot of additional upfront steps that people are doing to construct both the graph as well as the, the vector store. And once this indexing stage is complete, the retrieval stage can actually begin, which is the, the stage that we are very familiar with. You have a user query that comes in, in natural language. Uh, you transform that into an embedding. You find the similarity, uh, uh, search on that embedding, which is, uh, returns the most similar vectors from the vector database.

• 34:56Speaker 05

  And then simultaneously, you can also use, uh, whatever methods you have to translate that query into a graph query and retrieve the entities and relations from the graph that answer that same question. And then use a reranker to combine the retrievals in a way that provides additional context to the LLM for generation. So again, just to summarize, the two key stages in the RAG pipeline include indexing and serving. And each of these stages has a suite of tools that you can use to, uh, help the user achieve the required output.

• 35:35Speaker 01

  Well, our friends over at Speakeasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing docs, and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with PyDantic models for requests and response objects, and HTTP/AXE client for async and synchronous method calls, and support for server sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at speakeasy.com/python.

• 36:19Speaker 01

  Again, speakeasy.com/python.

• 36:37Chris Benson?

  So I am, here we are after break. I am still thinking about what you were telling us going into break and trying to grok it myself. And I'm kind of thinking about how I can use it in a practical sense to help me get it down and kind of get it from the notional sense into more of a practical thing that I can go do after we stop talking on the podcast. Can you give me like an example, something really hands-on that that folks out there might be doing that that really puts it into that, okay, I get it. Now I'm going to go do it kind of context?

• 37:12Speaker 05

  I actually have a repo that I can share once we're done with this conversation. And I'd love for people to pick this up and experiment with it. And obviously, because I work at a company that builds a graph database, I'm very eager to talk to users who are using these kinds of tools. So for my experiment, I've used Kuzu as a graph database and LanceDB as a vector database, because each of these, as I mentioned, have their own benefits in their domains. And a practical example that I want to demonstrate is this. So the data set I'm thinking about, which I can showcase in the repo, is You have a block of text about the scientist Madame Curie. You know, she discovered radium and polonium and she won two Nobel Prizes. And she was related to Pierre Curie, they were spouses.

• 37:56Speaker 05

  And she also was related to other scientists in the whole ecosystem. So I have a text sample that contains Madame Curie's contributions to science and the relationships that existed in her life. So this is unstructured text. So the first step is we can do two things here. We can do the conventional naive rag retrieval where I try to ask a question. Who did Pierre Curie work with? And that's a very simple question to answer. And a vector search will definitely give you an answer if you embed the text into the required steps. So what I noticed in this data set is there is an implicit relationship. And this is why I gave that example earlier about the professor and the students.

• 38:38Speaker 05

  There is one particular person in this data set, Paul Langvin, who was a student of Pierre Curie and who later had a relationship with Marie Curie after Pierre Curie passed away. So it's mentioned in the text that Paul Langvin was a student of Pierre Curie. Now, the question asks, who did Pierre Curie work with? We obviously know that Pierre Curie worked with Madame Curie to find or discover these elements. Now, we also know implicitly that Pierre Curie worked with Paul Langvin, who was his student. The vector search, if you naively chunk these and store them in a vector database, which I do in LanceDB, gives me the one of the answers. It gives me Marie Curie worked with Pierre Curie. But the graph search, because of the fact that I explicitly insert the relationship

• 39:21Speaker 05

  and I have some code that shows how the information was extracted from the unstructured data using the Llama Index framework. So it's very intuitive and easy to begin experimenting once you actually install the required packages. So what I'm getting at here is in the graph, I was able to retrieve both the answers, like the Paul Langvin as well as Marie Curie, who worked with Pierre Curie. And rather than just using the result from the graph, there may be other scenarios where my question may have been a little bit more vague or fuzzy and a vector search might have given me a better result. I'm sure if you tinker with the dataset and the questions here, you'll find such examples. So what I've done is I've included a re-ranker downstream of the vector search and the graph search.

• 40:03Speaker 05

  And when I retrieve the result and pass it to the, as context to the LLM, I'm adding that re-ranker step so that I get the most relevant graph search results as well as the most relevant vector search results. In this case, the vector search missed one of the entities, but the graph search captured it. So the combined context from both these retrievals allowed me to get the generator model to actually give me the correct response. And if you want to experiment more with this, I think it's pretty straightforward to come up with other queries that will show the reverse result to be true, where the semantic match between the vector search and the query might be closer, you might get a more relevant result from the vector search and the graph search missed the result because of a mismatch.

• 40:46Speaker 05

  So that's kind of where I'm going at here. There are many ways you can combine vector search and graph reversals to improve the retrieval accuracy. So I'd love for the community to think about these individual parts. I think my biggest takeaway from what I've seen in the last few months of reading the literature and talking to people who are questioning what GraphRAG is, is that people tend to think of GraphRAG as like a graph-based solution alone, whereas the more I think about it, I think that the two approaches of using dense vector retrieval and using graphs kind of go hand in hand for the purposes of RAG. All this being said, there are a lot of, I guess, conflicting articles that you may

• 41:30Speaker 05

  see online and blog posts claiming that this is the way to do GraphRAG. The key takeaway for everyone should be, think of it as a suite of tools and methodologies that come together to enhance retrieval in a way that you can get better generation.

• 41:43Daniel Whitenack?

  I would love to maybe double-click on one of the things that you mentioned, which I had in my mind and I think other people might have in their mind, mainly because I don't know how many years ago it was, five or six years ago, I was doing some graph-related data work. And I know I looked at a lot of things at that time. There's even a whole area of research called automatic or automated knowledge graph creation. And there's sort of this idea that, you know, if, once you have a graph, there's so much that is opened up to you in terms of querying and the rich structure and all of that. But sometimes it can be daunting to construct the graph in the first place. And you mentioned kind

• 42:27Daniel Whitenack?

  of this nice tooling around Llama Index. And of course, we had Jerry on the, on the show in the past, and such an amazing project there that's helping many people. But I'm wondering if I could double click on that point. Like, what has been your experience kind of current state in terms of how much work and how hard it is to do that graph construction piece in addition to, because I think one of the things people love about RAG, right, is you just kind of put a bunch of documents in and hands off, you construct these chunks and, oh, cool, like you get some, some nifty things out of it. Of course, there's the element that you, you highlighted, which is sometimes it's hard for people to

• 43:08Daniel Whitenack?

  get that last bit of performance that isn't, you know, captured by naive RAG. But yeah, on that, on that graph data construction piece, yeah, how, how does that look right now? And what have you found to be useful?

• 43:20Speaker 05

  You hit the nail on the head. I think the biggest issue that people I speak to are having in relation to both GraphRAG and in general using graphs is how do you create the graph from existing data that you have in other forms? And yeah, I was exactly going to touch upon this anyway as my next point, which is GraphRAG is by no means perfect. And the most significant challenge is indeed around graph construction. Now, there's two things regarding graph construction that we can delve into. The first is that the quality of the graph is absolutely paramount, because as we know, in any RAG system, the quality of your retrieval greatly impacts the quality of the generation downstream. Like, a poor retrieval with garbage results is going to result in a garbage output from the generation model.

• 44:03Speaker 05

  So first of all, we have to stress on the fact that to get the most out of Graphrag, we need a high-quality graph. But then you go one step further back, which is how do you even get a graph from unstructured text? So as you mentioned, LaMa Index and I think LangChain as well, some of these frameworks offer valuable tooling to help you extract triples or entities and relationships from unstructured text, and they do this primarily through the use of LLMs. But as we know, LLMs themselves have issues with hallucinations, or they just have issues in general with reproducibility. You are not guaranteed to get the same results if you run the same LLM multiple times. And although some APIs provide seeds where you can control the reproducibility, that

• 44:47Speaker 05

  still doesn't mean that it's not random. The output of an LLM is still more or less unpredictable. So there are a lot of other parallel works going on that are not using LLMs to extract triples from an unstructured text source. And I can talk about a few of those that I found really exciting, and in fact, that's kind of the stuff I've been exploring in this space, and I'd love to chat with people who've been doing this as well. So there are custom machine learning models coming out. One of them is called REBEL, which is R E B E L. And it's been around for a while, but I think they're upgrading their internals now for a newer version. So that's a model that's trained explicitly for the purposes of extracting triples for the purposes of a graph.

• 45:30Speaker 05

  And there's another model, I think, called RELIK. That's also an open source model that's been released recently, and people have been using it to extract relationships from unstructured text. Now, it's not to say that these are mature models. It still requires a little bit of coercing, and people need to understand their outputs and what they're getting from them. But the idea here is that these models are, I guess, more controllable in what you can output from them. It's not like an LLM where you just don't know what you're going to get. And of course, I think because of the fact that they're small models, you can actually use them at scale on very large data. So that's one angle of things. I'm sure a lot of users have heard of the NLP library Spacy. And I've been using Spacy for many years myself. It's a very like it's an amazing library.

• 46:15Speaker 05

  On its own, Spacy doesn't have the tools to extract relationships and entities from text. Of course, you can extract named entities or named entity recognition, NER. But recently, there have been some add-on modules or libraries that plug into Spacy. Two of them, which I'll name here, they're called GLiner and GLIRL, G L I N E R and G L I R E L. And as their names suggest, one of them is for extracting named entities from text. And the other one is for extracting explicit relationships from text. But they plug into the underlying Spacy tokenizer and the underlying Spacy representation of the data. So I feel like it's a lot more usable. And there are some experimental notebooks that I've been working on.

• 46:56Speaker 05

  And I'm going to be experimenting more on this to see how they compare in relation to using just L L M's alone to extract data. So yeah, I think this is a very active space. And there is no right answer in terms of how you can use these tools. But I feel like there's enough options and methods out there that people don't have to feel that I'm relying on, you know, completely black box unreliable L L M to do things.

• 47:20Chris Benson?

  What you've covered so far is really fascinating to me. It's and I'm learning a lot in this conversation. As we're kind of winding up on this, you know, and we've been covering, you know, Graphrag as well as the components that make that up and kind of talking about how to combine vector search, graph search. And it feels like a, you know, a brave new world that we're we're leaping into even even for the A I topic, which is that way anyway. Where's this going? Like where where where do you envision this going over various timelines in the future? You know, things are happening at light speed, so you just go a few months and there's a lot of change. But if you could maybe pick a couple of points out and and tell us kind of what you think might happen.

• 48:04Chris Benson?

  If you're wrong, no big deal, but I'd love to see what your imagination has in store for us there.

• 48:10Speaker 05

  Yeah, I'd love to see this from a year from now because I know half the stuff I'll say will be out of date or wrong, but.

• 48:15Chris Benson?

  Of course, but that's fine. That's this topic.

• 48:18Speaker 05

  The stuff I said last year, I think I was still spot on in the sense that I did talk about graphs and vectors last year too. And it's still relevant today. So yeah, I'll take my chances this time around. So I think, yeah, you're absolutely right. Things change way too fast. And my personal take here is that graph RAG is at a point in time. And we don't even know in the next three to five year timeframe whether RAG is going to be as hot as it is today, right? I don't think L L M's are going away anywhere. Let's face it. I mean, they're here to stay and they're going to continue to evolve. We've just seen with the O one model that came out from OpenAI. They are able to do reasoning. They're able to do so much more than what people give them credit for.

• 48:58Speaker 05

  So as this capability keeps evolving and L L M's keep, you know, morphing into whatever else they become, there's no guarantee that the kinds of tasks being done today using, you know, custom models, machine learning models, won't be done by an L L M. Now, some of the wild takes that I've seen involve the fact that you don't need to index anything in a database. You could potentially have your L L M's parameters store the index in a very fuzzy way and you could retrieve from that index, assuming that research goes in a certain way. But obviously, that's a long way away. I don't see that happening in the next few years. What I'm particularly excited about in the next few years is everyone's been talking about agentic systems.

• 49:40Speaker 05

  And if you look at the pivots that all these framework companies have had in the last year, Lang Che and Lama Index, like they're heavily trying to push this field forward in terms of how agents can help build, I guess, more capable systems. Now, RAG is just one small subset of the things agents are orchestrating. They can actually do many other things related to recommendation and could say like search and retrieval and a host of other activities. So the conventional way or the standard way of doing agents has been using this framework called React, which is, you could say like, it's sort of like a graph based framework, but it's sort of static where you kind of decide the behavior of the system by programming

• 50:22Speaker 05

  in these paths and the agent acts on it and then you send it back after the reasoning step is over and so on. But where I think actually graphs are going to be very interestingly used in the future, and I've seen some examples of frameworks that are already doing this, is can you use an underlying graph representation of action spaces to guide your agents? That is, can the agent actually actively update the action space via a graph structure? And once that happens, you can essentially have sort of more powerful agents that are not constrained by a rigid static sort of framework. You kind of have a dynamically updating agentic system, but it's not as open-ended as having this recursive agent calling that

• 51:05Speaker 05

  we have today, right? You still have a framework. The graph kind of acts as like a base structure for the agent to take its actions. So that's definitely one aspect of where agents are going. But that being said, the field of graph, or you could say knowledge graphs and their role in symbolic systems, they've been around. It's been around for so long. And in the future, there could potentially be symbolic systems that utilize graphs in conjunction with these statistical models that are based on L L Ms and a hybrid sort of symbolic statistical system that combines these tools together. That may not technically be called an agent, but it's still a hybrid sort of AI system in the sense that you might have leveraged the power of an L L M for the language capabilities, but then use symbolic systems

• 51:48Speaker 05

  to do the other tasks. All this is to say that I think graphs are way broader of an entity than what we are seeing today in terms of how they're used. There's a lot of other use cases that we couldn't go into in this discussion. And graph rag obviously is a very, very small part of it. So I would say I'm currently excited about graph rag, but I don't see this being like a topic that is going to be as talked about in the long term. But graphs as an entity and graph databases, these are systems that are going to actually be a core component of many systems of the future.

• 52:20Daniel Whitenack?

  Yeah, it's a good perspective to have because this is such a broad topic. And of course, we'll look forward to you coming back onto the show to give us more of a more of a deep dive and or reading your your blog post. I highly recommend people check out the links to Prashant's blog posts. And also, we'll make sure and include in the show notes links to the things that we've discussed here. But yeah, thank you so much for for joining again. This has been a real pleasure. And I feel like I've got a lot, a lot of things now that I want to go check out and and try hands on and learn more. So so thanks for, yeah, bringing out my my curiosity on the topic.

• 53:04Speaker 05

  Yeah, no, thanks a lot, Daniel. Thanks a lot, Chris. And I just want to end by saying, Kuzo is an embedded database, so you can just pip install Kuzo, and it's super easy to get started. And you can find me on Twitter and LinkedIn. I'm sure Daniel will share the links. And we can always chat more with anyone who's interested about graphs.

• 53:21Daniel Whitenack?

  Yeah, definitely check it out. And we'll also link some of the examples that you mentioned as well, so people can get hands on and try some things. I love this element of these embedded tools out there that both let you try things locally and then even have a pathway to getting those things into production. So also, thank you for building great tools. That's a great, great contribution. So thanks, and we'll talk to you soon.

• 53:51Speaker 05

  Thank you very much. Yeah, it was a pleasure talking to you both.

• 53:57Speaker 00

  Thank you for listening to Practical AI. You know what's cool? Free stickers. During the month of September, we're mailing out ChangeLog sticker packs to everyone who leaves us a thoughtful five-star review or blog post about our pods. Simply email proof of your review to [email protected] alongside your address, and we'll mail out the goods anywhere in the world. Once again, that's [email protected]. Picks or didn't happen only in the month of September. Let's do this. Thanks again to our partners at fly.io, to our beat freak in residence, the one and only Brickmaster Cylinder, and to our longtime sponsors at Sentry.

• 54:41Speaker 00

  Use code CHANGELOG when signing up for a new Sentry team plan and save $100. That's all for now. We'll talk to you again next time.