Data augmentation with LlamaIndex

Large Language Models (LLMs) continue to amaze us with their capabilities. However, the utilization of LLMs in production AI applications requires the integration of private data. Join us as we have a captivating conversation with Jerry Liu from LlamaIndex, where he provides valuable insights into the process of data ingestion, indexing, and query specifically tailored for LLM applications. Delving into the topic, we uncover different query patterns and venture beyond the realm of vector databases. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Jerry Liu – GitHub https://github.com/jerryjliu, X https://x.com/jerryjliu0Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: LlamaIndex Docs https://gpt-index.readthedocs.io/en/latest/LlamaHub https://llamahub.ai/LlamaIndex Blog https://medium.com/llamaindex-blogUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist and founder of a company called Prediction Guard. And I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 0:57Chris Benson?

  Doing very well, enjoying this fine springtime weather of LLMs.

• 1:03Daniel Whitenack?

  Yes, the spring LLM bloom, I guess.

• 1:06Chris Benson?

  That's right. That's right.

• 1:07Daniel Whitenack?

  Well, I don't even think we can use the word bloom because that's loaded now.

• 1:11Chris Benson?

  Yeah, I was gonna say, that has a whole different meaning.

• 1:13Daniel Whitenack?

  There's no word that's not loaded with some sort of AI meaning at this point.

• 1:18Chris Benson?

  Yeah, we should just go straight to our guest.

• 1:20Daniel Whitenack?

  Yeah, including Lamas, which we're excited today to have with us Jerry Liu, who is co-founder and creator of Llama Index. Welcome, Jerry.

• 1:31Speaker 03

  Yeah, thanks, Daniel and Chris, for having me. Super excited to be here.

• 1:35Daniel Whitenack?

  Yeah, I'm really excited because we've had a few conversations in the past, and I've used Llama Index in some of my own work and also kind of tried some integration stuff with various data sources. So I'm really excited to hear a little bit more of the story and kind of the vision behind the project. If I'm just reading from the docs, Llama Index is about connecting LLMs, or large language models, with external data. So maybe a first question, kind of a general question, not specific to Llama Index necessarily, is like, why would one want to connect large language models with external data?

• 2:14Speaker 03

  Yeah, it's a good question. And so for those of you who are already in the space of LLM application development, this might sound obvious to you, but for those of you who might be still somewhat unfamiliar, large language models have a lot of different sorts of capabilities that are really good at answering questions, you know, doing tasks, being able to summarize stuff, basically anything you throw at it, like generate a short story, write a poem, it can do. And the default mode of interacting with a language model like ChatGPT is that you would write stuff to it, you know, in a chat like interface, this query would hit the model and you'd get back some output. I think one of the next questions that people will get into, especially as they're trying to explore building applications on top of large language models,

• 2:55Speaker 03

  is how can this language model understand my own private data, right? Whether you're kind of like a single person or you're an entire organization. And these days, there is like a lot of different ways for actually trying to incorporate new knowledge into a language model. The models themselves are trained on just like a giant corpus of data. And so if you're like an ML researcher, your default mode is just, how can I train this model on more data so that I can try to memorize this knowledge, right? And the algorithm there is basically through some sort of like gradient descent or the weights or RLHF or any sort of like fancy ML algorithm that actually includes the knowledge in the weights of the model itself. I think one interesting thing about large language models these days is that instead

• 3:36Speaker 03

  of like training the model, you can actually take the model as is and just like figure out how to have it reason over new information. And so for instance, like use that input prompt as like the cache space to feed in new information, tell it to reason over that data and to answer questions over that data. And I think that's very interesting because you can take the model itself, which, you know, has been trained on a variety of data, but doesn't necessarily have inherent knowledge about, you know, you as a person or your organization data. But then you can tell it, hey, here are some new data that I have. Now, given this data, how can I answer the following questions? And this is part of the stack that a lot of people are discovering these days, where you can actually just use the language model itself as a pre-trained

• 4:19Speaker 03

  service and then wrap that in this overall software system to incorporate your data with the language model.

• 4:26Daniel Whitenack?

  Cool. Yeah. And your project is called Llama Index. Now, when I, before the past like few months or six months or whatever, when I was thinking about like indices or an index, one of the things that first came to my mind was like, oh, I have a database, maybe. And there's an index that I use to query over that database. And some of that is a little bit like fuzzy magic to me in terms of how that actually works at the lower level in a database. But like, what is this idea of an index or indexing in the context of Llama Index? Or in the context of data augmentation for large language models?

• 5:06Speaker 03

  It's kind of funny. I think when we first started the name, it was a bit more of like a casual naming convention, you know, it used to be called GPT Index. And I kind of made up the name because it sounded roughly relevant to what I was building at the time. But I think over time, especially as it's morphed into more of a project that people are actually using, this concept of index has become a bit more concrete. And so I can articulate that a bit better. The idea of Llama Index is, you know, just to step back and talk about the overall purpose of the project is to make it really easy and powerful and fast and cheap to connect your language models with your own private data. And we have a few constructs to do so within Llama Index. And so part of the way you can think about Llama Index is how can we build some sort of stateful service

• 5:51Speaker 03

  around your private data, around something that at the moment is somewhat stateless? Like the language model call is a stateless service, right? Because you feed in some input and you get back some output. So how can we wrap that in a staple service around your own data sources? So that, you know, if you want to ask a question or tell the LLM to do something, it can reference that state that you have stored. And so if you think about any sort of like data system, there is the raw data that's stored somewhere in some storage system. There might be like indexes or views like similar to like a database analogy where you can kind of look at the data in different ways. And I can talk a little bit about how that works. And then there's usually some sort of like query interface, right? That you can actually query and retrieve the data. So if you look at like a SQL database, right?

• 6:35Speaker 03

  You have, you know, the raw data stored and some sort of tables. You can define different indexes over different columns. And then the query interface is like a SQL interface. You run SQL and then it'll be able to execute the query against your database. And there's a lot of like kind of roughly similar concepts that apply to thinking about the Llama index itself as this toolset. Because if we're going to build this like staple service, right? On top of that can integrate with large language models. By the way, to clarify, we're not like really solving the storage part, right? Where we integrate with like a ton of different vector storage providers. We integrate with like other databases too. But if you even think about us as like some sort of data interface or orchestration, you know, there's the raw data which needs to be stored somewhere.

• 7:17Speaker 03

  And so if you have like a bundle of tax documents, you need to store that in like a vector database or MongoDB or S3, all those types of things. And then you can define these different indexes on top of this data. And the way we think about indexes is how do we structure your data in the right way? So that you can retrieve it later for use without lunch. And so then I can talk a little bit how this works, but the set of like indexes that you can define is actually pretty interesting. Basically, the set of data structures that offers like a view of your data in different ways. And then you wrap that in this overall query interface that can, you know, use these indexes on top of your data to do retrieval and LLM synthesis and give you back a final answer.

• 7:59Speaker 03

  And so I would look at this in terms of like the components of the overall system. There's just like, if you're building this like staple service, there is these three components. How do you, you know, address and store the raw data, index it and then query it?

• 8:12Chris Benson?

  So I want to actually pull you back for just a moment as we're kind of learning this. And if you're an app developer and you're interested in creating a staple service and you've started kind of going down the path about like, well, there's kind of the old school way of going and doing a SQL query and all that. And now we're using LLM models and adding our data to it. I know that we've kind of gone beyond that just a little bit, but if you can back up and talk a little bit about what are you getting? If you're the app developer and you're listening to this and you're trying to understand like, why would I go down that path? I'm, you know, I sense that there's value there, but we haven't talked about it versus a robust set of SQL queries on your own data. Why would you bring in that large language model in the beginning? What is it bringing to bear that's worth all of that effort?

• 8:58Chris Benson?

  Could you talk a little bit about that baseline value add to it?

• 9:02Speaker 03

  Yeah, that's a really good question and I think I might've jumped the gun a little bit, so I appreciate you bringing me back.

• 9:07Chris Benson?

  No worries. It's because you're excited, as are we, but I also want to make sure that people listening have a chance to truly understand it at the same way that you are.

• 9:16Speaker 03

  Definitely. I think one thing about language models that's very powerful is their ability to just comprehend unstructured text and also natural language. And so this matters in both ways in terms of how you can store the data as well as query the data. Because now, you know, let's say you're the end user, you can just type in a natural language like English question, right? Ideally into this interface and get back a response. And so the setup is way easier than having to learn SQL over some source of data. Or, you know, having to even code up like this very complex like pipeline to try to, you know, like parse the data in different ways. Because you could treat the language model itself as a black box, feed it something, get something out, right?

• 9:58Speaker 03

  And so I think that by itself is a very, very powerful tool. And I think these days people are trying to figure out what you can do with that tool. So another kind of like illustrative example of like the power of language models, using this as like intelligent natural language interface, is you actually don't have to do a ton of data parsing when you actually feed in a data. So for instance, let's say you have a PDF document or like any sort of like Microsoft Word document or even an HTML webpage. Just copy and paste that entire thing, right? You know, just extract the text from it, dump it into the input prompt. And then just like tell the LLM, hey, here's just this like giant blob of text I'd copied over. Now, given this text, can you please answer this following question?

• 10:41Speaker 03

  And the crazy thing is the language model can actually do that, assuming, you know, it fits within the prompt space. And that's also very powerful because this kind of affects the way you do like ETL and data pipelining, right? In the traditional sense, if you had a bunch of this like unstructured text, you'd have to spend either manual effort or write a complicated program to pull out the relevant bits from this text. Parse it into some table, store it, and then you'd run SQL or some other query over this text. Whereas here, with the power of language models, you can store this text in a bit more of like a messy, unstructured format as like raw, natural language, and then still figure out a way to pull out this unstructured text. Just dump it into the input prompt and ask a question over it.

• 11:23Chris Benson?

  Is it conceivable with what you're saying, if I'm thinking as an app developer about diving into this, that I'm hearing you say you're going to do this, which is an additional thing to learn and be able to go, you know, it's an additional skill set that you're adding on. But I also hear you talking about other things that I used to have to do that maybe I don't have to do anymore. And to some degree, is it realistic to say, from an effort standpoint, it becomes a wash once you have the skills a little bit, or maybe even you're gaining more power and doing less work along the way to do it so that it's kind of like, of course you would do it going forward? Is that a fair way of thinking about it?

• 11:59Speaker 03

  Yeah, so it's an interesting way of thinking about it because I think the high level question is just like, you know, what parts have become easier and what parts have gotten harder once you have this language model technology? Because on one hand, things have gotten a bit easier and powerful to build these expressive like question answering systems with less effort. You know, you take in this giant blob of unstructured text, you know, figure out how to store it. You feed it into the language model. And then all of a sudden, you can ask these questions over these like files that you couldn't really do before with more kind of like traditional AI technologies or just like manual programming. That said, I think this new paradigm kind of involves its own set of challenges that I'm happy to talk about. I think there is a lot of like stacks emerging about how to make the best use of language models on top of your data.

• 12:44Speaker 03

  And there's some very basic stuff that's happening these days, but there's also kind of like more advanced stuff that we're working on. And I do think it's very interesting to think about what are the technological challenges that are preventing us from unlocking the full capabilities of language models? Because again, with a very basic stack, and again, like you can see this if you just like play around chat GPT, you can already get a ton of value from your data by just like doing some very basic processing on top of it. And you can start asking questions that you couldn't really ask before. But you know, with some more advanced capabilities and some, once you're solving some more interesting technical problems, what are kind of like the additional queries that can ask on top of your data that you also couldn't do before?

• 13:26Daniel Whitenack?

  Before we jump into, so I want to kind of dive into the weeds about like the two things that you talked about, like how do I index my data? How do I query my data? All the goodness around that in Llama Index. Before we do that, maybe just as also like to set the stage for some people that are coming into this and maybe parsing some of the jargon that's thrown around. So one of the other things that people are really kind of diving into is thinking about like, how do I engineer my prompts? How do I chain prompts together and all of that sort of thing? And could you highlight, because my, at least the way I would phrase it, like those two things are complementary with the things that you're doing with Llama Index. But could you kind of help people understand like, how do those pieces fit together

• 14:10Daniel Whitenack?

  in terms of like architecting one of these systems?

• 14:13Speaker 03

  I guess in the end, LLM application development, to put it in a very oversimplified view, is just some fancy form of prompt engineering and prompt chaining, right? It's actually not super different with how we're thinking about building this interface with data. And so just as a very basic example, you know, if you're kind of coming into this space afresh, like a very basic prompt that you could put into a language model is something like the following. Here is my question, right? And then you put the question here and then you put in, here's some context. And then in this context variable, you just dump all the context that could be relevant to the question, right? You copy and paste a blog post, you copy and paste like API documentation, just copy and paste it into the input prompt space. And then now at the bottom say, given this context, give me the answers to this question.

• 14:56Speaker 03

  And you send it to a language model and then you get back an answer. So that's like the most basic like question, answer prompt that you could use to kind of perform some sort of like question answering and over your data. It really is just prompting, right? Because you're putting stuff into the prompts. You have this overall prompt template and you have variables that you want to fill in. I think one interesting kind of like challenge that arises is How can you feed in context that exceeds the prompt window? Because for GPT-3, it's 4,000 tokens. For Anthropic, I guess it's like 100,000 tokens. But if you look at like an Uber SEC 10K filing, it's like 160,000 tokens, right? So if you want to ask a question like, what's the summary of this entire document? Or like, what are the risk factors in this very specific section of the document?

• 15:39Speaker 03

  How do you feed that entire thing in so that you can basically answer the following question? And I think that's where things get a little bit more interesting. Because you can basically do one or more of the following things. One is you could have some external model, like something separate from the language model prompt, that's actually doing retrieval over your data to figure out what exactly is the best context that actually fit within this prompt space. Two is you can do some sort of synthesis strategies to synthesize an answer over a long context, even if that context doesn't actually fit into the prompt. For instance, you could chain repeated LLM calls over sequential chunks of data, then then combine the answers together to actually give you back a final answer. That's one example.

• 16:21Speaker 03

  In the end, all this architecture is just kind of designed around being able to feed in some input to the LLM and get back some output. And the core of that really is a prompt in, right? And so part of this is just like developing an overall system around the prompting.

• 16:48Daniel Whitenack?

  Well, Jerry, you had mentioned kind of these three levels of integrating external data into your LLM applications. There's sort of data ingestion, and there's like indexing and query. I'm assuming data ingestion has to do with like, oh, I'm going to connect to the Google Docs API and like pull the data over and then indexing and query build on top of that. But before we dive into those second two phases, which is where I think a lot of the cool stuff that you're doing is found, what should we know about sort of data in the data ingestion layer in terms of relevance to how Llama Index builds on that and other things?

• 17:29Speaker 03

  The data injection side is just like the entry point to building a language model application on top of your own data. I think LLMs are cool. I want to use it on top of some existing services. What are those services and how can I load in data from those services? One component of Llama Index is this kind of like community-driven hub of data loaders called Llama Hub, where we just offer a variety of different data connectors to a lot of different services. I think we have over like 90 something different data connectors now. And these include like connections to file formats. So for instance, like PDF files, HTML files, like PowerPoints, images even. They can include connectors, APIs like Notion, Slack, Discord, Salesforce. Actually, sorry, we actually don't have Salesforce yet. That's something that we want.

• 18:13Daniel Whitenack?

  It would be cool.

• 18:15Speaker 03

  Yeah, it'd be very useful. If you're interested in contributing a Salesforce loader, please, I would love that. And then the next part is just like being able to connect to kind of like different sorts of multiple formats, like audio, images, which I think I've already mentioned. So the idea here is you have all this data, it's stored in some format, it's unstructured, it could be text or it could even be like images or some other format. How do you just like load in this data in a pretty simple manner and just wrap it with some overall document obstruction? So there's not a ton of tech going on here. And the reason it's more just like a convenience utility for developers to just like easily load in a bunch of data. And again, going back to the earlier point, the reason there's not too much tech is LLMs are very good at reasoning over unstructured information. So you actually don't need to do like a ton of

• 19:00Speaker 03

  parsing on top of this data that you load to basically get some decent results from the language model. And so once you actually load in this data in a lightweight container, you can then use it for some of the downstream tasks like indexing and querying.

• 19:13Daniel Whitenack?

  Awesome. Yeah. And I see like this is, I think, where things get super interesting, like I mentioned. So in Llama Index, I'm in the docs right now, like you mentioned, list and table and tree and vector store and structured store and knowledge graph and empty indices. Could you describe like generally how to think about an index within Llama Index? And then like, why are there multiple of these? And what categories generally do they fit in?

• 19:41Speaker 03

  One way of thinking about this is just taking a step back at a high level. What exactly does the data pipeline look like if you're building an LLM application? So we started with data ingestion where you load in a document from some data source like a PDF document or API. And now you have this unstructured document. The next step typically is you want to chunk up the text into text chunks. So let's say naively, let's say you have just a giant blob of text from a PDF. You can split it, you know, every 4,000 words or so, or every 500 words into some, some set of text chunks. This just allows you to store this text in units that are easier to feed into the language model. And a lot of this is a function of the fact that the language model itself has limited prompt space, right?

• 20:25Speaker 03

  So you want to be able to chunk up a longer document into a set of smaller chunks. Now you have these chunks, they're stored somewhere. They could be stored, for instance, within a vector database, for instance, like a pine cone we via Chroma. They could also be stored, for instance, like a document store like MongoDB, or it could store it like file system on your local disk. Now they're stored. The next part is how do you actually want to define some sort of structure over this data? A basic way of like defining some sort of like structure over this data, and this is where we get into indices, is just like adding and embedding to each chunk. And so if you have a set of texts, how do you define like an embedding for each set of texts? And that in itself could be treated as like an index, right? An index is just like a lightweight view over your data.

• 21:11Speaker 03

  The vector index is just adding and embedding to each piece of text. There's other sorts of indexes that you could define to define this like view over your data. There's a keyword table that we have where you just have a mapping from keywords to the underlying text. You could have like a flat list where you just like basically store a subset of node IDs as like its own index. Before I dive into the technicals of like the indexes and you know, like what they actually do. One thing to maybe think about is just like, what are the end questions that you want to ask? And what are some of the use cases that you'd want to solve?

• 21:43Chris Benson?

  Before you dive into that, I was going to ask you really quick. Could you define what an embedding is for those people who are learning large language models at this point? Just so they'll understand what it is when you say you're defining that as the index.

• 21:55Speaker 03

  Embeddings is a part of kind of a very common stack emerging these days around this LLM data system that's emerging. And so an embedding is just a vector of numbers, usually like floating point numbers. You could have like a hundred of them, a thousand of them. It depends on the specific embedding model. And the way an embedding works is just think about this list of numbers as a condensed representation of the piece of content that you have. You know, if you can somehow in a very abstract manner, take in some piece of context, let's say this paragraph is about the biography of like a famous singer, right? And then you get an embedding from that. It's a string of numbers. The embedding has certain properties such that this string of numbers is closer to other

• 22:38Speaker 03

  numbers that are semantically about like similar content and farther away from other strings of numbers representing texts that are farther away in terms of semantic content. So for instance, like if you look at the biography of a singer, it's going to be pretty close to a biography of another singer versus if it's about, I don't know, like the American Revolution or something like that, embedding will probably be a little bit farther away. And so it's a way of like condensing a piece of text into some vector of numbers that has some mathematical properties where you can measure similarity between, you know, different pieces of content.

• 23:09Daniel Whitenack?

  Maybe this is another point of distinction and I, I get all these questions very often. So I think it's useful to discuss them on the show. Like last week at ODSC, I got a lot of these sorts of questions. So like, we're talking about bringing in data, creating an index to access that data. That index might involve like a vector store or embeddings, but Llama Index is sort of not a, a vector store. Like it's cool to be a, a vector database company right now, but Llama Index is, is something different. And like, again, these are two things that are complementary, I think. Could you draw out that distinction a little bit just to help people kind of formulate those compartments in their mind?

• 23:50Speaker 03

  I think these days there's a lot of vector store providers and they handle a lot of the underlying storage components. And so if you look at like a pine cone or WeV8, they're actually dealing with the storage of these unstructured documents. And one thing that we want to do is leverage these existing kind of like storage systems and expose query interfaces that I guess a broader range of query interfaces beyond the ones that are just like directly offered by a vector store. And so for instance, a vector store will offer a query interface where you can typically query the set of documents with an embedding, plus like a set of metadata filters, plus maybe some additional parameters. But we're really trying to build this broader set of abstractions

• 24:31Speaker 03

  and tools through our indices, our query interfaces, plus like other abstractions that we have under the hood to basically perform more interesting and advanced operations and manage the interaction between your language model and your data and almost be a data orchestrator on top of your existing storage solutions. And so we do see ourselves as separate because we're not trying to build the underlying storage solutions. We're more trying to provide a lot of this advanced query interface capability to the end user using the power of language models on top of your data.

• 25:01Daniel Whitenack?

  I think we got a little bit off track a little bit, but I think it was good. So kind of circling back to the indices that are available in Llama Index, and you've talked about this pipeline of processing and potentially like one index being like a vector store. And maybe listeners are a little bit more familiar with kind of like vector search or semantic search or that sort of thing with everything that's going on. But you have much more than that, like these other patterns and these other indices that enable other patterns. Could you describe like some of those alternatives or additions to like vector store index and when and how they might come into play?

• 25:46Speaker 03

  Yeah, that's a good question. And maybe just to kind of like frame this with a bit of context, I think it's useful to think about certain use cases for each index. So the thing about a vector index or being able to use a vector store is that they're typically well-suited for applications where you want to ask kind of like fact-based questions. And so if you want to ask a question about like specific facts and your knowledge purpose, using a vector store tends to be pretty effective. For instance, let's say your question is, let's say your knowledge purpose is about like American history or something, right? And your question is, hey, like what happened in the year of 1780? That type of question tends to lend well to using a vector store because the way the overall

• 26:27Speaker 03

  system works is you would take this query, you would generate an embedding for the query, you would first do retrieval from this vector store in order to fetch back the most relevant chunks to the query, and then you would put this into the input prompt of the language model. And so the set of retrieved items that you would get would be those that are most semantically similar to your query through embedding distance, right? So again, like going back to embeddings, like the closer different embeddings are between your query and your context, the more relevant that context is, and the farther apart it is, then the less relevant. And so you get back the most relevant context to your query, feed it to a language model, get back an answer. There are other settings where standard top K embedding based lookup, and I can dive into

• 27:09Speaker 03

  this in as much technical depth as you guys would want to, but there's a setting where like standard kind of like top K embedding based retrieval doesn't work well, right? And one example where it doesn't typically work well, and this is a very basic example, is if you just want to get a summary of like an entire document or an entire set of documents. Let's say, you know, instead of like asking a question about a specific fact, like what happened in like, you know, 1776, maybe you just want to ask the language model, could you just give me an entire like summary of American history in like the 1800s? That type of question tends to not lend well to embedding based lookup because you typically fix like a top K value when you do embedding based lookup, and you would get back very specific context. But sometimes you really want the language model to go through all the different contexts

• 27:54Speaker 03

  within your data. So a vector index, storing it with embeddings, would create a query interface where you can only fetch like the K most relevant nodes. If you store it, for instance, with like a list index, you could store the items in a way such that it's just like a flat list, right? So when you like query this list index, you actually get back all the relevant items within this list, and then you'd feed it to our synthesis module to synthesize a final answer. So the way you do retrieval over different indices actually depends on the nature of these indices. Another, just like a very basic example, is that we also have like a keyword table index where you can kind of like look up specific items. By keywords, right, instead of through like embedding based distance.

• 28:36Speaker 03

  Keywords, for instance, are typically good for stuff that requires high precision and a little bit like lower recall. So you really want to fetch like specific items that match exactly to the keywords. This has the advantage of actually allowing you to retrieve a bit more precise context than something that like vector based embedding lookup doesn't. The way I think about this is like, a lot of what Llama Index wants to provide is this overall query interface of your data. Given any class of queries that you might want to ask, whether it's like a fact based question, whether it's a summary question, or whether it's, you know, some more interesting questions, we want to provide the toolset so that you can answer those questions.

• 29:16Speaker 03

  And indices like defining like the right structure of your data is just one step of this overall process and helping us achieve this vision of like a very like journalizable query interface over your data. Some examples of like different types of queries that we support. There's the fact based question lookup, which is like semantic search using vector embeddings. You can ask like summarization questions through, you know, using our list index. You could actually run like a structured query. So you could, if you have a SQL database, you could actually run like structured analytics of your database and do like text to SQL. You can do like compare contrast type queries where you can actually look at different documents within your collection and then like look at the differences between them.

• 29:57Speaker 03

  You could even look at like temporal queries where you can reason about like time and then go forwards and backwards. And basically kind of like say, hey, this event actually happened after this event. Here is the right answer to this question that you're asking about. And so a lot of what Llama Index does provide is a set of like tools, the indices, the data interesters, query interface to solve like any of these queries that you might want to answer.

• 30:40Chris Benson?

  So Jerry, you really got me thinking a lot about this. The possibilities of the query schemes is pretty darn cool. You know, we kind of started with ingest and moved into kind of indexing, and now we're talking about queries. Could you kind of give me an example with the tool at a little bit more of a practical level? Because you kind of hit the concepts about like what the possibilities are. But as someone who hasn't used the tool myself, I'm trying to get a sense of what that workflow is like. Pick what would probably be like a really common query scheme that you're doing and dive into that just a little bit to give us a sense of a hands-on, practical, you know, fingers-on-keyboard sense of it. Because I'm trying to get a sense of where I'm going to go for playing after we get done with the episode. So I want to try it.

• 31:24Speaker 03

  100%. I think one thing that has popped up pretty extensively after talking to a variety of different users is actually financial analysis. I think looking at SEC 10Ks tends to be a pretty popular example. If you look at the Anthropic Cloud example, they also use SEC 10Ks. And my guess, the reason it's popular is, one, there's just like a ton of text, and so it's just very hard to parse if you read it as a human. Two, it's like a useful thing for people in like financial institutions like consulting, because you want to like compare and contrast the performance of like different businesses, you know, and look at the performance across years.

• 31:58Chris Benson?

  Believe it or not, I actually read 10Ks a lot, and that would be a really useful example for me. Believe it or not, I'm not kidding you.

• 32:06Speaker 03

  As a result, we've actually been playing around with a decent amount, too.

• 32:09Chris Benson?

  Yeah.

• 32:09Speaker 03

  Some of the cool like things that we've been, that we're showing that Loma Index can do on top of your 10Ks is, for instance, let's say you have like two companies, let's say Uber and Lyft, right, for the year 2021. You can actually ask a question like, Can you actually compare and contrast the risk factors for Uber and Lyft or their quarterly earnings across like these two documents? And one is the Uber 10K, one is the Lyft 10K. This is actually an example where if you do like just top K embedding based lookup, the query fails because if you ask the question, you know, comparing and contrast Uber and Lyft, I don't do anything to it. And let's say, you know, your Uber and Lyft documents are just in some one vector index, you don't really have a guarantee you're going to fetch the relevant context to this question that to be able to answer this thoroughly, right? And then the

• 32:54Speaker 03

  model might hallucinate, you'll get back the wrong answer. And then, you know, this is not a good experience. I think what you typically want to do is have some sort of like nicer abstraction layer on top of this query that can actually kind of map that query to some plan that that would roughly be like how a human would think about like executing or answering this question. Let's say you want to compare and contrast the financial performance of Uber and Lyft in the year 2021. Well, first, okay, what was the financial performance of Uber in 2021? What was the financial performance of Lyft? You break it down into those two questions. And then for each question, use it to kind of like look over your respective index. Let's say you have an index corresponding to Uber and an index corresponding to Lyft. Get back the answer,

• 33:37Speaker 03

  right? Get back the actual kind of like revenue, for instance, for Uber and Lyft, and then like synthesize both of them at the top level again. Be able to like pull in the individual components you extracted from each document and then synthesize the final response as able to compare the two. So that's like an example of something that we can actually do pretty well with WAMA index. And we have like a variety of toolsets for allowing to do that. And that's an example of a query that's kind of more advanced because it requires comparisons beyond just like kind of asking stuff over a single document. Another example, just to kind of like take the 10K analogy further is, let's say you have the yearly reports of the same company across different years, let's say from like 2018 to 2022. You can ask the question like,

• 34:19Speaker 03

  did revenue like decline, go up or down in the last like three years? And then you could actually do a very similar process where given the query interface that we provide, Break this question down into some questions over each year, pull out the revenue, and then basically at the end, do a comparison step to see whether or not it increased or declined.

• 34:39Chris Benson?

  Just as an aside to any listeners out there wondering why on earth somebody would read 10Ks, especially considering that our audience is focused on data and such as that. If you want to learn about another technology company and really understand what it does and be able to compare it, this is an example where you can gain tremendous intelligence on another company with publicly available information. And by comparing multi-year 10Ks, like you just said, you'll learn way more about that company than its own employees know about it. So anyway, just thought I'd mention that as an aside.

• 35:10Daniel Whitenack?

  Yeah, I look forward to hearing your success with speeding up your workflows around reading the 10Ks, Chris, with Llama Index.

• 35:18Chris Benson?

  I'm excited about this. This is going to save me a lot of time.

• 35:21Daniel Whitenack?

  Jerry, I, one of the things that we talked a little bit about in one of our previous conversations, which I know you've also thought very deeply about and even have like a portion of the docs and functionality in Llama Index devoted to is evaluation, like query response evaluation. Like, how do I know my large language model like barfed up an answer, right? Based on some query and I pulled in some external data and I, you know, inserted some context and maybe I strung a few things together, like how am I to evaluate the output of that? Could you give us maybe a high level from your perspective, how you think about this evaluation problem?

• 36:06Daniel Whitenack?

  And then maybe go into a little bit of some of the things that you're exploring in that space?

• 36:10Speaker 03

  Yeah, totally. Just to preface this, like, we are super interested in evaluation or more tailored towards this interface of like your data with LLMs. I didn't dive into that a bit more. And we have some initial evaluation capabilities, but we're super like community oriented. Like, we'd love to just like, you know, kind of like chat with, there's a lot of like different toolsets out there that allow you to do like different types of evals of your data and building like nice interfaces for doing so. And so I think this is an area of active exploration and interest for us as well. And so just kind of thinking about this a little bit more deeply about. Evaluation is very interesting because there is the evaluation of each language model call itself, and then there is the evaluation of the overall system.

• 36:52Speaker 03

  And so diving into this bit more, like at a very basic level, if you have a language model, you have an input and you get back some output. You can try to validate whether or not that output is correct, right? Given a single language model call. Did the model actually give you the correct answer given the input? Did it spit out garbage? Did it like hallucinate that type of thing? The interesting thing about a lot of systems that are emerging these days is that they're really like systems around like a repeated sequence of language model calls. And this applies whether or not you're dealing with like a more agent based framework, which you know, you ask a question that can like just repeatedly kind of like do react chain of thought prompting or like be able to pick a tool. But the end result is is able to give you back a response. Another example, this like AutoGPT where you just let it run for like five minutes

• 37:35Speaker 03

  and just like keeps on doing stuff like over and over again, until it gives you back something. Or, you know, even in the case of retrieval augmented generation is just like a fancy name for roughly like what we're doing with Llama Index, which is like a query interface of your data. Like even with within our system, there could be a sequence of repeated LLM calls. But the end result is that you send in some input into the system and you get back some output. You know, given this high level system, how do you evaluate the input and output properly? I think in traditional machine learning, typically what you want to have is you want to have like ground truth labels for every input that you send in. So if you like, for instance, ask a question, you want to know the ground truth answer and you want to compare the predicted answer to the ground truth answer and see how well the predicted answer matches up. This is still like something that people are exploring these days, even in the space

• 38:20Speaker 03

  of Grown AI and LLMs, you have ground truth like text and then you have predicted text and you want some way of scoring how close this predicted text is to a ground truth text. I think the core set of eval modules that we have within Llama Index actually are ground truth free or label free. And that part in itself is actually very interesting because you have this input, you ask a question, you get back this predicted response. You also get back the retrieve like sources, like the documents themselves. What we found is that you can actually make another LLM call to just like Compare the sources against the response and then also compare the query against the sources and the response to see how well all two or three of these components match up.

• 39:01Speaker 03

  And this doesn't require you to actually specify what the ground truth answer is. You just look at the predictive answers, see if it matches up to the query or the context in a separate LLM call. And it's interesting because, one, it makes use of LLM-based evaluation, which is kind of like an interesting way to think about it, basically using the language model to evaluate itself, right? I'm sure there's like downsides which we can get into, but you know, a lot of people are doing it these days too. And then the second part is it doesn't require any ground truth because you're using the language model to evaluate the capabilities of its own answer plus context. You don't actually need to, as a human, feed in the actual answer. And the benefit of this is that it just saves a bunch of time and cost. You don't actually need to like label your entire dataset to run evals.

• 39:43Speaker 03

  I still think this overall like space is probably relatively early. I think there's still some big questions around like latency and cost. If you're trying to really do LLM-based evals more fully, like using the LLM to evaluate a large dataset takes a lot of time and costs a lot of money. And so this is generally kind of like an area that we're still kind of like actively thinking about.

• 40:04Daniel Whitenack?

  Yeah, that's awesome. As we kind of like get nearer to the end here, I know things are like progressing so quickly. I can't keep up with all of your tweet threads about like awesome new stuff that's happening in Llama Index. But I know there's a lot, as you look to kind of the next year and like where your mind is at, what you really want to dive into? And also what what's really exciting to you about the field? There's a lot of people excited about a lot of different things, but from your perspective, having been in the trenches, building large language model applications, interacting with users of Llama Index, what is it that really excites you moving into this next year in terms of the possibilities?

• 40:46Daniel Whitenack?

  The real practical possibilities on the horizon and how kind of our development workflows will be changing in the near future?

• 40:53Speaker 03

  Yeah, totally. I think there's a few related components to this I'm both excited for as well as like, you know, the challenges that we're going to solve. Probably the first component is just being able to build this automated query interface over your data. You know, if you're looking at all the query use cases that we solve, one of the key questions that we keep going back to is here's a new use case on top of this data, and here's a new question that you'd want to ask. How do we like figure out how to best like fulfill a query request, right? And I think especially as your data sources become more complicated, then it's just like, how do you think about like how to index and structure the data properly? How do you think about interesting automated interactions that happen at the query layer between the language model and your data? How do you like basically just like make sure we solve

• 41:38Speaker 03

  this request? And then the second component to this is we want to make sure that we build this interface that can handle any type of query you throw at it. Now, how do we do this in a way that is like cheap, fast, and easy to use? For a lot of users, once they move on beyond the initial prototyping phase, they think about starting to minimize like cost and latency, and they think about how do you like pick the best models for the job, right? There's like the OpenAI API, which works well, generally, you know, they're probably the best model out there, but it can also be quite slow. And then there's also these like open source LLMs popping up, probably like a few new ones every week. And then how do users make the best decisions whether to use that over their data? And then I think the next part to this is, you know, a lot of LLM development is

• 42:19Speaker 03

  moving in this overall trend of like automated reasoning, right? If you look at like agents and tools, it's and AutoGPT and all this stuff. It's just like, how do you make like automated decisions over your data? And then I think as a consequence, there's always going to be this trade off between how few constraints can we give it and how many like, like, should we give it more constraints or fewer constraints, right? Because if your constraints you give to something like this, it has more flexibility to potentially do way more things, but then also just like error, right? Like it'll just like make mistakes and then there's no really no way to correct it easily. And then you can't really trust the decisions. Whereas if you kind of like constrain the outputs of these like automated decision makers or agents, then you can potentially get more interpretable

• 43:04Speaker 03

  outputs, maybe at the cost of like a little bit of flexibility in terms of functionality. And I think we've been thinking a lot about that with respect to like the data retrieval and synthesis space too, right? Like, how can we give you back results that are expressive, but also, you know, like perform well and are going to make mistakes like a ton of time?

• 43:22Daniel Whitenack?

  Awesome. Yeah, well, I'm really happy that we had a chance to talk through all the great Lama Index things. And make sure if you're not following Jerry on Twitter, find him there. He posts a lot of great stuff that they're working on. And of course, you can find Lama Index if you just search for Lama Index. There's like a great set of docs and all those things. We'll make sure we include those links in our show notes for people to find out about that and get linked to their docs and their blog and all the good things. So check out Lama Index and thank you so much for joining us, Jerry. It's been awesome.

• 44:04Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.