Ep 18: LlamaIndex CEO Jerry Liu on Trends in LLM Applications

Jacob and Pat sit down with LlamaIndex CEO Jerry Liu to discuss his motivations for building LlamaIndex, thoughts on building enterprise-ready LLM applications and agents, and when fine-tuning makes sense.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast. I'm Jacob Effron and I'm joined by my co-host, Pat Chase. And today, we had a fascinating conversation with Llama Index CEO Jerry Liu. Llama Index is an open source data framework that connects data sources to large language models. It's a really popular open source project and a key part of the retrieval algorithm generation stack. We touched on retrieval algorithm, the generation versus fine tuning, and one each makes sense.

• 0:20Speaker 01

  If you want to get better retrieval, you can try fine tuning the embedding model to basically get better performance over that domain of data that you have.

• 0:27Jacob Effron?

  We talked about ways to improve information retrieval.

• 0:29Speaker 01

  It was found out on this key concept called recursive retrieval to allow users to model retrieval and retrieve documents by their summaries first, and then go into a document and retrieve different chunks within it.

• 0:40Jacob Effron?

  We hit on how do you get agents into production?

• 0:41Speaker 01

  I think as these companies move beyond productionizing chatbots and start actually implementing some best practices for LLM development, then they'll gradually start to build up again to solve more advanced use cases and then include agent-like capabilities into their apps.

• 0:55Jacob Effron?

  I think folks are really going to enjoy this.

• 0:58Jacob Effron?

  Jerry, thanks so much for coming on the podcast.

• 1:00Speaker 01

  Thanks, Jacob. Thanks for having me.

• 1:01Jacob Effron?

  Awesome. Well, to kick things off, I'd be curious just to hear a little bit about your initial inspiration and vision for Llama, and then kind of your story on how the product's evolved since you got started.

• 1:10Speaker 01

  Yeah, for sure. I think I've said this during like a few of these talks before, but at a high level, you know, when I was first starting out playing around with GPT, one of the first things I was trying to do was figure out, how do I just feed in my own data into ChatGPT? Because it was clear to me that, or not ChatGPT, ChatGPT wasn't out at that point, it was GPT-3. But I was playing around with DaVinci, and it was clear that, you know, it was great at reasoning and starting to actually, you know, almost like make decisions and synthesize insights just based on whatever you give it in the input prompt. And so the question was, how can I basically just like create kind of some abstraction to allow me to feed in like arbitrary amounts of data into GPT without actually training the model? And so the project really started off as a design project.

• 1:54Speaker 01

  You might have known this as like GPT Index in the beginning days. And like, to be honest, it didn't actually work super well. It was more just like a, like, piece of thought leadership, like, just like, hey, you know, I didn't actually use embeddings in the initial iteration. It was more just an exercise of like, what if the LLM itself could just like organize knowledge and do like LLM-based retrieval and synthesis? Like, just, you know, organize information by hierarchically summarizing it. And then actually use the language model itself to traverse this like index that you just built and try to actually figure out the right piece of information. So I mean, I think that did strike a chord. I think a lot of people were interested in that type of problem. I mean, like, I'd readily admit, like, it turns out embedding is where a lot more practical in terms of actually just doing retrieval from like

• 2:37Speaker 01

  massive volumes of data. But like, you know, the initial inspiration I think did kind of strike a chord because it was kind of like thinking, you know, how do we not just use existing retrieval methods, but actually kind of go beyond that as well into something that's just more natively LLM based. Anyhow, I think it's evolved like a lot since the beginning days into this overall framework for people to help build like LLM apps with their data. I would say that the mission statement hasn't really changed though since the beginning. The mission statement has always been around, how do you basically just unlock LLM capabilities over your own data? And it was something that, you know, motivated the initial iteration of this project, but it still motivates us today. And I know this kind of maybe gets into some of the other stuff we're going to talk about,

• 3:22Speaker 01

  but like, yeah, if we think about stuff like RAG, like retrieval augmented generation, all the way to even stuff like fine tuning. I mean, I think all of those are kind of interesting concepts and they all like fall within this overall mission, right? Because it's, it's just like, there's some open questions on how do you best do like RAG or fine tuning to memorize knowledge, reason over it, and then unlock insights. And that's something that, you know, we're also very interested in exploring.

• 3:46Speaker 02

  I think of Llama Index as one of the first tools that someone would go to when they really want to get LLMs into production. What have you seen from where developers are today? Like, are they, it seems like there's a lot of people that are, you know, building and shipping things really quickly, but also prototyping really, really quickly. You know, are people in production today? Or like, what have you seen from just, you know, the apps that are being built on top of this LLM stack?

• 4:10Speaker 01

  Yeah, yeah, people have built apps, you know, using Llama Index, other frameworks, and they've started to build it in production. I would say the ones I've seen in production are more chatbots, like knowledge synthesis, and then also maybe like structured data extraction. I think those are pretty popular use cases. And we've actually seen them either deployed both like to internal users or actually externally facing to. I think stuff that's more complex, so stuff that's more like, like, like complex in terms of like reasoning, in terms of interactivity, so stuff around agents, basically. The fancier stuff around agents, we've seen less of in production.

• 4:51Speaker 01

  So you typically don't see as much, like many agents just operating on like a full React loop over like 20 different tools in production, or at least I haven't as well. And I think the reason for that is just like reliability. I think it's easier for people to justify inserting LLMs as like one call in this overall stack that will like, you know, provide some purpose in a more constrained fashion, but maybe harder for people to justify, how do you actually, you know, evaluate like a full agent loop and then make sure that this thing doesn't just break like half the time. So I think that's something that, I think people, for what it's worth, will probably, the way I think about agents is like, like for, like, there was a period for like two or three months where agents were super popular.

• 5:33Speaker 01

  There was like, you know, like the React paper, open AI function calling, like, or even before that, like the, like agent simulation stuff, like Darn of Agents, like Voyager, and then also AutoGPT, like, it was just, but I think like the hype around that has probably like died down a little bit. But I think as these companies move beyond productionizing chatbots and start actually like implementing some best practices for LLM development, then they'll probably gradually start to build up again towards like, hey, how can I like try to solve more advanced use cases and then include like agent-like capabilities into their apps. And so even something where you can imagine like a chatbot interface moving slightly

• 6:15Speaker 01

  beyond like a pure chat experience to, oh, like click a button and then this will actually go and, you know, send an email for you, right? Or kind of go and, you know, like perform some action on the web. That's something that I think I've talked to some users that are exploring this, but we'll, we'll see in production more and more.

• 6:33Jacob Effron?

  That was interesting. I feel like we've had a full hype cycle with agents in like three months. And so, I mean, obviously you guys have, you know, recently released some interesting features around agents. And I'm curious kind of like the learnings you guys have had around, you know, how to make them more effective, where they do, don't work. I mean, I know you've talked before about like, you know, the importance of hierarchical agents versus flat agents, but kind of, you know, what you've been seeing on the ground of just You know, learning this would be super interesting.

• 6:58Speaker 01

  Yeah, I think from our experience, one aspect is being able to effectively like define the system prompt for each agent. It matters a lot. So making it operate in a more constrained setting. So for instance, for like a sequel agent that can query a sequel database, one, that's pretty constrained. Two, you can actually just preload the system prompt to say like, hey, here is like the specific set of tables you're allowed to query over. And then that, instead of having it do like three different things at once, it only does this, you only give it access. Wonderful. It does a lot better in terms of actually being able to kind of like craft queries to execute like sequel statements. So generally like careful definition of like the system prompt. Careful like kind of input representations of like the tool.

• 7:42Speaker 01

  So really like prompt engineering, the doc string for agents to use it in like a structured manner. Giving the agent some, like I think tool design is actually very important for designing reliable agents. So for instance, like if your tool requires like an API interface with like not just a natural language string, but a few other parameters, if you like, if you like basically force the agent to specify all exactly right in order for the tool to actually execute, then the tool will probably like, there's a good chance it will fail. Because you're not really kind of baking in some room for error. So one thing we found that's pretty interesting is like, let's just say you made all the parameters optional.

• 8:24Speaker 01

  And so if the agent actually didn't infer certain parameters, that's fine. We can use like a default value. And if the agent actually use like invalid parameter, you can throw an exception, but not actually like encode, you return that as like a prompt message to the agent. Say like, hey, you know, like this thing you cannot pass like this because you know, like it's, it cannot be less than zero. Right? Like that, that's an example that you return as a prompt back to the agent. And then the agent can go reason about it and actually return you something new. So like kind of baking in like proper tool design practices. It's kind of like a new way of like defining API interfaces, which is super interesting. But like more specifically for LLM use, it's something that we found helps.

• 9:09Speaker 01

  Those are just like a few practical tips. I think in general, these are things we're still like generally exploring. I do think like, like the amount that you'll need to prompt engineer our agents will probably depend on these capabilities, the model capabilities, as well as like how much we can just like fine tune these models to be better at like adrian tech reasoning. But I think that will probably get better over time. But in the meantime, these are some like practical tips.

• 9:36Speaker 02

  One thing I'm really curious to get your, your take on, because I think of, you know, Llama Index is really a cornerstone of the LLM stack. And I'm curious how you see that, that stack evolving. It seems like it started with OpenAI and people were like, okay, I'm just going to write Python on top of OpenAI. And then there's started to become these emerging patterns around like Llama Index, Vector Store, Lane Chain. How do you see the, the stack evolving?

• 10:00Speaker 01

  For us, like really the key categories of Llama Index is two main components. One is this new way of defining like ETL orchestration for any sort of L and maps if you're a data. So one is doing like extraction from a data source, being able to like transform it and the certain ways to basically load it into like a vector storage system. And it's obviously different than existing ETL, right? Like you're kind of like five trend, air byte and DBTs of the world where you like are mostly using that for like structured analytics purposes. Like here, you're loading in like a bunch of PDFs, like slicing them up, like adding metadata, tagging, doing a bunch of transformations and then like maybe storing it in like a vector DB.

• 10:43Speaker 01

  And so this, you know, has become kind of like this pipeline for building RAG. So I think like in general, we are investing a lot more efforts towards really making that part of the pipeline robust. And even for like the current, like, you know, like, like who knows in the future, maybe fine tuning will overtake RAG, but like assuming RAG has like, like its place at least for the next year or so. There's a lot of value in investing in kind of better techniques and also practices for letting users build more robust data pipelines. I think people generally, when they build these apps, don't really think about what is like good parsing, what is like good ways of like splitting your text? How do you actually properly like add metadata for good retrieval?

• 11:26Speaker 01

  And I think it is part of our mission to try to like really make that stack robust and give people the tools to define like proper data pipelines. The next step here is now that like, you know, some of your data is kind of in the relevant storage systems and defined the right way, then how do you kind of like have that interplay between LLMs and retrieval and agent, like, yeah, like LLMs and your data for like a good kind of synthesis and agent algorithms. That's something we've invested a decent amount of time in in the past few months in terms of like advanced retrieval using LLMs. How do you basically invest in all the like data agent stuff? And that's something we'll probably invest more time in as well. Just like, how do you basically do better retrieval?

• 12:11Speaker 01

  And how do you like interleave LLM reasoning with retrieval over your data? And so I think like those will probably be some of the key things that we'll try to focus on in like defining this this stack. And then the way I see fine tuning, at least in this current stack, is optimizing like different bits of this too. Like, if you just imagine this as like a software pipeline, how do you use fine tuning to like optimize like the parts that are parameterized, like the embeddings, and then also the language model itself? And this is probably going to be like the core part of our roadmap, at least in the next like six months or so. And then, you know, as the space evolves, like, who knows, like OpenAI will probably release some like multimodal model. There will probably be like advancements in like fine tuning and just like better open source models. And

• 12:54Speaker 01

  we'll try to like account for that as like a side thing too.

• 12:57Jacob Effron?

  Would love to get your thoughts on like the state of retrieval augmented generation today. You know, both kind of the use cases that you're most commonly seeing. And, you know, I think something that's so powerful about Llama indexes, you obviously have the ability to do more complex implementations. I think a lot of people start out with just kind of like basic indexing and like a top K embedding lookup. And so it'd be helpful, you know, both the use cases that are out there and then where you're seeing kind of the the basic approach maybe fall short today for some of these use cases.

• 13:22Speaker 01

  Any sort of files that you have, like chats, like conversations, a lot of people index like Slack, like Notion, or like Salesforce, even like YouTube videos. I mean, I think in the end, this is basically just like knowledge extraction and synthesis, whether or not it's like in the form of a search bar or a conversational format. I think the challenges come both when you think about like iterating on performance and also like scale and and kind of like other aspects of production writing this. As you as you said, like the the basic stack right now is basically people just do like top K embedding lookup. And and and the thing is that it's not even just that. It's like the the like initial decisions they make on the data is they just like split it up into a bunch of like text chunks and throw it into vector database, right?

• 14:07Speaker 01

  And you know, like I like it might just be due to the simplicity of abstractions like like llama index, where you can do stuff like that in three lines of code. But we do stuff like super like the defaults are are pretty naive these days where you just, you know, maybe you just separate on sentences. You don't really take into account like the domain. You don't really inject like metadata or anything like that. And so as a result, like when you add all these pieces of text to like a vector database and then you do like top K retrieval, there's gonna be a bunch of cases where like you're not really retrieving the most relevant context to actually answer the set of questions that you want to have. And we kind of realized this early on, right? Like even back in kind of like March or April, we were kind of thinking, yeah, I mean, like

• 14:48Speaker 01

  just this existing stack of just like doing top K embedding lookup seems pretty limiting in terms of like really replicating the experience that you can get of like chat GPT over your data. Because if you think about just like taking a step back, like the types of stuff you can ask over chat GPT, it's way more than just like simple like question answering over like specific facts. You can ask it to like synthesize stuff from like disparate data sources, reason, stuff where you teach you things, right? Like what write poems, all these things over like any source of data that's trained on. Like, how do we just allow like rag stacks to handle like more complex queries like that? So I think like one interesting development that I think we've worked on for a few months now is this idea of like, how do you just like deal with better retrieval over larger

• 15:33Speaker 01

  numbers of documents that, you know, might be spread out across like different document sources? So if you have like a bunch of different documents, like maybe there are SEC filings or there are like a bunch of research papers and you ask questions, how do you like properly, you know, like index these documents so that you can kind of make sure you fetch the relevant context dynamically, depending on the context you want to ask? And so that's kind of what motivated some of the approaches, right? And I have like an entire slide deck going into this. And so, you know, that by itself would probably take like 30 minutes.

• 16:04Jacob Effron?

  Well, definitely linked to that, it's quite good.

• 16:06Speaker 01

  You know, there's like some basic things from like simple to kind of more advanced that like it's not just like us. We've like talked to users during this too, that people are playing with for better retrieval. One idea here is just like defining metadata, like structured filters on top of your data. To kind of add that element of like structured filtering along with embedding search. Another aspect here is basically kind of organizing information in like a hierarchy. Where you can like kind of maybe embed documents at like a summary level and then fetch the documents. And then you go in and actually retrieve stuff within the document. A lot of this stuff tends to have like, enable you to fetch like more relevant context compared to just like standard.

• 16:48Jacob Effron?

  I'm curious, like as you and your team kind of keep focusing on where to build, like, you know, how do you think about what the right level of abstraction is for developers? I mean, obviously you've got some folks that want to just, you know, like you said, in three lines of code spin up like the most basic, you know, rag stack out there. And then other folks are super in the weeds of kind of cost optimizations and latency and whatnot. Like, you know, how do you think about what the right abstraction level is for you guys?

• 17:10Speaker 01

  Yeah, I think, actually, I think one of the principles of our library is that we try to like cater to every level. So that like to start with, yeah, like to build a rag, it is like three lines of code. And you don't really need to see what's going on under the hood if you don't want to. But this is something that's like under active development and we made like a ton of progress in this library in the past few months. Where like if you like whatever piece that you want to customize, whether it's like the retrieval algorithm. Or you want to write like your own like L1 synthesis code, or you want to write some like Adrian loop at the top. Like, we encourage you to basically plug and play like different components, whether or not it's like other building blocks or your own custom models as well.

• 17:51Speaker 01

  And we've been trying to make a push towards this aspect of like, almost like a lower level APIs and bottoms up development. And it's been like a key design principle in the library that we want to push towards, which is for a lot of people, actually. Really one of their, if you look at like, you know, people on Hacker News complaining about stuff like want my index line train in the past few months, like one of the key complaints is they don't know how to customize things. And, and so especially as you get more advanced, like you don't want them to just rip out the framework, right? You want the still like have modules that provide value, but still allow them to uh kind of customize the key aspects that they want. Um, and so for us, like, um, we call this like bottoms up development, like just like, uh, really like encouraging users to not just look at the quick start in our docs,

• 18:35Speaker 01

  but really going into each module and kind of playing around with it on their own to create these like kind of custom subclasses of, of abstractions like retrievers, uh, like agents, uh, like query engines or rag pipelines. And it's something that's still under, uh, uh, active element, but like I, I do think, um, for what it's worth, like I, I've talked to a decent number of users where, uh, uh, like a lot of them building kind of, um, like AI applications these days, especially if they're like, if they've been kind of hacking on chat GPT since the beginning. Yeah, they, they probably wouldn't use that out of the box retrieval abstractions because they probably like hack their own stuff. Um, and it's good, right? Like it's, it's good because like, you know, they're relatively advanced in this space and they tend to know what they're doing.

• 19:20Speaker 01

  But like for any level of these users, there's probably some level of boilerplate that like they just wouldn't want to write themselves. And it's nice to kind of have some base level of abstraction that will handle certain things. This includes stuff like callbacks, like debug handling. Um, this includes stuff like just being able to plug and play like different vector source, like LM abstractions, or even like, you know, kind of like retriever modules too.

• 19:41Speaker 02

  When we had Mateo Zaharia, um, on the podcast, one of the things that he was talking about was like really this vision for separating facts from reasoning. Um, and that people would have kind of a traditional index for facts and then, you know, be either using a model in the cloud, um, for, for doing more of the reasoning and. It seemed like one of his hypotheses at the time was that the models would be getting smaller, kind of given that that architecture, because you'd be, you know, reasoning over kind of exactly what you tell it. Have you seen any trends around the, you know, the setup of like indexing plus generation? Or any, anything that that is kind of top of mind for you there?

• 20:24Speaker 01

  I think if you're talking about like this idea, this balance between like reasoning and knowledge, I think that's something that is interesting. Yeah, I think there is one world where, yeah, the models get smaller and they get better at kind of like reasoning over certain tasks. But then like in terms of actually feeding in facts, it really depends on like actual indexes that you feed to the language model itself. That definitely is something that we could see happening, especially as you have like LLMs operate in more like specialized roles. I think there is another world where we just see, and this is kind of like a hot take of mine, where we just see the LLMs themselves becoming just general like knowledge engines.

• 21:03Speaker 01

  If you can just like fine tune, I think I was talking to like Shashir, who's the author of like Gorilla, right? Like, I think if you just like fine tune the language model properly over like a bunch of data, you can actually just get it and just memorize a bunch of stuff. It's basically just like a compressed representation of of knowledge.

• 21:22Speaker 01

  And but that would probably necessitate like larger language models. And so that's a counter to this idea of like models getting smaller over time. I think there is something nice from like an at all perspective from that, just because it's kind of like you're just like treating this as like a giant like black box optimization problem, where you're just like baking in a bunch of knowledge into the model itself.

• 21:48Speaker 01

  But in terms of like feasibility, practicality, like, I don't know. But it's something that I think, you know, like people are probably generally interested in, especially as like, you know, fine tuning, like just like this idea of like even like continuous training or something gets better. And so I think there are maybe like two, two worlds there. I think for me, I'm not actually sure. I think practically for a lot of enterprises, what will probably happen is you like start off doing these like large models for prototyping and then you start like fine tuning and making it more specific and smaller to help like optimize for class and performance and other things as you go on the production pathway.

• 22:27Speaker 01

  So that is an argument for maybe the former world. I think the argument for the latter world is it's just kind of like there's something nice about it. I mean, there is like a reason large language models exist, right? It's because like when you just open up chat GPT, it's not just like a random ML model. You can just pretty much ask it anything that it knows about and it'll be able to answer it.

• 22:46Jacob Effron?

  Yeah, it's a super interesting point. I mean, I guess in favor of maybe like a more rag approach over a fine tuning. You do have like the audit ability, right? And the ability to trace exactly back where, you know, where a piece of information is from. So I'm curious kind of what the mix ends up looking like. And then relevant to you guys, I mean, obviously I think a lot of the growth in in Lama index has been around, you know, rag. It feels like you guys have talked about doing stuff on the fine tuning side, like, how do you think about how like fine tuning workflows fit into, you know, the Lama index future?

• 23:12Speaker 01

  Yeah, I mean, I think short term, like fine tuning, we're already working on on on stuff around fine tuning in terms of optimizing existing rag pipelines. Obviously, Lama index for most of the development has been very focused on on rag. And I think there's a good reason for that. It's just like in terms of the UX for developers, it's just way easier for people to get something set up and running over their data by just using these models purely in inference mode as opposed to actually training them. Once you train models, you do open up a whole can of worms because then you have to worry about like, oh, how do you properly version this? How do you measure like performance aggravations, all these different things? Whereas if you can fix that, then you eliminate some area of complexity, right? And you can just focus on like prompt tuning and the data itself.

• 23:53Speaker 01

  So there, yeah, so there's like a reason why we're focusing on rag right now. I think in terms of like short term, like we see fine tuning is optimizing different pieces of the stack. So being able to optimize like the embedding model or optimizing the language model for like better reasoning. To include like, you know, being able to either fine tune or like distill a model for better like synthesis and reasoning. If like your data is already stored in like an external index. I do think, though, that like, kind of, in the long term, I mean, honestly, I don't know. Like, I, I, I think that's actually like an open question. I think people say like, oh, RAG and fine tuning are like complementary, not competitive.

• 24:37Speaker 01

  I mean, I think they can pretty easily be competitive. Fine tuning just gets much better in terms of like a WUX and also capabilities. Because it's starting to show hints of that. I mean, like, fine tuning has gone a lot cheaper for what it's worth. You can find and stuff for like, like 10 bucks, like 20 bucks every like smaller amounts of data. I think the issue is partly like education and also the ability for it to actually just, you know, memorize and reason over a bunch of stuff. And so there is definitely a chance that, you know, like, parts of RAG might be supplanted by fine tuning. But exactly as you said, there are generally like certain aspects of RAG that will be hard for fine tuning to kind of like take the place of.

• 25:20Speaker 01

  And those specific aspects I see is one, exactly as you said, like the auditability, like being able to see the data flowing through the model. And then the other piece is like, yeah, like stuff around that, like access and full, like being able to actually like properly get access to certain pieces of data. Like, you just can't do that with a line of follow.

• 25:37Jacob Effron?

  That's a great point. I mean, it's like, yeah, if you have certain, you know, certain external data that you don't want certain folks to see, how in the world do you encode that into a fine tuned model?

• 25:45Speaker 01

  I do think there'll probably be complementary in, in general. And, but, but I, I, I think there is probably a world where like, as fine tuning gets better. And this is something that I think people generally don't have like conclusive opinions about, like that you can start using fine tuning more and more for just like, kind of memorizing certain pieces of, of information. That is like my hot take and we'll, we'll build towards that world. It's just maybe not like an immediate thing right now.

• 26:16Jacob Effron?

  I know one thing you've also talked about, you know, publicly on the roadmap is, is, you know, building out an enterprise product. And I'm curious like how you think about, you know, the challenges today and moving from demos to production and, and how you might address that, both, you know, I mean, higher data volumes, kind of the need for data refreshes, but like, wondering how you're thinking about what that. Enterprise product might look like and some of the most common barriers you see when folks take these cool demos and then want to put them in production.

• 26:38Speaker 01

  Yeah, for sure. So we're actually building this out right now. And so I think like public details, we're probably be in like a month or two, once we have like a beta release. At a high level, I think we're noticing like two big areas of pain points in terms of like, basically try to build L M maps in production. One is this whole thread of like, how do I actually evaluate and observe what's going on within the model itself? Because I don't really, you know, understand like, you know, I built this entire thing that does a bunch of like prompt training and, you know, like agent algorithms and I wired it with a vector database. But how like, how do I know what's going on under the hood? And how do I iterate on like prompts? And so that is, for what it's worth, like a pretty big pain point that we talk to with respective users.

• 27:23Speaker 01

  The other aspect with RAG is the idea of like data, I think, like, and this relates to this idea of having like suboptimal performance. But I think a lot of users don't really know how to like properly like manage a wide, a wide variety of data sources. How do you actually load in, you know, all your data within an organization? Especially as you scale beyond just a few like PDFs into just like heterogeneous data sources. And how do you properly make these like good data decisions for good like search retrieval? So that's like another category that like has been a big pain point. And so I think for what it's worth, like our enterprise project will probably be a little bit of both, maybe a little bit more towards the latter aspect.

• 28:05Speaker 01

  And just like helping users like build these like robust like data pipelines I was talking about. And I think like open source does provide some of the tools to help users do that. The main issue with kind of like, you know, letting developers figure out like a lot of the problems purely with open source modules is that it's almost like too unopinionated, like something like there are certain decisions that developer might not actually want to make. Because there's just like, like stuff around like, like scalability, like setting up infrastructure, like how do you actually like version track these things? How do you actually observe what's going on under the hood? That they would have to write like a ton of code for. So to whatever extent we can provide like a managed service to help them to set up these pipelines.

• 28:48Speaker 01

  That complements open source. That's basically the high level of what we're thinking about.

• 28:52Jacob Effron?

  Makes a ton of sense. And what I'm super curious about, like, obviously, I feel like there's this interesting balance where the space is just moving so fast. And to your point, it's so unclear what like the end state of all this is. And you mentioned things like knowledge graphs and the role they might play in the stack going forward. And then at the same time, like, there's probably, you know, on the enterprise side, there's a desire to really, you know, put into production, the stack that we have today, and and kind of the best way we do things. And so how do you think about that balance of like, you know, finding ways to to productize, you know, state of the art today, while also kind of following wherever the space keeps developing, kind of dog ears month over month?

• 29:25Speaker 01

  Yeah, I mean, I think, like, from a company perspective, there is basically just like two aspects, right? There's open source, and there's this enterprise thing that we're building. And so that's basically the two kind of vehicles we're thinking of addressing the two directions that you mentioned, like, one is addressing people's needs as they try to like, productionize existing applications. And then two is just being able to stay like, you know, flexible and adapt to whatever the latest trend is going on, especially with the volume of releases. So we see the open source as basically being the vehicle for a kind of experimentation, thought leadership. And as these abstractions develop, we start baking it. Like, we basically explicitly tag a lot of our abstractions as like, beta,

• 30:06Speaker 01

  like, experimental. And then as we develop more and more, we start, you know, investing more resources, building guides, and then people start using it for like, actual production use. And so that's something that, you know, like, we see the open source as being a perfect vehicle for it, because it's a lot faster to iterate on like, an open source project than it is to iterate on like, a full stack app, right? Just by nature of like, the like, types of work that you need to go into this, like, you don't really need a PM, you don't need a designer. It's mostly just like code modules. And so you can triple that faster. And so the enterprise product would be kind of like the vehicle for managing existing user needs as they go into production.

• 30:44Speaker 02

  One question I had was around the storage mechanisms and how people are storing these embeddings. Because you were talking about knowledge graphs and potentially that being an emerging trend. And then you also have all of the traditional DBs now announcing their support for vectors. And like, I think Mongo announced vector search and that sort of thing. Like, do you still see the vector DBs as kind of the de facto storage mechanism for these embeddings? Or are you starting to see other patterns emerge?

• 31:15Speaker 01

  Yeah, I think embedding search has its place. It definitely has its place. I mean, embeddings are just like a very concise way of doing like fast lookup from like semantics. And it's going to be a lot faster than, for instance, like crossing coding or using LLM to do retrieval over like massive volumes of data. So I do think embedding search itself will probably be here to stay. I think combining embedding search with other with like other ways of doing retrieval is something that we're basically actively exploring. So a lot of that stuff I talked about with like recursive retrieval requires that you do embedding search on some index first, and then maybe you exploit relationships within that index to then continuously like fetch more documents. And then maybe you

• 31:58Speaker 01

  want to do embedding search there, too. So kind of like defining this like high level orchestration, or like being able to do like combine embedding search with like other types of data that you have and exploiting those relationships is something that we're very interested in.

• 32:15Speaker 02

  So you would run the embedding search first, and then that would give you, you know, a bunch of knowledge back. And then you would use that to then query the graph? Is that the right way to think about it?

• 32:25Speaker 01

  Yeah, so clearing like a knowledge graph, yeah, that's that's basically one way of thinking about it. It's funny, these days, I think like Neo4j and NebulaGraph like have also baked in embedding search now. And so kind of like figuring out how much you need to use like two storage abstractions, or one will probably kind of be a function of the implementation. The other aspect is if we like just within some of the like recursive retrieval abstractions that we have in Laman index, like If you combine like a vector store with like a document store, something that just gives you like key value, like allows you to like store key value document pairs, like that

• 33:07Speaker 01

  allows you to do stuff like recursive retrieval, because you know, you would first do embedding search, find this initial set of IDs, use those IDs, like maybe fetch information about the source documents and relevant chunks, and then you can use those to then continuously do embedding search, or just directly fetch the raw content. And so this interaction between like a vector store or doc store is something that you can basically stitch together to implement some of these advanced retrieval algorithms. One thing I think from our view that we tried to not like, that we tried to do is to not like overfit on existing abstractions, because I imagine like the like Pinecone, like we've like existing vector stores are adding like document search capabilities, and then document stores are adding like vector search capabilities. And for us, like,

• 33:52Speaker 01

  you know, we want to enable users to do these advanced things, regardless of what underlying like kind of storage system they're using. And so whether it is like just using like we've eight or Pinecone itself to store both documents and vectors, or keeping it a little bit separate, like using vector stores purely for embedding search, but then document stores for like key value lookup, regardless of what the solution is, like we want to enable users to implement these advanced algorithms.

• 34:20Jacob Effron?

  And one thing I'm also curious for your thoughts on is obviously, you know, OpenAI has been kind of heavily leaning into the enterprise product recently. And I think, you know, from the announcements, they made it seems kind of inevitable to have something in like rag or fine tuning, you know, how do you think about how Llama Index might, you know, live alongside that?

• 34:35Speaker 01

  Yeah, I mean, I think, I mean, like, yeah, they probably will. I mean, I think Koshir has like a rag endpoint. I think in the end, there's, it's basically just like, okay, what is the value of like a framework? The value of frameworks, so you don't have to be locked into OpenAI's ecosystem. And so the idea here is like, you can use our abstractions at any point, pretty much like all these different things, but also allow to plug and play different outlines and embedding models, so that you don't just live within like OpenAI's ecosystem. I think the main risk is, I think, like, in the world where OpenAI, OpenAI's models are just better and also faster and cheaper than than literally everything else, and also somehow like has all the advantages of the open source,

• 35:18Speaker 01

  then yeah, like they would just monopolize the market. I mean, like then then like a bunch of companies would probably just die. Like, so so I think, you know, that that is a risk. I think one thing that we're betting on is just like, like that there is kind of like a healthy ecosystem of like this like Pareto curve of like, like, you know, performance versus like, um, uh, cost. And so like users can basically have their, uh, like, basket of choices and kind of like deciding, you know, like what makes sense given my use case, right? Like, do I want to spend more money, you know, send stuff to the cloud? Uh, and then also like get back, uh, stuff from OpenAI so that I can have like better performance? Do I want to fine tune an open source model because it's more specialized? I can host it and it's cheaper to to run. Um, I I think we're kind of banking on that future.

• 36:02Jacob Effron?

  I feel like you guys have been rolling out a lot of, uh, you know, more detailed, uh, you know, customizations. And, you know, it feels like it's early days of folks beginning to explore and use some of those. And so wondering, as you think about like the future of of how folks use Llama, are there some parts that you guys have recently shipped that you're like, I think, you know, maybe not a ton of folks are using that today, but it's something that, you know, I imagine a lot of people will use going forward.

• 36:21Speaker 01

  Yeah, actually, okay, so there's, um, uh, a few things. One is probably like some of our, um, uh, like, uh, better retrieval abstraction. So we spent a lot of time on this kind of key concept called like recursive retrieval, um, which is basically just this idea of, it's actually turns out to be a pretty flexible abstraction, um, to allow users to model retrieval, not just as like a one shot, like top K pass, but actually you can kind of like retrieve documents by their summaries first, and then go into, for instance, like documents and and retrieve like different chunks within it. Or you can model like kind of document hierarchies, like being able to retrieve like different nodes corresponding within a document. But then if a document has like embedded tables or something going in it, actually, you know, querying those as well.

• 37:07Speaker 01

  It also actually helps for just like being able to kind of explore this concept of like decoupling the thing that you want to retrieve, which with like the actual like kind of text or content that you want to feed to the language model, which also ends up being a powerful concept. So for instance, like embedding the raw text, for instance, like can lead to some downsides and just like the saliency of the representation in embedding space. But if you embed like a summary of that text, for instance, or embed like just a metadata, then you can basically kind of have better retrieval on top of that data, but then still link to enough context for the LLM to synthesize information over. And so this kind of is all captured by this concept called recursive retrieval, which we've been kind of, we've

• 37:52Speaker 01

  developed actually for about like two months or so, but it captures pretty much all three of those cases, like document summaries of documents, like embedded like kind of tables and stuff within documents, as well as like references to data as well. So decoupling the embedding retrieval from like the synthesis piece. And it turns out like when you actually do this, for a lot of like retrieval tasks, it actually does improve performance. And you can try this out on kind of like synthetically generated data sets, too. And so it's an abstraction that I really like, and I would definitely encourage like developers to try it out. And I think it's something that, you know, we've also been kind of like promoting a lot of use cases on Twitter as well. The other aspect here is like fine tuning, which

• 38:35Speaker 01

  is interesting. I think these days the talk is all about like fine tuning, like LLMs, especially with OpenAI's embedding endpoints, and also people playing around with like fine tuning Llama 2 with like Q-Lora and, you know, other techniques. But if you want to build like better RAG pipelines, one thing you could basically try to squeeze on with performance is just like fine tuning your embeddings. Like if you want to get like better retrieval, right, you can like try fine tuning the embedding model to basically get better performance over that domain of data that you have. So when you're just starting out before you actually index like a giant set of documents, you could try basically like fine tuning the embedding model itself. Once you actually have like documents in a more production setting, then like kind of continuously fine tuning the embedding model becomes more

• 39:18Speaker 01

  expensive, because then you have to re-index all your documents. So I think like Joe from Vesta was actually saying like, We, I'm on our webinar a few weeks ago and there's this idea of like just fine tuning like an adapter on the query instead of the document embeddings itself. If you can fine tune an adapter on the query side, then you don't actually need to re-index all your embeddings, but still, you know, you can kind of like optimize the representation of the query embedding for better retrieval. So basically, you know, we're interested in like fine tuning, both embeddings and LLMs as a general concept. Also interested in just like production grade retrieval and data representations and how you can like create links between your data for better retrieval.

• 40:01Speaker 01

  A third thing that like, you know, we don't have like kind of, we're still kind of exploring. We haven't like necessarily demonstrated like quantitative evidence that it's much better, but is very interesting area is this idea of like just modeling data in terms of like knowledge graphs as opposed to purely within like a vector database. So if you can combine data representations in like a graph store, but also within a vector store. And so like entities will have relationships and linkages to other entities. How can you use that for like better retrieval? And it's something that we're basically exploring as well.

• 40:32Speaker 02

  Maybe we can move to a quick fire round to wrap up. One thing that we always like to ask folks is what's one thing that you think is overhyped in AI today and one thing that you think is underhyped in AI today?

• 40:44Speaker 01

  I mean, I don't know if it's actually underhyped. I just, the hot take about like, like fine tuning, I feel like people generally haven't really fully understood the implications of like fine tuning in general. And I think what, like, I feel like all the existing comparisons of like RAG and fine tuning, like when you should use both will probably change quite a bit if fine tuning evolves, especially like given the current rate of like technological progress. And so like six months, a year from now, I imagine we'd be having like a different conversation as like the value of fine tuning.

• 41:18Speaker 01

  Other aspects are under hyped. Honestly, pretty much literally everything else in AI. Like, I feel like we've talked, like, pretty much everybody's just doing LLMs these days. Like, I mean, I used to come from, like, self-driving. I mean, I still think robotics is cool. I mean, I think, like, computer vision is cool. I figure, I feel like once OpenAI, like, releases, like, multimodal stuff, there might be a resurgence of interest in just, like, image and other types of complex data, too. Kind of like when Apple, like, releases a new product, like, everyone becomes interested in that, like, new trend. Like, I, I feel like there's, like, a vast area of AI that's not just, like, uh, like, you know, generative AI, specifically LLMs, like, task-based stuff. Um, and, and I think, like, there'll probably be, like, interesting intersections between the two, too. But, like, that part, I'm less qualified to speak about.

• 42:01Jacob Effron?

  I mean, as you alluded to, the space has just been changing so fast, and I feel like you've got a lot of these great hot takes. I'm curious, like, what's one thing, you know, what, what is one thing that you've, like, changed your mind on over the last, like, six months? That you thought pretty strongly, uh, you know, three, six months ago, and now you're like, oh, no, the world's definitely not going that way?

• 42:16Speaker 01

  Yeah, it's a good question. Um, I guess, like, probably, I think, one thing, I think when I was first starting out, I thought, like, I think a lot of people were asking about, like, the value of, like, um, doing RAG, like, at all, um, given that, like, context windows would get bigger. Um, and I think I remember, like, kind of talking to some folks back in, like, December or January as to their thoughts on, like, you know, like, if these context windows gets, like, I guess, 100K with Anthropic or even longer, like, what's the point of, like, trying to share how to hack the context window? Because, like, yeah, at the time, I mean, it is kind of a hack, right? You're basically just, like, doing retrieval and figuring out, uh, best way to, like, stuff the context window. Um, and it turns out RAG has at least prevailed up until now.

• 42:58Speaker 01

  Like, it's been a pretty predominant paradigm of, like, building question answering. I mean, I think the reason for that, there is, like, a few reasons. One is, like, um, uh, like, there is a fundamental, like, cost, like, time trade-off, um, like, when you try to stuff the context window. Um, I think that will probably come down, uh, for what it's worth. But, like, right now, when you stuff, like, Anthropic's context window, it takes, like, 50 to 60 seconds. Um, and, like, even if it gets faster, like, people will probably just expand their data to try to fit the context window anyways. So as you get to, like, a million, to, like, 10 million tokens for, like, people will just try to, like, stuff it full of, like, all different types of data. And so there will always be kind of like some argument for, hey, maybe you don't actually

• 43:40Speaker 01

  need to do this and you can just like get like 100x faster latency by just like, you know, doing retrieval first before actually stuffing it. So that's something that, you know, I kind of worried was a risk at the beginning of this year. Turns out, I think it is less of a risk. And then like, you know, as I do think that argument probably prevails even as like context windows get bigger.

• 44:01Speaker 02

  The next fire round question, I noticed that you have that the the llama index repo is in your personal GitHub. Any plans to to move that or is it staying in the personal GitHub for a while?

• 44:15Speaker 01

  That's a good question. You know, that actually has not really been something that's like consciously popped in my brain, but yeah, we should probably move it. Yeah. We, yeah, no, I mean, we should move it. Yeah. We have like a, we have like a organizational repo. It's called Run Llama. Yeah, it's a good point. We should, we should move that.

• 44:34Jacob Effron?

  Do you, do you take full credit for Facebook deciding to copy your name like a week after you guys finally settled on a, on llama index?

• 44:40Speaker 01

  Yeah, I, I feel like I haven't really been loud enough about this. And sometimes when I'm at these talks, people ask me why I named my, this project after the llama model. And then, like, yeah, so, so like, I think, anyhow, I think it's just a naming decision that I'll probably have to live with for the rest of my life. I don't want to change the name again. Renaming stuff from GPT index, the llama index already was was a huge pain because I had a, we like maintained, we still maintain, I think, backwards compatibility with GPT index. Like, we basically, like, have some hacky script that, like, duplicates the package and renames all imports to GPT index and just like uploads it to

• 45:22Speaker 01

  PyPI. And so having to deal with that same process again is just too painful. So it's something we'll live with.

• 45:27Jacob Effron?

  Well, Jerry, look, this has been a super fascinating conversation. I feel like our listeners are going to get a ton out of this. You know, one, I'd ask just where people can go to learn more about llama index and you. And then two, we have a lot of, you know, listeners that are building applications. You guys put out a host of just really great resources for developers. Like, anything specifically you want to point folks to? This is, this is your chance to plug.

• 45:48Speaker 01

  Yeah, no, I mean, I think I would probably follow our Twitter. I think that's probably the latest content. I would definitely join the Discord for discussions. I think there's a lot of like community discussions, especially if you have like questions on how to use like certain abstractions. We welcome feedback on our docs. Obviously Twitter content's all like transient. And like, so it's like, you know, it's not going to live beyond, you know, like 24 hours. And so that content, we try to put in our docs as well. If you have feedback on that, please let us know. We try to put all these resources there. And we're always actively trying to update our docs to make sure that we kind of like stay on top of the latest concepts. Besides that, um, no, I think that's it. I mean, I think I would check out some of the other cool repos.

• 46:32Speaker 01

  One repo I'll quickly plug is we recently launched like a open source full stack application called SEC Insights. Where we basically built like a full on like finance app and just like fully release it to the community. And this allows you to help like basically have a reference template for not just like building an LM app like truly in like Python, but actually figuring out how to like integrate it into our website. And so, yeah, I think it's gone a decent amount of traction already. But if you haven't checked it out, you definitely should.