Long-Term Memory for LLMs, with HippoRAG author Bernal Jiménez Gutierrez

Nathan interviews Bernal Jimenez Gutierrez, creator of HippoRAG, a novel approach to retrieval augmented generation inspired by the human hippocampus. In this episode of The Cognitive Revolution, we explore how HippoRAG improves on traditional RAG systems, its neuroanatomical inspiration, and its potential for handling complex queries requiring multi-hop reasoning. Join us for an insightful discussion on the future of AI memory systems. Apply to join over 400 founders and execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj RECOMMENDED PODCAST: 🎙️ Second Opinion - A new podcast for health-tech insiders from Christina Farr of the Second Opinion newsletter. Join Christina Farr, Luba Greenwood, and Ash Zenooz every week as they challenge industry experts with tough questions about the best bets in health-tech. Apple Podcasts: https://podcasts.apple.com/us/podcast/id1759267211 Spotify: https://open.spotify.com/show/0A8NwQE976s32zdBbZw6bv - 🎙️ History 102 with WhatifAltHist Every week, creator of WhatifAltHist Rudyard Lynch and Erik Torenberg cover a major topic in history in depth -- in under an hour. This season will cover classical Greece, early America, the Vikings, medieval Islam, ancient China, the fall of the Roman Empire, and more. Subscribe on Spotify: https://open.spotify.com/show/36Kqo3BMMUBGTDo1IEYihm Apple: https://podcasts.apple.com/us/podcast/history-102-with-whatifalthists-rudyard-lynch-and/id1730633913 YouTube: https://www.youtube.com/@History102-qg5oj SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitiveThe Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS: (00:00:00) About the Show (00:02:52) Intro (00:05:10) RAG (00:09:29) Hippocampal Memory Indexing Theory (00:14:29) Neocortex, Parahippocampal Regions, Hippocampus (00:18:35) Hipporag (Part 1) (00:21:12) Sponsors: Oracle | Brave (00:23:20) Hipporag (Part 2) (00:23:29) Understanding the Hippocampus (00:29:37) RAG (00:31:38) Runtime (Part 1) (00:36:30) Sponsors: Omneky | Squad (00:38:17) Runtime (Part 2) (00:38:17) PageRank (00:39:29) Headline Results (00:43:22) Gaia Benchmark (00:46:33) Pathfinding vs Path Following (00:51:24) Future work (00:53:26) Starting with a query (00:58:30) Long context LLMs (01:01:50) Hybrid approach (01:05:03) AI Town (01:06:58) Flair (01:09:05) Getting it right, not cheap (01:12:33) Technologies to highlight (01:16:57) AI capabilities that are still unsolved (01:19:47) Transformers meet neural algorithmic reasoners (01:21:02) Closing (01:21:31) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Bernal Jimenez Gutierrez, PhD candidate at Ohio State University and the lead author of HippoRAG, A Novel Approach to Retrieval Augmented Generation Inspired by the Human Hippocampus. RAG, of course, has become one of the most important techniques for grounding large language models

• 0:43Nathan Labenz?

  in specific knowledge bases. But as anyone who's implemented a RAG system knows, there are still major challenges, particularly when it comes to questions whose answers are not recorded explicitly in a single document, but have to be inferred from a more diffuse collection of sources. That's where HippoRAG comes in. Drawing inspiration from the Hippocample Indexing Theory of Human Memory, Bernal and his collaborators have developed a system that uses LLM-powered entity recognition and embedding clustering synonym identification techniques to pre-process knowledge into a graph structure, which can then handle complex queries requiring multi-hop reasoning far more quickly and affordably than previous state-of-the-art approaches. In our conversation today,

• 1:26Nathan Labenz?

  Bernal walks us through the neuroanatomical inspiration for HippoRAG, explains how it works under the hood, and discusses its current capabilities and limitations. We also brainstorm ways that it could be extended and improved, considering how it might complement other recent advances like Raptor, as well as the strategy of periodically creating synthetic reflective memories, which I first encountered in the AI Town project. Notably, it seems to me that these techniques are pretty much conceptually orthogonal, such that an application developer might be well-served to implement all three, and perhaps even more, into a single RAG system, assuming, of course, that the use case justifies the investment. As always, if you're finding value in the show, we'd appreciate it if you'd take

• 2:09Nathan Labenz?

  a moment to share it with a friend. We welcome your feedback via our website, cognitiverevolution.ai, and you're always welcome to DM me on your favorite social network. Finally, for now, a programming note. I'll be away for a short vacation and to speak at the Adaptus Summit in Sao Paulo for the next couple of weeks. But don't worry, we've got a full schedule queued up for while I'm gone, and I plan to make a big push to generate more connections between AI engineers and advisors and actively hiring companies when I'm back. So definitely remember to submit your resume if you're interested in being a part of that. Now, I hope you enjoyed this fascinating exploration of biologically-inspired AI memory systems with Bernal Jimenez Gutierrez, creator of HippoRAG.

• 2:52Nathan Labenz?

  Bernal Jimenez Gutierrez, author of HippoRAG, welcome to the Cognitive Revolution.

• 2:58Speaker 01

  Matt, thanks so much for having me. I'm excited.

• 3:01Nathan Labenz?

  I guess for starters, two things caught my attention about this project. One is just the performance, which we'll get into, and the fact that it works and seems very practically useful. The other is that you're drawing inspiration from biological systems to kind of motivate the design of this new RAG system. So as the name would suggest, the project is inspired by the hippocampus, which I think is cool. You want to just kind of give us a little bit of background of, first of all, who you are, the group you're working in, and then specifically for this project, like, what was the kind of original motivation? What was the bug that got you to dive deep into this?

• 3:37Speaker 01

  Yeah, sure thing. So I'm entering my last year here at Ohio State. I'm working with Professor Yusu at the OSU NLP Lab, and our group does a lot of different things, from embodied agents to more like mind-to-web came out of our group, so like agents in the web. And Dr. Yusu is really interested in the biological inspiration side, so that also is a big part of our group. But me, in particular, I graduated from Berkeley in 2015 with an applied math BA, and I started working at a healthcare startup that did

• 4:18Speaker 01

  NLP for matching cancer patients with clinical trials. And I really got into this biomedical NLP direction of research.

• 4:27Speaker 01

  And I've been doing that for many years, using AI to facilitate biomedical research through NLP tools like. Well, in the past, it was just like ontologies, knowledge bases like UMLS. And I've been kind of transitioning into how do we really use LLMs effectively in this biomedical NLP space? And from all of this biology accumulated knowledge, I started to become more and more passionate about how the the brain works and how we can get inspiration from that to actually make LLMs better. So I think of my research as this two-way street between biology and LLMs.

• 5:10Speaker 01

  For Hipporag, how it initially came about, it was also this kind of two-way street where at the beginning, before even starting, I was thinking about domain-specific LLMs and how they're gonna fit into the current NLP landscape. And really, because of the limited amount of data that there is in specific domains, it's kind of not a great idea to just pre-train from scratch in these limited data domains. And so I was more leaning towards like, okay, RAG plus a very strong LLM trained on everything is probably your best bet. But in looking into that, I realized that RAG has some very major limitations.

• 5:55Speaker 01

  And at the same time, me and my advisor were both thinking about mirror biological memory, so like human long-term memory and our ability to do continual learning. Like, we do continual learning in a way that is basically impossible for current LLMs to do. And so it was kind of this back and forth between these two ideas, like there's issues with. Current models, a current RAG, and how really to best utilize associative memory, like the concept of associative memory in humans, to make this kind of synthesis of the two. And I think early this year, I kind of realized the nugget of inspiration that kind of made

• 6:40Speaker 01

  everything fall into place was this idea of knowledge consolidation through path-finding multi-help questions, like this idea that like LLM chain of reasoning is not all you need for finding in for knowledge-seeking questions, really. You need to have this preprocessing knowledge consolidation of your information in order to really answer all the questions that you want. So in the paper, we use this example of someone trying to find the Stanford professor that also works on the neuroscience of Alzheimer's, theme appropriate. And then we basically show that if there's a bunch of, like a thousand Stanford professors

• 7:23Speaker 01

  and a thousand Alzheimer's researchers, if you just use path-following, like kind of have an LLM, look at your corpus, find the appropriate next step and then follow it, you would have to be looking at like multiple thousands of documents in order to do this correctly. And a human that just knows this professor is at Stanford and does Alzheimer's research will likely just remember this professor immediately. So yeah, that was kind of the link that brought these two directions of thought together.

• 7:56Nathan Labenz?

  Number of interesting themes there. I'm big on the application of language models to specific domains and I often call myself an adoption accelerationist in the sense that I think it's really important that we not just keep scaling up systems and hope that one day they work, but actually take the systems that we have now and really put the effort into dialing the performance, validate the performance and start to get practical utility. Sooner rather than later. I think, by the way, medical domain is is one of the great candidates for that. And I've had multiple conversations with the team that was the Med Palm team is now the Med Gemini team. And a huge, huge fan of that line of work and that

• 8:36Nathan Labenz?

  line of thinking. The theme also of continual learning is definitely a big one. You know, anybody who's built with language models sort of has this experience of the missing middle memory is sometimes how I describe it, where you've got kind of this superhuman long term memory that knows more than people do. But then you've got this very finite and kind of, you know, either the information is or isn't there in the context window. And then in between you have RAG. But as you said, that obviously has a lot of limitations, especially with the implementations that we've had to date. I think I get how the kind of prior state of the art in RAG works. But maybe you could run through that and talk about how it actually works, why it's slow, why it's expensive, and then maybe give us

• 9:21Nathan Labenz?

  a little bit of the understanding of how the hippocampus works. And then we can get into how you implemented that to bring that approach to the technology side.

• 9:30Speaker 01

  Cool. So RAG Retrieval Augmented Generation, basically, at its most basic form, uses some sort of information retrieval system, be it BM25, the most simple lexical based system, or more recent semantic embedding approaches like Contriver, Colbert, we use in our paper, or you could even use LLM embeddings. There's many ways of building specific vectors for each document in your Corvora. And then the idea is that when the user has a question, the LLM first looks up the relevant documents in the Corvora, puts them in the context window,

• 10:15Speaker 01

  and then the LLM has all the information it needs, and then it generates the correct response, right? So that's kind of the current state of retrieval augmented generation. There's a lot of different ideas with how to augment the generation, like you could correct errors as you go, like FLARE does, like forward looking active retrieval, or you could retrieve many, many times and generate many times and then kind of do majority voting. There's a lot of different ideas about how to do this retrieval. And also in order to do like more complex consolidation of knowledge

• 10:56Speaker 01

  within this framework, you would suppose that the model decomposes your question into steps. So if your question is more complex and it requires more than just one retrieval step, for example, in any sort of multi hop question, you would first retrieve something, then reason over it, and then do another step of retrieval as many times as you want in order to accomplish that. So this is not perfect knowledge, right? This is just kind of theoretical at this point, but the hippocampal memory indexing theory, which is what we based Hipporag on, tries to explain a lot of empirical data on the hippocampus being important

• 11:37Speaker 01

  for long term memory storage, because, well, in the like mid nineteen nineteen hundreds, the patient HM that got their hippocampus resected basically can't form new episodic or semantic memories, like long term memory. And so people are trying to explain why the hippocampus is so important there. And the hippocampal memory indexing theory basically posits that it's an index that just stores pointers to the actual memories in the neocortex, and it also stores associations between these pointers, right? So it it doesn't have any true information about what these pointers are.

• 12:21Speaker 01

  It just points them to the actual representations in whatever area of neocortex they should lie in, right? Like visual cortex, motor cortex, your language areas, et cetera. And so.

• 12:36Speaker 01

  The idea is that as you obtain new knowledge, you build this associative index iteratively. So in our Stanford professor example, you would kind of create a representation for the Stanford professor, link it to Stanford and link it to the Alzheimer's research idea. And then you'd have this index that you can traverse to find the appropriate Stanford professor. And I could go into like three components that are proposed for in the hippocampal memory indexing theory and also pattern separation, pattern completion, but not sure how detailed we should go in there.

• 13:14Nathan Labenz?

  Yeah, uh, let's do it. Just kind of take one step back into the naive rag approach. Cause then this will motivate like how we're doing it better and how the, the new system inspired by the biology is doing it better. But if you had a giant corpus of information and you know that there's lots of professor bios and researcher profiles in there, you could embed this, but if you're unlucky and there's not a single document that contains both the Stanford and the Alzheimer's keywords, if, if those are not co-located in a single bit of text, then the naive way would be you could start with all the ones that work on Alzheimer's or all the ones that work at Stanford.

• 13:57Nathan Labenz?

  But again, then you'd have to look at each one and say, okay, now get me more information about that person. Then bring that back and see if that's right. And so you'd have this like long, kind of brute force search through the document space to ultimately find the thing that you're looking for. You don't really have a great way of like quickly zeroing in on the one that you want. And that's why it either can be expensive if you paralyze it highly or slow. If you just grind through it serially or potentially both, depending on the details of your setup. But that's obviously not ideal.

• 14:29Speaker 01

  Exactly.

• 14:30Nathan Labenz?

  And then, as you said, just generally, if you have those two pointers, like I'm not sure I can get there just through introspection, but it does seem like intuitively clear that. If given a couple pointers, I can usually quickly locate the memory without having to do some sort of like, you know, it's pretty clear to me that I am not doing that process where I'm not thinking like, what's every professor that I know, is that the one, you know, that satisfies this criteria? So with that motivation of kind of the slow naive approach, let's do the little bit deeper dive into the hippocampal indexing theory and then we'll map that on to the technology implementation.

• 15:09Speaker 01

  Great, yeah, that was very helpful. I think for viewers to get a better sense, so the hippocampal memory indexing theory kind of posits that there's three major components of human long-term memory. There's the hippocampus, obviously, that kind of does this associative index, which is like the centerpiece of this framework, but there's also the neocortex that actually represents the memories that you want to retrieve or re-experience. And then there's the parahippocampal regions, which are like the interrheinal cortex and other areas that sit around the hippocampus.

• 15:47Speaker 01

  And theoretically, it seems that this area serves as a link between the hippocampus and the neocortex, like a bidirectional information channel, basically. And there's some ideas that some part of working memory is done in the parahippocampal regions, but this is still not very clear. And then these three components work together to create two fundamental characteristics of human long-term memory. So there's pattern separation, which basically allows you to disentangle different features that you experience, like create concepts like Stanford professor and Alzheimer's

• 16:30Speaker 01

  research, or any sort of feature disambiguation or separation. And then pattern completion, which actually allows you to Join together things that were relevant to a particular experience in the relevant ways, right? So the pattern separation is done through the neocortex and the parahippocreambral regions. Through the neocortex, it's mostly done, I I believe this is not like explicitly said in the literature, but it's a a part of the language system that kind of separates concepts out. And also, the parahippocramal region has a lot of sparsity. It's very over parameterized, so it allows for this kind of separation of patterns and features

• 17:15Speaker 01

  that then map into the hippocampus. And the hippocampus is mostly for the pattern completion part. Specifically, there's a subregion called CA3 that like is hyperconnected. You could think of it as like that. So there's a lot more associative connections than in other parts. And so you would have this recurrence nature that allows for you to remember things that are only associated because of your prior experience, not because they're associated semantically. And so ba- based on these two fundamental characteristics and also on the three components of hippocampal memory, we built Hipporag to like kind of mirror those things.

• 17:58Speaker 01

  So the neocortex and the language abilities of neocortex, we modeled through a large language model because it basically has these like local, at least local reasoning abilities and the ability to extract and disentangle features from language. So that seemed like a good parallel. We used retrieval encoders for the bidirectional bridge between the neocortex and the hippocampus. And we used a knowledge graph to represent the actual structure of the hippocampus. And then we used personalized page rank.

• 18:39Speaker 01

  To model the associative memory component of the hippocampus. So those are the the three components that make pattern separation, pattern completion possible. To to actually do pattern separation. We link pattern separation to the offline indexing process, which is basically the pre-processing that you have to do to turn the corpus that you have into an associative index, right? And so we first go by each document through the large language model to extract entities. And that basically just grounds the next step of generation, which is open information extraction or schemaless knowledge-based construction.

• 19:22Speaker 01

  So this basically has no rigid constraints on the type of entities that you can generate or the relations you can generate or even the entities themselves. So we are kind of using the named entity recognition of the first step to ground that second step. But it's just a bias. It's not like a forced constraint. So it generates a lot of different entities and relations. And we built the knowledge graph based on those triples extracted. And we then use the retrieval encoders to build synonymy edges. So this is a detail of the implementation that is actually quite important in specific cases. But you need some some sort of disambiguation

• 20:08Speaker 01

  or or actually like synonymy detection, the opposite of disambiguation for for this knowledge base to be more connected. And then the online retrieval, which we kind of linked or it it's analogous more to pattern completion. It also has a pattern separation part, which is basically the named entity recognition from your question. So whenever a question enters where you already have your knowledge graph constructed, you would first extract the features that you want to attend to, then link them to the knowledge graph via the retrieval encoders. So you find the nodes that are most relevant.

• 20:48Speaker 01

  And then with those entities, you begin your personalized page rank algorithm, which basically does a random walks around your relevant entities until you converge on a distribution of probability. And that gives you a distribution of probability over the nodes in your graph, which inform your ranking for your corpus.

• 21:12Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:17Nathan Labenz?

  Okay, so let me rewind and try to restate some of that back to you and

• 21:23Speaker 01

  Perfect.

• 21:24Nathan Labenz?

  make sure we have clarity on all the key parts. First of all, just to the biology, I often say this is an analogy free zone and for the reason that I find analogies often kind of misleading. In certain cases, you know, the tight enough analogy and obviously you're designing this, you know, by analogy. So here I do think it's kind of interesting to just look at those three regions of the brain and try to reconceptualize them in terms of familiar aspects of AI systems that probably listeners will be more familiar with. So it sounds like in the kind of detailed memories, which would be sort of akin to a passage or a chunk out of a corpus of text, those are thought to be encoded, or you might

• 22:09Nathan Labenz?

  say embedded and stored in the cortex. So that's like where the raw, most detailed form of memory is, right? And I guess there's a lot of space in the cortex to store all these memories. The challenge then becomes how do you access them? Same as we talked about before, if you just like have all this out there, searching through it one by one, brute force is not going to work well. So then I think that middle region, and was that the para hippocampal region? Is that what you called that? Correct. Okay, so that's really interesting because it seems to perhaps have a similar function to like activations, you know, you can almost think of it as like playing a similar role to

• 22:54Nathan Labenz?

  like a sparse autoencoder, which has definitely been a hot topic recently. And the phrase you used is pattern separation. So that's like, understanding which, within any given high detail, rich memory, what are the sort of salient concepts that matter when you abstract away all the little details from whatever the raw memory is? That is kind of striking. I hadn't really ever come across the idea that we have something quite like that, but it is an interesting parallel that there's this separated semantic encoding that sits

• 23:32Nathan Labenz?

  between the raw encoded or embedded memories and then this like index that connects them. Definitely, again, reminds me of sort of an activation, like a sparse autoencoder type of a function. Presumably, it's not, you can maybe tell me if there's more that's known about this, but if I had to guess, I would guess that it looks a lot more like the dense polys semantic, hard to disentangle representation that we find in the large language models as opposed to the sparse, clean, interpretable form that we find in the sparse autoencoders. But one might imagine that you could perform a similar sparse autoencoder type analysis

• 24:13Nathan Labenz?

  on the activity of that region and gradually figure out what are the high dimensional messy representations that seem to kind of correspond to recognizable features. I don't know if anything like that has ever been done, but does that seem like the right intuition for the role that that region is playing?

• 24:32Speaker 01

  Yes, I think as I see it, it's kind of a sparsity constraint on the dense representations. The role of the neocortex is to experience the world and predict the world in real time. And so the the representations have to be really dense and rich, right? But in order to remember them, you you need sparsity. So I think the parahippocampal region is kind of doing this sparsity constraint. And the interesting thing about that is that you said that the sparsity is probably not like polys semantic, like the large language model sparsity, but I'm actually not super sure about that.

• 25:12Speaker 01

  Because in the hippocampus, we know there's like several studies that find that there are such things as place cells, people cells, grammar neuron, different sorts of very specific monos semantic concepts that we care about that are encoded in the hippocampus. And so somehow it went from this dense representation in in the brain that has all sorts of polys semantic stuff in a in a high high dimensional vector space. And then it uh gets sparsified in ways that we maybe care about stored in the hippocampus. So in that parahippocampal region, you are doing a lot of pattern separation, a lot of sparsification in order to get to this level.

• 25:57Speaker 01

  So I think there is a lot that goes on in the parahippocampal region that is not super fully understood. And it's definitely not exactly like HippoRag in any like. Yeah, it's not exactly a HippoRag at all. This is just like a good implementation that works.

• 26:13Nathan Labenz?

  Well, hopefully, at this point, for our loyal listeners, goes to that saying, but yeah, made of different stuff, you know, definitely different mechanism, lots of lurking differences which could cause us issues, but. It does seem that there is something quite interesting there. And my statement of like imagining that it was probably highly polysemitic and messy is just based on the general prior that like everything in biology often feels like it's very messy. But I mean, and I have heard about those studies that kind of the Halle Berry neuron and all that crazy stuff is suggestive. Yeah, that maybe there is indeed a very sparse representation. Finishing on the bio,

• 26:51Nathan Labenz?

  the hippocampus itself then stores the associations between the, let's say, feature representations or the separated patterns, the perhaps sparse representations, so that you can quickly understand which of these concepts have co-occurred. And then that becomes like the core insight for this whole technology development, right? That becomes a way to quickly get from one memory or one concept to the related co-occurring concepts. So you can find that Stanford professor that studies Alzheimer's without having to like root force your way through all the professors

• 27:33Nathan Labenz?

  or all the studies or what have you. Then on the technology side, basically two parts to the process. There's kind of the preprocessing part, which is your setup. And this is where like there's major disanalogies, obviously, between humans and these systems. And then there's the runtime part. The key idea in both cases is entity recognition is like your starting point, right? So in the setup, you've got some corpus of information that you want to have command of, you want to be able to search through effectively, going through and using a language model to identify all the entities. I'd be interested if you could shed a little more light on like how that has worked or prompts or whatever. I also noticed, and I think in the paper that

• 28:14Nathan Labenz?

  you were using GPT 3.5. So curious as to why that was versus a more, you know, perhaps just cost or whatever. But I'm interested to understand what the constraints you were under and how you think that might change if you used a better model. But you're sort of identifying all these things. Okay, here's a professor, you know, here's a condition. This is Alzheimer's. This is another professor. This is another different condition, et cetera, right? Identifying all these sorts of things. Then the synonymy part is important. Again, this is moving from syntax to semantics, basically. If you have two words that mean the same thing, obviously, I think everybody is familiar at this point with the fact that in embedding space, those will be very close to one another. And so they'll both have high inner product with other relevant

• 28:58Nathan Labenz?

  things. And that's a big part of how the standard RAG systems work. But it's important to perform that step so that you have the ability to still match on entities that are extracted slightly differently. And then finally, you have a graph database that you compose out of all these entity extraction. So your ultimate format out of the preprocessing is all the original documents, all the entities that are extracted, then kind of collapsing those into synonyms, and then associating those synonyms so that you can quickly traverse from one starting point to things that co-occur. Okay, cool. Now, runtime, a query comes in,

• 29:38Nathan Labenz?

  you do that same entity extraction. Those entities now serve as the seeds for the search. Now, with the graph structure in place, you know, should be pretty intuitive that, hey, now we can go spider out to all of the relevant original raw passages, which is ultimately what we're going to retrieve to then have the language model reason over and answer whatever question we have. I mean, I can easily imagine like a naive thing where I might say, okay, I've got this entity and this entity detected, you know, just spider out and grab me everything. I'm a little bit unclear as to how that gets refined with this personal page rank algorithm. So maybe three questions would be, what if anything doesn't get wrong? Two, what's the deal with 3.5 and the and using that for the entity

• 30:23Nathan Labenz?

  recognition? And then, three, can you give me a little more detail on the personal page rank and and sort of what the choice parameters are there?

• 30:32Speaker 01

  Cool, yeah, so let's start with the simplest question. So GPT three point five, basically it's much cheaper than other models. The difference between three point five and Claude is like Claude is like twice as expensive in the inputs and like ten times as expensive in the outputs. And so that kind of puts a constraint on our methodology because offline indexing is somewhat expensive. So you have to produce a lot of like Open I E triples, basically. And so if you were to do our same experiments in using Claude or GPT four O, you would have to spend around like five hundred dollars to do the small subset of the three benchmarks that we did.

• 31:16Speaker 01

  And that's like a very limited subset, right? We use a thousand questions, around ten thousand documents per benchmark. So yeah, that would just cost too much money. And honestly, the biggest reason I used three point five is that the task itself is not really a relaying task, it's an extraction task. So it's like a salience detection and then when two entities appear in a sentence together, they have some relation, so detect the relation. And so it's kind of simple and different L L M's, even like Lama three seven billion is able to do a fairly good job at this task. And so apart from the formatting, the formatting is kind of challenging and you probably need to fine tune it.

• 32:00Speaker 01

  But because it's just an extraction task, it seems quite simple for like less intelligent L L M's to do. And so I wouldn't expect Claude or GPT four O to do much, much better on this task, just right up about. And then the second question about P P R. So this is kind of like the biggest limitation of our framework, I would say. And the thing that we're most excited to work on, because P P R is obviously a very naive way to spider out from the relevant nodes. Basically, if a node is connected to your relevant nodes through like directly and indirectly,

• 32:43Speaker 01

  it'll get more probability. The closer nodes to you get more probability. And if a node is connected to both relevant nodes, it'll get more probability. So it's based on the connectivity structure of the graph. And so there's no Relation guidedness or anything like that. It is basically just how connected these nodes are to the relevant nodes. And if you like turn up the amount of steps that your random walk takes before converging, you'll just do page rank on the whole graph. Basically, if you turn it up to basically like all the maximum am- amount of steps,

• 33:26Speaker 01

  then you just get the nodes that are the most connected in your graph, like the the nodes that are most popular. So, and that was used in the original Google search, right? Like if your websites are very popular, just give me those instead of other ones. But that's not really a good idea here. And if you turn it down too much, then the nodes that you get are like basically all you're gonna attend to. And so there's like a middle ground that you want for the neighborhoods around the relevant nodes and like between them in order to obtain good performance. I think our good performance in the benchmarks is due like somewhat to biases in there and

• 34:10Speaker 01

  the lack of some of a lot of destructors. But if you make this a large scale production corpora, you definitely need to make that graph traversal better. And so that's what like one of the things we're working on.

• 34:27Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 34:32Nathan Labenz?

  Yeah, interesting. So in this initial version, the the page rank algorithm itself is not smart, so to speak. And if I understand correctly now, it's like where you are starting in the graph and the limit of how far you're willing to go explore the graph determines what is pulled back. If you were to turn the limit all the way up, you would get. The whole graph sorted by connectivity, not sorted by semantics. And so you're sort of constraining to the relevant semantic region just based on where you're starting. So there's definitely more opportunity there to then say, let's also navigate in a more intelligent way.

• 35:12Speaker 01

  Yeah, navigate intelligently and efficiently. That's like the biggest thing, right? It's like you could just add like some sort of L N to do that traversal, but then you would increase the cost. So there's a range of ideas there to trade off on efficiency and intelligence. But yeah, it's not just where you start and the amount of hops. It's also like the local connectivity. So there's like both local connectivity and where you start determines your final PPR ranking. But that's it.

• 35:44Nathan Labenz?

  Okay, cool. So give us the headline results on this. You've run this on a number of different benchmarks. There's cost savings, there's speed improvement, there's accuracy improvement. And then from there, I kind of want to do a little brainstorm with you on like, what are the ways that this can still get better? And what would it look like if we just did a, you know, commercial no holds barred? Let's go make the very best RAG system that we can, borrowing from any and all methods, kind of brainstorming session. But for starters, headline results.

• 36:17Speaker 01

  Yeah, so on average, we compete with iterative retrieval, basically kind of head to head. We get a little bit better at some benchmarks and some benchmarks we get a little bit worse. But this is with like a tenfold decrease in cost and time spent doing the online retrieval. So that's huge for any production system that requires fast online access by your users, right? And the indexing costs can be brought down a lot by using smaller L Ns, as I discussed earlier.

• 36:57Speaker 01

  And another really interesting thing is that when you combine it with just naive iterative RAG.

• 37:05Speaker 01

  You can get very large boosts on average, like seven to eight percent across benchmarks. And so that is basically kind of proto like self-correction that you were discussing earlier. Because the model is seeing like the results of your ranking and it's able to reason better because you're able to do some associative memory retrieval, right? So yeah, that is generally the highlights of the empirical benchmarks that we showed. And then obviously, there's no empirical benchmark that exists that tests pathfinding question answering, sort of this like Stanford professor Alzheimer's researcher idea.

• 37:49Speaker 01

  But this is something that you could not do with current approaches. And then we tried it on a few like test cases, more qualitative. And it seems to do okay, this, it it seems to do well on some of these questions. And we need to make the graph traversal better in order to like fully unlock its potential. But the idea that you can answer all these pathfinding questions, some of which are extremely important. Like, one of the questions in our case study talked about what medication for some condition works by interacting with X enzyme or like through this mechanism?

• 38:31Speaker 01

  And so that is a question that doctors would benefit from answering. Because if there's some condition that would interact poorly with that mechanism, then you shouldn't use it, right? So that's a very important question. And I think having systems that can answer that type of question is important.

• 38:48Nathan Labenz?

  So the before, after and synthesis is before you have this iterative approach. It does this kind of brute force, which we've talked about a couple of times. After is.

• 38:59Nathan Labenz?

  You're just giving your own system one shot, basically, in saying you have an input, extract the entities, use those as the starting place to traverse through the graph, pull back whatever that process returns, that's what you get, and you compare that to the iterative approach, which has obviously multiple rounds of going back to the database, and find those to be roughly equal, although a lot cheaper and faster with the HippoRAG method. And then the natural next step is to say, well, what if we allow the HippoRAG method to also go back again if needed? And then with that, that's where you get the big performance boost. Cool. So yeah, I mean, the lack of benchmark on this is interesting. There's one benchmark that's coming to mind. I believe it is the Gaia

• 39:43Nathan Labenz?

  benchmark. You're familiar with this one?

• 39:47Speaker 01

  Yeah, it's from Meta, I think. Yeah, I'm familiar with this benchmark. It's like, it's related, and I think we explored it as an option for this. I think the web browsing portion of this is similar. However, because there's no like standard corpus, it makes it a little bit challenging to do it.

• 40:08Nathan Labenz?

  Preprocessing the whole internet is obviously still a challenge for ramping this up to like the highest scale.

• 40:13Speaker 01

  Or more like a cost, right? Like it's too costly. But yeah, so I think something like Gaia, but just maybe more constrained would be a good benchmark for setting.

• 40:25Nathan Labenz?

  Yeah, so I guess this also does highlight one of the other nice aspects of this, which is I tend to think about kind of a couple of different kinds of systems. One would be this like general purpose research assistant that I would need to like send out onto the web to find things for me. And then another would be like, and maybe these even sort of converge in some ways. Another would be like my own personal, you know, context navigator, right? Something that can like tap into my emails and tap into my Slack messages and, you know, hopefully kind of make sense of all of that and help me find the stuff that is really relevant. The the nice aspect of this that maybe we could sort of think about adapting or taking advantage of for these use cases is that

• 41:11Nathan Labenz?

  It's easy to like expand the corpus, right? You know, it's highly extensible to say, okay, if there's a new record, we can always just at runtime, like do a bit more preprocessing, do the entity extraction, add a few more records to the graph database, and then, you know, query and in theory that should work well. So I wonder if you could take a system like this and instead of having like a totally predefined corpus, if you could attack something like the Gaia benchmark with the addition of like some web search, you know, you could, and I'm just thinking about past podcast guests here. We've done episodes of a lot of different like research and kind of new approaches to search.

• 41:53Nathan Labenz?

  But Exa comes to mind as something that is like really good at finding sort of similar documents that can often be like off the deep web. The Brave Search API is also another really good source for just really quick and pretty robust results. And then there's perplexity as an API and Elizabeth doesn't have an API, but certainly has some interesting search technology. But I wonder if you could sort of set something up where it's like the agent has to, maybe it starts with like a seed corpus that could be like my own emails, or maybe it starts with a, you know, a English Wikipedia or something. Maybe that's too big to get started with, but if you had something that's kind of a start and then you allow it to like go out onto the web at runtime,

• 42:35Nathan Labenz?

  find new documents and expand the corpus with those new documents to then try to answer. And then in your iterative approach, it would be like, maybe you have to go back to the web and not just to locally indexed body of knowledge. This does start to look like something though, that could really tackle a benchmark like Gaia. Because from what I mean, to give you a sense, I should read one of these questions. Here's one Gaia question that I recently tried. A paper about AI regulation was originally submitted to archive.org in June 2022. Shows a figure with three axes where each axis has a label word at both ends. Which of these words is used to describe a type of society? In a physics and society article submitted to archive on August 11, 2016.

• 43:19Nathan Labenz?

  So it's this sort of, I mean, it's very much in line, right, with what you're trying to achieve here with this multi hop. Like, I have to ultimately triangulate to to find this one paper, I have to find this other paper, I have to locate this thing, after whatever. And then, you know, I have to figure out what the overlap is. It seems like, I mean, that one's kind of a weird one because like the connection is sort of the last step, I guess. In that, at least as I think about how I would solve that problem, I would probably make, I would like go all the way to the end of the one. And then all the way to the end of the other, and then like do the comparison last. But they have other ones too that are like maybe more suited for this. Here's another one. I'm researching species that became invasive after people who kept them as pets released them.

• 44:00Nathan Labenz?

  There's a certain species of fish that was popularized as a pet by being the main character of the movie Finding Nemo. According to the USGS, where was this fish found as a non-native species before the year 2020? I need an answer formatted as the five-digit zip code of the place the species was found. That one seems like a little bit more suited this way because like we're, these sort of hops are sort of exactly the sort of associations that one needs to make to reason from like Finding Nemo all the way through to ultimately a zip code, it would seem like.

• 44:33Speaker 01

  Yeah, so it's kind of funny because I don't think the distinction between path following and path finding is like very crisp in the current literature. And I think that's like honestly, one of the coolest like theoretical, like idea based things that HippoRag does. Is to kind of just very specifically define this distinction because both of those questions seem very hard, but they're actually just path following question. Because you, there's like a bunch of details about what type of fish you're talking about, right? Like, it's just the main character in Finding Nemo. And so you can like literally just like continue that down the chain of thought. And you find that fish and then you continue the process.

• 45:18Speaker 01

  The complexity of the question increases due to the number of hops.

• 45:26Speaker 01

  Not really because of the nature of the task. So the nature of the task in pathfinding questions is like set intersection, right? And that's not something that you need to do in this case. It's just kind of interesting. Yeah, and so, but like if you wanted to use HippoRAG in this Gaia case, you could do it with, as you said, like searching the web, then doing indexing. But that's also gonna take a lot of time, like unnecessarily indexing a bunch of documents in your online step. So that's a little bit sort of difficult to motivate. I think it would be better to like, kind of figure out, what's the corpus that I need in this?

• 46:10Speaker 01

  Like, I think it's archive, archive papers for your first question. Maybe like Wikipedia and some other stuff for your second question. I don't know, like, I'm sure they did it on some specific corpus. It would probably be pricey to index all of these documents. Like, for example, all of Wikipedia, I think it's like six million articles or something. If we just take the first paragraph of each of them, it'll be like nine thousand dollars to index Wikipedia with GPT three point five. But that's just the first paragraph, right? So you, like, you can imagine that going much higher. So that, for one experiment, for one lab, that seems like too much, right? So that's why.

• 46:50Nathan Labenz?

  Although by startup budget, it's extremely manageable.

• 46:53Speaker 01

  It's extremely manageable, yeah, exactly. For a startup or a production, or like a big tech company, then that wouldn't matter, but, yeah.

• 47:02Nathan Labenz?

  So let's, I guess, for a second, look at any other weaknesses or known shortcomings of the, the current approach? And then I'd love to just take all the budgetary constraints off for a minute and just imagine like, what would we do if? You know, not the same money as no object, but let's say we have like a substantial budget and we were gonna try to build the absolute best system that we could on either, you know, sort of a public search the web basis or a my context basis, or maybe even just say like a hybrid basis, right? Something where I'm gonna give it a lot of my context. That's like the starting base. As needed, you know, this hypothetical system can go out onto the web and find more stuff.

• 47:43Nathan Labenz?

  And then it has to use that, but also like ideally incorporate that knowledge or save what's relevant. So that next time it doesn't have to like redo it every single time. But first, anything else in the current setup that you think is like obviously calling out for improvement? Or anything else that you're already planning to work on?

• 48:02Speaker 01

  So there's graph traversal. That's like one of these directions where I think that we need to improve it ASAP. And then I talked about in the paper as well, the concept versus context trade off, which it's like basically the entity centric nature of Hipporag allows it to do a lot of these multi help question answering. Like it allows it to do well on these benchmarks, but it's not necessarily super generalizable. So if you look at the examples I used in the paper, there's one, whenever a question requires an entity to answer it, this is a great approach, right?

• 48:43Speaker 01

  But when there's a, like, there's one question in music that's like about the person that discovered there's a unique number of protons in every element. And so that doesn't have a named entity. And so the model kind of hallucinates proton as a named entity or as like an entity of some sort. And if you just start from there in the graph, you'll completely screw up the results, right? So this is a query that requires a lot of context and a lot of like contextualization of the entities in the query. And so we're working on finding a better way to map from the query to entry points in the graph. And that's something that a lot of people have worked on in the past.

• 49:27Speaker 01

  So we're kind of playing around with things that people have done. And I don't think it's an extremely difficult thing, but it's definitely a very obvious improvement that Hipporag will have to go through.

• 49:41Nathan Labenz?

  Yeah, one, so I guess starting to shift gears toward imagining like the dream system that can do it all.

• 49:48Speaker 01

  Yeah, let's go for it.

• 49:48Nathan Labenz?

  One thing that kind of jumped out to me as missing was like the sort of outward in search, starting with a, you know, the standard rag, right, is like some version of you have your passages embedded, you like either embed the runtime query, or people do like hypothetical answers that they then embed, whatever. But there's some sort of like passage to passage similarity search. So that was the first thing that I was thinking is like, why don't we also, we're starting, you know, with the entities, that's kind of starting at the core and fanning out. But why not also like take the query, look for a sort of the leaf nodes, the original like raw docs that sort of match that most closely, and then use those as another place from which to traverse the graph.

• 50:34Speaker 01

  Yeah, yeah, I think that's a great question. So like our naive, like very basic solution for this is to use ensembling, right? So you have a simple rag framework where you get the top documents based on your query, and you're using context and semantic embeddings for that. And then you have HippoRAG on the other side. So you can normalize the scores and then combine or average them to obtain like kind of signal from both frameworks, right? But that's like the naive, basic idea. The adding the nodes to your graph is something we thought about. And after working with PPR a lot and kind of having a feel for what works and what doesn't work,

• 51:16Speaker 01

  that will add a lot of density to the graph and make it much harder to do any sort of relevant traversal from it. So basically, one node will be connected to all the entities that happen in that node. And so if you give that node any probability, you'll just get like the probability distribution will start being too spread out. And so it won't really work too well. But I think like the most appropriate way to address those two things is with finding better ways to map the query to the graph, and then finding better ways to traverse through the graph. And then like also ensembling, of course. But I think in terms of changing the graph structure too much with that sort of document specific embedding,

• 52:00Speaker 01

  seems to not be the best approach. I think this also kind of links to how the Hippocampus is. Works directly because you don't really map like the actual full memory into the hippocampus and join it with a bunch of stuff. You map it, you represent that like full memory with a bunch of different sparse concepts, separated across the network. And that allows for better like associative memory pattern completion, something that we can like directly see in the hippocampus. So that's interesting.

• 52:34Nathan Labenz?

  So what's the problem that you're anticipating where if I do a direct document to document comparison and then that matches with however many entities and then that sort of creates like a very wide net? Is that a problem for accuracy or is it a problem for cost? Like when you are doing this with the current setup, how many tokens are ultimately getting pulled back and and put into the L L M for reasoning? And I guess I'm wondering like if I sort of relax that constraint, because one interesting data point on this is I've been using Gemini one point five flash recently to experiment with various ways of like post processing my email history. And I've had really good results with its ability to summarize, but it's summarize

• 53:19Nathan Labenz?

  feels like I'm selling it short. Basically what I've done, I've talked about this in another episode. So briefly, I take the last 250 emails that I've sent, which for me turns out to be roughly a thousand tokens per email, 250,000 tokens. And then ask Gemini one point five flash to write a character sketch of me based on those emails. And it does a remarkable job. It gives me back like a two page, you know, here's this person, what they care about, what they're involved with, the various projects, all from just this raw dump of 250,000 tokens. And it's funny, this just also goes to show how fast things are moving, right? You published this paper in May, I think flash was announced probably right around the time the paper came out, which means you didn't have access to it at the time

• 54:04Nathan Labenz?

  of doing the work. But that whole analysis from the 250,000 tokens costs under 20 cents and takes like 40 seconds or so. And so I I'm feeling like the power of these very long context, very cheap and fast language models is maybe that it allows us to cast a wider net through the graph. And maybe not have to worry about it as much. I mean, obviously, you can still blow way past a million tokens if you're, or even now up to 2 million tokens. You can blow way past that easily with a big corpus and a wide net. But anyway, I guess like, where were you when you did this? Project like, how many tokens did you allow yourself to ultimately consider? And, you know, how do you think that would change if I said?

• 54:48Nathan Labenz?

  Hey, it's worth it to me to spend a quarter, you know, if I can get the right answer. So if I accepted hundreds of thousands of tokens, how might that change things?

• 54:55Speaker 01

  Yeah, that's, that's an interesting point. So first, like, the, what I was talking about with the document in the graph part is that's an accuracy issue. Right? So if you want multi-hop reasoning to happen within your graph, then adding too much like density to it. Through adding a document node that actually allows the probability through it. You'll add too much noise to the, to PPR and it won't be able to do multi-hop reasoning in any significant way unless. Yeah, like in as it stands now, it wouldn't be able to do.

• 55:33Speaker 01

  As far as your other question with long context L L M's, I think it's very interesting how good these models are at synthesizing stuff from long context. So like in your framework, and honestly, I've seen this myself, like they can detect relevant things within that very long context. And use it effectively, but I am not super confident in their ability to get things that are not actually directly relevant. To what they're looking at, but only relevant because they did some extra processing in some area of the long context. In order to get that relevance. So like basically multi-hop reasoning within the prompt is still somewhat of a challenge.

• 56:18Speaker 01

  It's currently under review, but I was working on this project that used long context L L M's for retrieval. Like basically answering this question like, can we just use long context L L M's for this multi-hop question answering setting? And they are better than just. Embedding each thing separately. So compared to RAG, like RAG just can't do multi-hub. But if you add this long context LLMs, they'll be able to do the multi-hub to some extent. And we actually like see that in the attention patterns, but it's not perfect. And as it increases uh in length, visibility drops. So I don't know how much of this multi-hub process was happening in your two hundred fifty thousand documents.

• 57:03Speaker 01

  I think if you look at the synthesis of the summary that it created, I think you would probably be able to link each of the points it produced, one to one to some passage in your long context. I think that they might even be like doing some aggregation and summarization of multiple passages on multiple emails. But I'm not sure you're gonna find some extremely good consolidation of knowledge. And I think that's the barrier that I see with these long contexts or where they could improve, I guess.

• 57:37Nathan Labenz?

  Is that recent work done with the Gemini one point five or what were the?

• 57:42Speaker 01

  So we were using different models. So we didn't get to one million tokens, but we're doing like forty or eighty passages even with GPT four. So it's like somewhat long context, but even in the medium context, this is a problem. So in the very long context, I think it'll still be a problem. That's my assumption. I haven't like actually tried it with Gemini, but.

• 58:05Nathan Labenz?

  So it seems like there's got to be some way to bring back this. I mean, maybe you could say it's like not needed, but it feels like to me, I want to triangulate from kind of both directions. Like my intuition is I want to start at the entities and fan out, but I also want to see, is there something in this knowledge base that's a very high match for what I'm looking for? And just make sure that I also like pull that in and and use that as some sort of a starting point? I guess you could imagine that they could be like almost quite independent. Like maybe you don't need to fold that into the graph, but maybe you just like do a hybrid system where you sort of say, use HippoRAG and then use like simple RAG and just like pull in the top X documents

• 58:50Nathan Labenz?

  and then put all that into content, like augment what the HippoRAG retrieves with another form of retrieval and just like make that the LLM's problem. But how? Refund this with me, like, keeping in mind, you know, I'm willing to pay a quarter per query if I can really get it right.

• 59:06Speaker 01

  Yeah, if you can pay a quarter per query, like, basically, an LLM could do the graph entries probably very, very well. If we extract like thousands or tens of thousands of entities of potential nodes to enter through, or facts even, like, I think the facts might be even better in this scenario. You get a bunch of facts from the knowledge graph that you created, and then you have your query. And long context LLMs are clearly going to be good at finding the relevant facts within this huge pile of facts, even in a huge knowledge base. Even if you built it on the entire internet, I don't think you'll find like that many or like enough relevant facts to your query

• 59:51Speaker 01

  that you won't find in the top 10,000 programs. So if you have 10,000 facts, you can use Gemini to just find the entry point to the graph very well. And then you're going to also probably do graph traversal with the LLM fairly well as well, if you don't care about cost that much. So I think it is interesting to think about how to combine the idea of preprocessing your corpus with the power of long context LLMs and just LLMs in general to do self-correction and also like maintenance and kind of augmentation of this knowledge graph as you go.

• 1:00:35Speaker 01

  Because you also need a lot of that, like Raptor, MemWalk, MemWalker, Graphrag all do some sort of summarization of nodes. And I think that's probably also important in like a no bounds HippoRAG setting. I would definitely combine all sorts of ideas from those papers or basically with this like summarization and consolidation of documents based on some clustering. So, yeah, I'm by no means saying that like long context LLMs don't have a role to play here. Obviously, they have a role to play. It's just about like how exactly, how big that role has to be. And like, how far can we go without them is also an interesting question, I think. So, yeah, I don't know.

• 1:01:18Nathan Labenz?

  I'm very interested in Raptor as well. I also think about the AI Town Project, pretty often in this context, where, and again, it's sort of like, am I violating my no analogy rule here or not? But what they did in that AI Town Project is basically just periodically come through and create, I believe what they called reflective memories from the observational memories. So each little agent is walking around having these interactions and it's just recording all the interactions and observations that it has. And then it's like, kind of reminds one of what seems to happen when we sleep,

• 1:01:58Nathan Labenz?

  where this memory is like, post-processed and put into a more usable form, where again, like some of the low-level details are abstracted away. And this is where they start, I think they call it self-notion in that paper, where the agents start to, or I mean, they're instructed to, it's not like a spontaneous happening, but with the right setup, the language models are able to create sort of a sense of identity that kind of integrates these different episodes. And it's like, you know, this episode, this episode, this episode add up to, I like this person because of these like, multiple interactions that have happened. And I think that is a really interesting concept as well.

• 1:02:42Nathan Labenz?

  The Raptor one is less like, let's say, less agent-y and more just sort of, you know, takes raw passages, clusters them semantically, and then tries to like, create a sort of synthetic, like higher level summarization of those clusters. But then again, that's meant to sort of give you better ways into the graph, right? Yeah. You mentioned FLARE also, curious, that's not one that I've studied, so I wonder what you could tell us about that?

• 1:03:13Speaker 01

  Oh, I don't remember the specifics, but there's some forward-looking like framework that allows the model to determine if their retrieval was correct or when to do retrieval. I don't quite remember exactly, but yeah, it's just like i- in order to make the retrieval more relevant to the generation part through forward-looking. But the Raptor and the AI town thing. So one of HippoRAG's limitations is that it only deals with explicit knowledge. So any sort of things that are implicit or obtained from processing or summarizing your

• 1:03:54Speaker 01

  explicit facts in any way. So for example, you record that like Cinderella lives happily ever after, but you don't record that the ending to the story is happy explicitly, like tha- that's just something that you reflect on and you say, okay, yeah, this is a happy ending. And so I think HippoRAG would definitely need some sort of, okay, let's look at the graph that we created. Let's expand some nodes, create more relations, maybe delete relations that seem unnecessary. Kind of cluster happy ending stories together. Make nodes about like this, the ending of a story, but all sorts of

• 1:04:37Speaker 01

  interesting like post-processing or maintenance of the graph that could be done automatically with LLMs would be really interesting. And I think if money were no object and I was focusing on like how to build the best HippoRAG or the best RAG system out there, I would pursue something like an automatically generated semantic web using concepts from HippoRAG, concepts from Raptor, from Memwalker. And I think because the dream of the semantic web is is pretty great and we really haven't figured out a way to do that efficiently. And so I think LLMs provide a really interesting way to approach that that I think should be capitalized on.

• 1:05:20Nathan Labenz?

  It is an interesting phenomenon in general in AI right now that Like the path to the small efficient models seems to pass through in some cases, the large models like Gemini 1.5 flash is reportedly distilled from Gemini 1.5 pro, or maybe even, you know, the highly anticipated Gemini 1.5 ultra that nobody has seen, but it's reported that it is distilled from a larger model. And I often feel like even in the startup realm where people are more resourced to go after these sorts of things, I often feel like people are too concerned with cost and latency, but especially cost.

• 1:06:00Nathan Labenz?

  I mean, latency has a more like, in my view, like a little bit more disruptive impact on the user experience. Whereas if you can make it work well, typically I'm like, it's all pretty cheap. It's all cheap relative to hiring it done. And my biggest pain point with all the AI products that I use, almost invariably is that it gets it wrong. So I'm like, just make it work right, get me the right answer, charge me whatever it needs to be to satisfy that. And then, you know, you can ride the cost trend down, I'm sure. Or, you know, there may be sort of a similar thing with distillation, which would be like, if we take like some sort of maximalist approach and we're willing to spend a quarter, or even who knows, a dollar, right?

• 1:06:42Nathan Labenz?

  Per, I mean, some of these questions that I want to answer, like it would definitely cost me more than a dollar to hire it done. Like probably a lot more, you know, than a dollar to hire it done. So if I really care, I probably should be willing to spend a dollar. And if the trends continue in anything like the recent history would suggest, then the language model part is going to get a lot cheaper for free. And then maybe we can also use the system that's working to sort of, you know, then prune off of that or otherwise figure out ways to kind of make it more efficient. But it feels like maybe the path to that more efficient memory system goes through a more like maximalist system, where we're just like, let's give it every trick in the book and then kind of look back and see what tricks is it actually using in any given case?

• 1:07:26Nathan Labenz?

  And then maybe can sort of layer on heuristics or other efficiency. Method, any reflections on that or any coaching or you want to go start a startup together or what?

• 1:07:35Speaker 01

  Yeah, no, I think that's a good general notion. Like it doesn't matter how much it costs, just get it right. I honestly feel that that's kind of what they're doing, but it's just very hard to get everything right, right? Like a full generalization is still pretty much not achievable yet. But I think this idea of like full indexing, the internet is definitely worth pursuing. And I'm sure I kind of think that Google is doing something like this. But maybe their desire to compete with the super large language models is barring them from spending inference money on this better index, right?

• 1:08:15Speaker 01

  But yeah, I think there's like technological improvements to HippoRag that need to happen before we can actually use such a semantic web as well. And I think that's kind of the more important things to address right now before we just like throw the kitchen sink at it. Think about the big picture as you're building it, it's important. But because we have no idea whether it's gonna generalize well, even at the medium scale, we need to do that first, I think.

• 1:08:48Nathan Labenz?

  Any technologies that you would want to highlight based on your experience? I mean, I think people are increasingly overwhelmed by just like all the different data ingestion or like pipeline tools that are out there. I'm especially interested to know if you've experimented with any in the multimodal domain. Then there's like which embedding models do you use for things like this? Which vector database should you use? Which graph database should you use? I don't expect that you'll have like encyclopedic knowledge because I don't think anybody does in today's world. But any highlights, lowlights, you know, general findings, directions, tips would be much appreciated.

• 1:09:27Speaker 01

  Yeah, so unfortunately, I'm not the best person to ask about multimodal stuff. Cause I've been working basically exclusively in text for a while. But in terms of like embedding models, I was actually really surprised and impressed by Colbert. Just like as a small model that generalizes super well. I think it like combines simplicity of BM twenty five and old information retrieval. And like semantic and embedding methods really, really well. But if you want more like higher performance and you're willing to pay a little more. I think there are some other embedding models like. I was pretty happy with great LM from contextual AI.

• 1:10:11Speaker 01

  They kind of have an instruction tuned retriever, which is a neat idea. Where you can actually give it some instruction to kind of shape your embedding in a in one direction or another. And so if the corpus that you're embedding is kind of like documents about a specific thing. And you're gonna retrieve like mainly from that corpus in one specific way. You can give it that sort of instruction and then it is more likely to distill more of that semantic information into the embeddings. And so that's kind of nice. And it also has like normal generation. So it's like this hybrid model. That's pretty good, I think. In terms of like production level vector database, query databases and that kind of stuff.

• 1:10:55Speaker 01

  I think that's also something that is better suited for another person to answer. Since I don't work in like production level systems right now.

• 1:11:05Nathan Labenz?

  Okay, cool. How about on the chunking side? It just occurred to me, I appreciate this earlier, but I didn't have clarity on like. Are you chunking these documents before doing entry extraction? Or is there any chunking strategy built into HippoRAG?

• 1:11:20Speaker 01

  Yeah, so currently chunking was not really necessary because the passages in the benchmarks are kind of small. Like you have two to four sentences per passage. And that is very reasonable for three point five to figure out which triples are most important. And I honestly think that if you start making this much longer, all models start like freaking out a little bit. Cause you need to keep track of what you've produced and like where in the paragraph they come. And then you can also hallucinate relations between different sentences in the paragraph. And so I think it kind of becomes maybe a little bit too complex.

• 1:12:03Speaker 01

  And so I would advocate for chunking with at least the current version of Hipporag. And current L M L M is like, yeah, just a few sentences, a paragraph, maybe. And then like hierarchically summarize longer passages. But it's a really interesting question because there's a current work called Longrag. That kind of says, okay, let's just do knowledge consolidation through no chunking. So you just like have all of the semantically similar documents, let's put them together. And then just use an L M encoder to embed those huge document. And that's how they solve this like knowledge consolidation problem.

• 1:12:43Speaker 01

  And I think it's an interesting idea, but in order to do better, the smaller the text snippets, the better. Like attention you could place on different chunks. So I think there's a trade off there between a system like Longrag and a system that like Hipporag where you can have very small chunks, but then you can consolidate them or like connect the ideas between them. But like via pre-processing, but for now it's just like two to three sentences.

• 1:13:13Nathan Labenz?

  Okay, cool, helpful. Maybe last question, what other human capabilities do you see that you think appear to be unsolved in the literature? Obviously, there are probably multiple critical things that we can do that the the systems can't yet do, but I wonder if you have any in mind that you've sort of isolated conceptually that you think people should be looking into trying to design a system to to fill these kind of, you know, what I sometimes call them micro cognitive skill gaps that the A I systems still have.

• 1:13:46Speaker 01

  Yeah, so like. There's a fun paper from some people from MIT, like Nancy Kenwisher, Avlina Ferenko, that talks about how LLMs are really good at formal linguistic competency. So they're very good at phonology, syntax, local semantics, right? But then when it comes to, well, specifically world knowledge is what we're talking about with continual learning, basically. But also like formal logic and like social reasoning. And so I think the biggest competency gap apart from world knowledge that I see is formal logic.

• 1:14:26Speaker 01

  Like, I think the bias towards like S- System One reasoning, kind of like fast heuristic-based reasoning, instead of like more restrained, more restrictive, like constrained reasoning, is a huge problem for LLMs. And in the same paper, they talk about how like this problem seems to be resolved through modularity in the brain. So you have like, you definitely have a, a language circuit, right? That is mostly doing, or i- it's probably doing a lot of the System One, like, language-based reasoning. And then you have a bunch of other modules, like, and one of them mo- is more responsible for the System Two reasoning.

• 1:15:09Speaker 01

  And so I think there's also like the school of thought that this world knowledge and like formal logic competencies are just gonna arise from scale. And I think this is possible, but not guaranteed, in my opinion. And so adding some sort of inductive priors of modularity. So you have like different parts of the system that are architecturally different. Learning different things could actually accelerate this, the learning of more formal logic. And I think that idea, research-wise, is really exciting to me. And obviously, it's not quite clear how the architectures are, like, have to look like.

• 1:15:51Speaker 01

  But the idea that we need different architectures for different parts and learning to happen, like, using these two architectures or these multiple architectures is probably the best path forward, in my opinion, yeah.

• 1:16:04Nathan Labenz?

  Cool. That reminds me of one that I've been moving to study more deeply, which just came out of DeepMind not too long ago called Transformers Meet Neuroalgorithmic Reasoners. And if you haven't seen this one, it's definitely kind of a trip. It's like they have sort of a hybrid structure where there's cross-attention between a standard transformer. And then, on the other hand, and this is not something I've like even really gropped yet, but these neural algorithmic reasoners are like more structured, more like graph-type entities. And they're still trained things, but they're like, they're more structured, they're more sort of principled,

• 1:16:49Nathan Labenz?

  and they've been demonstrated to be good at solving certain kinds of logical problems. And so then trying to kind of merge these together with some sort of cross-attention paradigm seems like a pretty promising approach, but I can't say I've fully understood that yet or have an idea of exactly how I would apply it next. But for anybody who's inspired by what you just said, you know, that'd be one thing I would at least encourage you to check out.

• 1:17:13Speaker 01

  Yeah, there's definitely a lot of directions in this like big idea. Yeah.

• 1:17:18Nathan Labenz?

  Well, this has been fantastic. Thank you so much for the time. Anything else that we didn't touch on that you wanted to make sure we?

• 1:17:24Speaker 01

  No, I think we've covered a good amount. Thanks so much. This was really fun.

• 1:17:29Nathan Labenz?

  Cool. My pleasure. I love the work and definitely find it very thought-provoking and something that I can see building into any number of future projects. So I appreciate it. And for now, I will say, Bernal Jimenez-Gutierrez, thank you for being part of the cognitive revolution.

• 1:17:45Speaker 01

  Thanks so much.

• 1:17:47Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.