⚡️The Rise and Fall of the Vector DB Category

Note from your hosts: we were off this week for ICLR and RSA! This week we’re bringing you one of the top episodes from our lightning podcast series, the shorter format, Youtube-only side podcast we do for breaking news and faster turnaround. Please support our work on YouTube! https://www.youtube.com/playlist?list=PLWEAb1SXhjlc5qgVK4NgehdCzMYCwZtiB The explosion of embedding-based applications created a new challenge: efficiently storing, indexing, and searching these high-dimensional vectors at scale. This gap gave rise to the vector database category, with companies like Pinecone leading the charge in 2022-2023 by defining specialized infrastructure for vector operations. The category saw explosive growth following ChatGPT’s launch in late 2022, as developers rushed to build AI applications using Retrieval-Augmented Generation (RAG). This surge was partly driven by a widespread misconception that embedding-based similarity search was the only viable method for retrieving context for LLMs!!! The resulting “vector database gold rush” saw massive investment and attention directed toward vector search infrastructure, even though traditional information retrieval techniques remained equally valuable for many RAG applications. Full Video Episode Timestamps 00:00 Introduction to Trondheim and Background03:03 The Rise and Fall of Vector Databases06:08 Convergence of Search Technologies09:04 Embeddings and Their Importance12:03 Building Effective Search Systems15:00 RAG Applications and Recommendations17:55 The Role of Knowledge Graphs20:49 Future of Embedding Models and Innovations This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:02Speaker 00

  Okay, hi. So this is another Lightning Pod with Joe Christian Bergam. Is that, did I get it right?

• 0:08Speaker 01

  Yeah, I'm over in Norway. Trondheim, Norway, in the center of Norway, yes.

• 0:13Speaker 00

  What should people know about Trondheim?

• 0:15Speaker 01

  It's a small city. It's easy to get around. There's a great technical university here. The climate sucks a little bit, but it's easy to get things done in the winter, so.

• 0:25Speaker 00

  Yeah, yeah, yeah. I've never been over. I've been to OraDev, I think, which is over near you guys. But yeah, what we're here to talk about, just generally, your hot takes on RAG, search, vector databases, all that stuff. I think you've taken to publishing a lot more recently on X, and that's gone really well. So I'll just kind of go into that main thing that is that everybody knows you for, which is the, your piece on the vector databases, the rise and fall of vector databases. So maybe you could give us the background of like, why you felt compelled to write this?

• 0:59Speaker 01

  Yeah, first of all, I think I had to go a little bit back, right? So I have a long background in search and working on infrastructure for search. Like, I've been in search, working on search systems for 20 years.

• 1:13Speaker 01

  At Yahoo, company, also fast search and transfer here in Trondheim and Norway. And also working on embeddings, neural search, all of those things, right? Leading up until ChatGPT, the ChatGPT moment, like November 2022. And then there was some kind of cookbook, I think, from OpenAI where they said, okay, this is how you can do, connect ChatGPT with your data, and here's embeddings. And I think then a lot of developers, right, got into this, this is how we can build search, this is how we can do RAG. I think there was like this unnatural connection, meaning that between retrieval in RAG, that it had to be vector embeddings.

• 1:57Speaker 00

  By the way, I have a small role in that. I actually was the one who wrote the Chroma example in the OpenAI cookbook.

• 2:04Speaker 01

  You did? Okay.

• 2:05Speaker 00

  Yeah, I was an angel investor in Chroma before they became a vector database, and then I was just helping out.

• 2:13Speaker 01

  I'm actually a huge fan of Jeff and Anton from Chroma. I mean, I think Anton left, but I think they've done a great job at promoting retrieval for AI and infrastructure, and they did a lot of great things. So I really enjoy talking to them on X. Anyway. And then we had the whole vector database. I think Pinecoin was one of the pioneers framing it as a new infrastructure category. If you need to work on embeddings, you have to use a vector database. And naturally then, if you want to do anything in AI, then you need to have a vector database. And that was my primary motivation for writing that piece and looking a little bit back, you know, what happened and where we are now and how I see it.

• 2:58Speaker 01

  And yeah, so that was the pure motivation.

• 3:03Speaker 00

  Okay. And the general thesis, I guess, if you wanted to sort of recap that, like, you know, I think it's a very fast rise and fall, like Pinecone was a dominant player for a long, long time. And, you know, I don't know my exact sources because there's a lot of rumors going back and forth. But apparently they went up to like a hundred million AR very, very quickly. They raised a big round and then suddenly a lot of people started leaving. Like suddenly it went from cool to uncool very quickly and I don't understand why.

• 3:32Speaker 01

  I don't understand that either. And I think also they repositioned a little bit, going back to their core messaging. If you go to their website now, it looks more developer focused. It's not the memory for AI. It's not like enterprise-ish. It's more towards developers now. So I try to think that they were trying to go back to their original roots. I think that's a good thing. But also, of course, there's been a lot of competition in this space. A lot of new companies, one of the upcoming stars is TurboPuffer, kind of same SaaS model, a little bit different pricing, and they really talk to developers. And I'm not saying that the companies are dying, right?

• 4:13Speaker 01

  I'm just saying that the separate infrastructure category is dying, right? Because you have vector search capabilities in almost any DB technology nowadays, right? And you have it also in more traditional search engines like Elasticsearch, Solr, Vespa.

• 4:33Speaker 01

  So I think there's like convergence on on features on both parts. So and then you have things like PG vector in Postgres. A lot of people, you know, get confused. Okay, I have already a DB. It has vector search. Why do I need another DB like a vector DB? So the whole database concept. So I think those companies, I mean, there are lots of great technology here. Don't get me wrong on that, but I don't saying that the companies are dying, but I'm saying that the category is dying. And so there's there's this distinction. And I think a lot of people like oversaw that and like came at me and say that, you know, because they had some kind of hate around some of these companies. And they said, yeah, you know, go fuck bank or whatnot, right?

• 5:17Speaker 01

  But I actually say that the category is dying and I'm actually want to call these new companies that they are like search engines. And I want to go back to the natural. I think that's a more natural abstraction for connecting AI with knowledge and all the arguments for doing rag. I think the natural concept there is search. And I think one of the insight I have from I use Winserve a lot. I love Winserve. They're like cascade mode. And if you ask it like, what are the tools you have available and like list like seventeen, eighteen tools like edit files? But there's also like things like search code base, search the web, grep. And these are like search abstractions, right? And I love that idea where you like just connect the reasoning model with

• 6:02Speaker 01

  with these tools that are essentially search tools. And that can help the agent or the LLM to actually formulate the query. You know, should I do a grep? Or should I do a more of a semantic search? Or should I do more keyword search? Or should I just search the web? So I think that's more like the natural abstraction. Instead of jumping into, you know, vectors and how you represent, that is more of like a detail like of how you implement search.

• 6:28Speaker 00

  Yeah, it's interesting that we we fixated a lot on on vector, like, you know, dense abidings and all that. And I think now we're we're sort of broadening out. I will also mention that Chroma, I think from the start, has always said they're kind of going after information. Retrieval and not so much, not so much, you know, the narrow sense of rag. Yeah, I think like broadly, this is the consensus that, you know, like the category was was never really going to be lasting for that long. It was just, there was just a brief period of time from my, one of my favorite early tweets in AI.

• 7:07Speaker 00

  You know, this post GPT phase was I summed up all of the fundraising that happened in vector databases for

• 7:14Speaker 01

  Wow, yeah.

• 7:16Speaker 00

  And it was something like 230 million and all put into all the vector databases. And that was more than the entire lifespan fundraising of MongoDB.

• 7:27Speaker 01

  Right.

• 7:29Speaker 00

  So like basically they cannot all win because they they've already taken more money than supports, you know, one of the de facto winner companies in NoSQL.

• 7:39Speaker 01

  Yep.

• 7:40Speaker 00

  Interesting.

• 7:41Speaker 01

  Yeah, I think also on MongoDB, right, they they brought our new category in NoSQL. Right. And but nowadays, all the other database players have also caught up, right? So now even MongoDB has relational SQL, right? So there's always like this convergence, but MongoDB kind of it sticks. But I don't think that for Pinecoin that was originally leading that movement, it won't like stick in the same way. It's too narrow. It's too narrow. Yeah. But I would like to say one more thing about embedding. So people like, okay, Joe, but you know, embeddings is really important. And I'm also think that embeddings is really important, right? Because you can represent more data than ever before, right?

• 8:24Speaker 01

  Multi-modal, whatnot, run into a neural network, get an embedding representation. And then you can move this embedding representation around in vector space and adjust to your domain or whatever you're doing. So it's really important. But what happened was that it went mainstream, right? It went from these big tech companies like Google, Yahoo, Facebook, all of them, you know, been working on embeddings for a long time for a lot of different tasks, right? But with

• 8:52Speaker 01

  Post ChatGPT and we got the embedding APIs from OpenAPI. It suddenly became mainstream, like every developer would start using embeddings, right? And what to do with them and similarity search and so forth. So, so I'm, I'm not against embeddings, like embeddings are here to stay. It's just that it's only, it's not only about similarity searches in this kind of embedding space. But, and then I think more people actually realize that, you know, you actually need something more to it than just embedding and a cosine similarity to do search well. Like things like freshness or authority and all of other signals, right? That, that really plays into role in web search.

• 9:35Speaker 01

  And I, I remember one of the OpenAI guys wrote like, you can embed the whole web and then you can build the next generation web search. And I thought like, okay, just looking at semantic similarity, that's not going to play out too well. So, yeah.

• 9:49Speaker 00

  I mean, they're trying to sell you their, their model, right? So what I'm gonna do is say those very hypey things.

• 9:56Speaker 00

  Yeah, I mean, the, the way that I put it is always, you're always going to want to do a hybrid query. You always want to add metadata and like do all that stuff. I think my question to you is maybe a very ageless question, which is, should they all be the same system? Right? Your search system, like Elasticsearch is typically like you duplicate your, whatever your main

• 10:18Speaker 00

  storage of record is, and then, and then you have that search index that is basically almost a complete duplicate. Like you just copy over the documents. Do you believe in that? Do you think there's a convergence here?

• 10:27Speaker 01

  This is a fantastic question. I think for a lot of use cases, right? And if you're already using some database like Postgres, right? It has this great extension, PG vector, right? And I know that I tweeted things about PG vector that was true in the start around the limitations of PG vector. But there was a rally around P G Vector, like adding new algorithm, going introducing actually two algorithms, both E B F and H N S W, adding half vec, adding binary vectors. So actually, what you can see, P G Vector is doing more in the capabilities of vector search than some of the real vector database players, right?

• 11:12Speaker 01

  So if you're only looking at like vector search capabilities and you already have your data in Postgres and you're operating at a reasonable scale, I think it's fair to use.

• 11:24Speaker 01

  Postgres. Sorry? I think it's fair to use Postgres or use one database if you're like operating at, you know, operating at a really large scale and you do, you know, some vector search related workloads and you also use a database for other types of workload, then it might make sense to just keep the data there. But if you're actually building something that really depends on search quality and your business depends on it, yeah, then definitely I think that you should consider, you know, actually using a real kind of retrieval or search engine to represent the data there, right?

• 12:02Speaker 00

  Yeah. And is the search system, how closely entwined is Rexus and search in your mind?

• 12:08Speaker 01

  Yeah, and that's the thing with embeddings, right? Embeddings, I think with embeddings and embedding based retrieval, because embedding based retrieval has been used for a long time in recommender systems, like large scale recommender systems, like, you know, TikTok or Yahoo News or things like that.

• 12:27Speaker 00

  Apparently TikTok published their Rexus recently, which is kind of interesting.

• 12:32Speaker 01

  Yeah, it's like a cascade of, there's always in this system that operates at a really large scale, there's always a cascade of different stages where you first have to retrieve over the candidate pool and it's typically using embedding based retrieval. And then you have layers of reranking layers that finally you end up with hundred candidates or something like that, that you actually present to the user.

• 13:00Speaker 01

  So I think definitely there's convergence in that embedding based retrieval is also no more common for search systems. So there's convergence there on how it actually sold on the technology spectrum.

• 13:18Speaker 00

  Yeah, yeah. Any other thoughts on like,

• 13:22Speaker 00

  I guess the confusion for a lot of folks who are newer to this, right? Who are, they understand now that you cannot just have embeddings only and cosine similarity only. It's just the sequencing of like, what should I do first? What should I do second? What should I do third? Everyone says like, you know, re-ranking is like super important, but that, you know, it adds like maybe like three to 4% to your results. And maybe that's like the lowest hanging fruit.

• 13:53Speaker 00

  So I'm always trying to figure out like, what should I recommend to people, right? Like that they should start with, like, be, you know, like a Postgres or MongoDB as their transactional and in the end vector store. Then they could split it out to maybe use Elasticsearch or Vespa. I don't know if that was, that would be the recommendation there. Redis, I think, is also trying to push themselves there very, very hard. And then you add the Rexus. Is that, is that a good sequence?

• 14:22Speaker 01

  Well, I think it's, it's really hard to come up with general recommendations, like without knowing what you're doing. But if you're like looking to build a RAG application, like I think most people are interested in some, something related to RAG, right? When you have some data and you have to transform your data. I think, I think first, I think it's in Hamal that always talks about look at your data. Or, you know, everyone's talking about look at your data. So first of all, you know, how to get your data in a cleaned up way? If it's like PDFs or whatnot to do that, there's things there. I think actually that a very strong baseline is the classical BM25, like algorithm that's been around for 30 years, right?

• 15:03Speaker 01

  It's keyword matching, but it offers a very useful baseline for a lot of different search use cases. Because it gives you that baseline, right? Then you can start looking at using an off-the-shelf embedding model to also embed a model and all of the engines, more or less, most of the engines have some kind of hybrid search capability, start to play with that. And then if you can afford it, both from a latency perspective and a cost perspective, you can look at adding like a re-ranking layer on top of that. How you stitch that together uh depends on, you know, your do your framework of choice. But I think most of you, you can, you can stitch this together, like with multiple different APIs,

• 15:46Speaker 01

  depending on uh your budget, I guess.

• 15:51Speaker 00

  Yeah, you know, I, I, I always tend to recommend people to do this offline as much as possible, like batch online, whatever. Most people don't need fully online systems.

• 16:00Speaker 01

  Yeah, that, and that, that's a friction point because I've been used to working on kind of constrained online systems, like at, at pretty significant scale. I mean, it's always like, everything is online, needs to be a low latency. And then I have problems adjusting to, you know, when you wanna do things at a, at a much lower scale. So I'll give you an example, like calling out to some kind of embedding API to get JSON floats. You know, it's not something, you know, you want to do if you're running at thousands of QPS. You don't wanna add that dependency. You wanna have something local, something that is faster. So I've always been like, okay, I'm gonna call out to this endpoint.

• 16:42Speaker 01

  It's gonna take three hundred milliseconds to get this large float. It's, it's something that I like, oh, shrug. But now I, I'm shifting towards more, you know, it's easy, it's an API-based service. You don't have to think about it. It's just there. So it's much easier to build from, right? To have something that is API-based. So I'm trying to embrace that my, my.

• 17:05Speaker 00

  I see, I see. Um, no, so when I say offline, I mean more like, um, not in the critical path, like batch.

• 17:13Speaker 01

  Yep.

• 17:13Speaker 00

  Systems because, yeah, a, a, and it's interesting. I, I don't know if you've looked at Postgres ML for running the models alongside of the database. Are you, are you bullish on that kind of stuff?

• 17:23Speaker 01

  No, I'm not. I'm sorry. I I I'm not. I I think this is, yeah, we also seen other players that tries to, you know, move a lot of the logic into the database, agentic embedding inference and whatnot, L L M's. I think it's the right direction is to keep infrastructure a little bit separate from that because they're like different scaling properties. I I think people can stitch those two things together instead of trying to do everything with one single platform. So no, I'm I'm not bullish on that because I don't believe in the developer experience of writing like these huge SQL statements for transforming data from this and then embedding it and then writing it back

• 18:03Speaker 01

  and expressing this in databases. Like, what does this do to my database? Is it like calling out? Is it, what's going on? I I I I tend to want to have more control over cost and performance and what's going on than just writing some really large SQL to to execute.

• 18:21Speaker 00

  Yeah. It's interesting. I I think there's this constant tension between what should live in the database versus what is an external system. I don't think it's a clear cut, like, you know, a classic, it's like the the the cron service which we have in in Superbase. Okay, so cool. Like, any other, like, hot takes or, you know, what are the biggest criticisms that you got after you published this? You know, like, you know, what do you agree with? What do you, what do you disagree with? You know, just.

• 18:45Speaker 01

  Yeah, I think one of the things that people pointed out, if something goes semi-viral after a few days, you discover that there's a lot of replies that you didn't see and you're like, okay. But I think one of the things that stood out was that people said that Joe is saying that RAG is dead because of vector database infrastructure is dead, right? And I think that was a misunderstanding as well. And I think that comes from people making the connection between RAG and vector databases, like, it's so strong. So when I'm saying that the vector database infrastructure category is dead is like, okay, RAG is dead. And I I think that RAG is definitely not dead, right?

• 19:23Speaker 01

  Um, augmenting AI with retrieval or search is still gonna be relevant. And I think it's gonna be relevant for a very long time, right?

• 19:35Speaker 01

  So that was one of the things.

• 19:38Speaker 00

  I mean, the.

• 19:39Speaker 01

  And I saw that, you know, now we have ten million model, longer context, and you have the same cycle. Repeat.

• 19:48Speaker 00

  Every time, every time. I'm just like, you know, for me, you know, I I put out this like cryptic tweet. I was like, um, you know, this is Lama four is going to reignite the long context with respect debate. But it it will actually resolve the debate, but not in the way that you want.

• 20:02Speaker 01

  Yeah, I saw that. This is too cryptic to me.

• 20:07Speaker 00

  No, it's just like, there's, there's, there are like five other guys like saying, oh, like, you know, long context skills rag, like R P rag. I'm just like, guys, you you are, you're idiots or like your your engagement farming, basically, like a lot, most, most likely. You believe like they know what they're doing and they're just saying nonsense, just, just to, just to have fun. And like, they're like people who don't want to take them seriously.

• 20:29Speaker 01

  Yeah. Now, but that, but also it's nuanced to this, right? So I've seen people. Do rag when there is no need to do rag, meaning that, you know, if you have one PDF. Like with visual information and things and you want to chat with that, definitely that case is probably like that if you don't have like high QPS and things like that. So. I think there's nuances around this, that there's definitely, I had a call with someone that had like three hundred articles and I said, you know. This will just fit into the context window of one of these Gemini models. You don't have to have a vector database for this case. And they were so surprised when I, that when I said this. Oh, can you really do that? Yeah. But it's also, look at it. It's like just.

• 21:15Speaker 01

  We had like four K context window, right? And now ten million, right? So. And that's fast, right? And then people are still running their. Initial demos from early January two thousand twenty three, right? So where you were dealing with four K or eight K, right? So. So some parts of it is still not, is is not relevant now because we have longer context windows, but I think retrieval, of course, it's gonna be there for a long time. One example I love to bring up is like one of these small toy data sets from. Trek called with is like hundred and seventy thousand documents and it's already thirty six million tokens and you're not gonna load all of, all of that, you know, for a single query.

• 21:56Speaker 00

  Yeah, awesome. Um, do you have a take on knowledge graphs and graph reg?

• 22:01Speaker 01

  I think that the graph rag, well, I have a lot of takes around it. I think that one issue, I mean, graph databases is a database that kind of solves one particular problem. And it does it well to traverse the edges in the graph and, you know, random access and jump across. But the core issue is actually to build the knowledge graph, right? The entity is the relationship. So if you say graph, graph, you know, databases or graph rag is going to kill vector rag and all that discussion. I think the first issue is to actually build the knowledge graph the first place, right? And if you use a search engine or a dedicated graph DB to actually speed up and accelerate the searches.

• 22:46Speaker 01

  Okay, fine, but I think people like, okay, if I'm gonna do graph rag, then I need a graph database. And I hate that connection between doing something and then connecting it to some specific technology. And I think a lot of people do that, right? You jump from some concept into some technology. You can also do graph exploration with a search engine, right? You can, so you don't need a specific technology to do it. And can graph rag be better than vector rag? Yeah, for sure. In some cases, it might make sense or hybrid or whatnot. But I think people get caught up in some specific technology all the time.

• 23:27Speaker 00

  Yeah, but like, I, I think that's okay. But I'm still trying to validate the presence of knowledge graphs in LLM applications. Because obviously with LLMs, it is much easier, better to create these entity triplets and all that. So theoretically, it should be better.

• 23:42Speaker 01

  Yeah, yeah.

• 23:42Speaker 00

  I mean, in the past, knowledge graph has been a dirty word, but now maybe it's not.

• 23:47Speaker 01

  Maybe, maybe, maybe, yeah, maybe. I think with LLMs, you can do a lot more things around data generation, you know, in general. So generating those triples is a bottleneck, right? It's been a bottleneck. And now you have a LLM. So I agree, you know, now it could be easier to actually build what matters, right? Which is those triplets.

• 24:07Speaker 00

  Yeah. Okay, awesome. Any other opportunities that you find? I, I know that you, you mentioned Gina. I think they're a prominent European startup in, in rag.

• 24:17Speaker 00

  And then I think over here Voyage just got acquired by Nvidia. You know, anything on the embedding side, like, do we need a lot better embedding models? Do we, is what we have from the big labs good enough?

• 24:30Speaker 01

  Oh, I, I hope to see. I mean Voyage was really leading the pack on doing domain-specific embedding models, like legal PDFs. And what I want to see is more embedding models in that direction. Where you essentially, you know, represent this PDF as an embedding or multiple embeddings, you know, for legal domain or finance or health. I, I hope to see that grow so that you can have a better starting point than just those text models. And I've been a huge believer in using visual language models as a backbone for embedding models. Where you essentially take a screenshot of a page, you don't have to go through OCR.

• 25:15Speaker 01

  So you, you, you then get a much richer representation. You don't have to go through these complex processing pipelines. So I hope to see more innovation. I'm not sure if it's going to happen because I think it's a difficult business model to be in. Like, because you have to have an API-based service and you have to do batching and you have to make up for the compute. And then, you know, are people willing to pay for it? And, and I think maybe that's why Voyage got acquired. I think also Gina is doing a lot of great things in this space now, especially in European languages. But I think every company is trying to move up in the value ladder, right? They want to move into enterprise search or move into a different direction. So, yeah, but I, but I do hope that we will see more and better, like, general embedding models.

• 26:01Speaker 00

  Yeah, yeah. Um, I mean, I'm sure, I think the Voyage guys are very happy because it seems like they got acquired for, for a lot.

• 26:07Speaker 01

  Yeah, okay.

• 26:09Speaker 00

  So, okay, cool. Anything else before we we wrap? Any calls to action, any, you know, parting rants on the topics of the day?

• 26:18Speaker 01

  No, I would like. I mean, if you want to connect with me, you know, for the audience, you can find me on X. I'm under the handle Joe Bergen there. So I love a show that shouts out on on X. So I hang out there quite sometimes. Yeah.

• 26:30Speaker 00

  Yeah, I mean, it's it's where the AI community is, you know, although I've been, I'm always trying to like grow on LinkedIn or YouTube. I mean, you know, there's a lot more people there, you know, there's Twitter, it's just this like echo chamber.

• 26:43Speaker 01

  Yeah, but it's it's not the same. I mean, access. I mean, we wouldn't have this meeting, me and you, without X there, right? So it's a great place for really high signal to noise. And I think the AI community there is really great. So.

• 26:57Speaker 00

  Yeah, awesome. Well, thank you.

• 27:00Speaker 01

  Thank you so much for having me, Srix. It's been awesome.