Improving search with RAG architecture with Pinecone CEO Edo Liberty

Accurate, customizable search is one of the most immediate AI use cases for companies and general users. Today on No Priors, Elad and Sarah are joined by Pinecone CEO, Edo Liberty, to talk about how RAG architecture is improving syntax search and making LLMs more available. By using a RAG model Pinecone makes it possible for companies to vectorize their data and query it for the most accurate responses. In this episode, they talk about how Pinecone’s Canopy product is making search more accurate by using larger data sets in a way that is more efficient and cost effective—which was almost impossible before there were serverless options. They also get into how RAG architecture uniformly increases accuracy across the board, how these models can increase “operational sanity” in the dataset for their customers, and hybrid search models that are using keywords and embeds. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @EdoLiberty Show Notes: (0:00) Introduction to Edo and Pinecone (2:01) Use cases for Pinecone and RAG models (6:02) Corporate internal uses for syntax search (10:13) Removing the limits of RAG with Canopy (14:02) Hybrid search (16:51) Why keep Pinecone closed source (22:29) Infinite context (23:11) Embeddings and data leakage (25:35) Fine tuning the data set (27:33) What’s next for Pinecone (28:58) Separating reasoning and knowledge in AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Hi listeners and welcome to another episode of No Priors. Today, Alad and I are talking with Ito Liberty, the founder and CEO of Pinecone, a vector database company designed to power AI applications by providing long-term memory. Before Pinecone, Ito was the Director of Research at AWS AI Labs and also previously at Yahoo. We're excited to talk about the increasingly popular RAG architecture and how to make LLMs more reliable. Welcome, Ito.

• 0:30Speaker 01

  Hi.

• 0:30Sarah Guo?

  Okay, let's start with some basic background. Can you tell us more about Pinecone for listeners who haven't heard of it? Like, what does it do and how does it differ from other databases?

• 0:40Speaker 01

  So Pinecone is a vector database. And what vector databases do very differently is that they deal with data that has been analyzed and vectorized, I'll explain in a second what that means, by machine learning models, by large language models, by foundational models and so on. The most large language models or foundational models, actually any models, really understand data in a numeric way. Models are a thematical objects, right? And when they read a document or a paragraph or an image, they don't save the pixels or the words. They save a numeric representation called an embedding or a vector.

• 1:23Speaker 01

  And that is the object that is manipulated, stored, retrieved, and searched over and operated on by vector databases very efficiently at large scale.

• 1:36Speaker 01

  And that is Pinecone. When we started that category, people called me concerned and said, "What is the vector and why are you starting a database?" And now I think they know the answer.

• 1:49Speaker 02

  How did you think about this early on? Because you started the company in 2019. At the time, this wave of generative AI hadn't happened quite yet. And so I was wondering what applications you had in mind, given that there's so much excitement around Pinecone for the AI world. The prior AI world had a slightly different approach to a variety of these things. And I'm just curious, like, were you thinking of different types of embeddings back then? Were you thinking of other use cases? Like, what was the original thinking in terms of starting Pinecone?

• 2:17Speaker 01

  The tsunami wave of AI that we're... Going through right now didn't hit yet, but in 2019, the earthquake had already happened. Deep learning models and so on have already been grappled with. Large language models and transformer models like BERT and others started being used by the more mainstream engineering cohorts. You could already kind of connect the dots and see that where this is going. In fact, before starting, Pineco and I myself had found our anxiety between are we already too late versus nobody knows what the hell this is and we're way too early.

• 2:55Speaker 01

  And it took me several months of like wild swings between those two things until I figured maybe the fact that I have those too early, too late mood swings maybe means it's exactly the right time.

• 3:06Sarah Guo?

  Maybe, Ito, you can actually just expand a little bit about, you know, in what what use cases people want to use embeddings, right? I think there are ways to interact directly with language models and then reasons for, for example, reliability or context length like that people and performance that people interact with embeddings in like a RAG architecture or in semantic search. So maybe you can sort of talk about some of the driving use cases.

• 3:35Speaker 01

  I mean, the obvious way, in some sense, to add knowledge to your conversational agent, whether it's chat or what have you. We talk about it as generative AI now, but it's much more general than that. Is to, again, not shockingly, bring the relevant information into the context, right? So that you can actually arm the foundational model with with the right pieces of content, with text, with images, with what have you, right? You want to be able to retrieve that from a very large corpus of knowledge that you have, whether it's your own company's data or whether it's the internet or what have you.

• 4:16Speaker 01

  It so happens that LLMs are already very, very good at representing data in the way that they want to consume it, which is these embeddings. And so you can, at question time, in real time, okay, at the time of the interaction, go and find relevant information. And relevant might be associated with, or called with, or something that is similar to whatever it is that you're being asked about. And once you bring that into the context, you can now give much more accurate answers, right? And as a side experiment, we actually loaded, it's called Common Crawl, which is the top

• 5:00Speaker 01

  internet pages crawled fairly frequently. We loaded that into Pinecone and saw what happens when you augment GPT 3.5 and 4 and Llama and Mistrall and models from Cohere and Thropic. And you could see that if you augment all of them with RAG, even on the internet, which is data that they were trained on, you can reduce hallucinations significantly, up to 50% sometimes. Interestingly enough, many of them actually start behaving quite similarly in terms of level of accuracy, even though without RAG, they actually have quite different behaviors. So it's sort of both like a uniform improvement and a little bit of leveling the playing field.

• 5:43Speaker 01

  Now, you know, because we know we can do that very well now, now you can do that also with proprietary data, with data inside your company and so on, stuff that, of course, is not available on the internet and stuff that those models were never trained on. And interestingly enough, again, the quality ends up being incredibly high.

• 6:01Sarah Guo?

  I assume most Pinecone users are not, you know, using LLMs and retrieving against general internet data. Like, what kinds of companies were your earliest or biggest users? Like, what kind of data do they want to retrieve against?

• 6:13Speaker 01

  So most companies do use their own company data. It could be whatever it is, depends on the application they're building. Could be legal data, medical records, internal wiki information, sales calls, you name it. There's an infinite variety. I want to say that this is just RAG. I mean, this is just semantic search. I mean, there are many other applications that we didn't talk about, but we can keep it focused on this application for this conversation.

• 6:42Sarah Guo?

  And is it dominated by a specific use case? Like, were there customers that you feel like really represent the Pinecone use case well?

• 6:49Speaker 01

  Yeah, 100%. First, text is probably most of what we see. Nowadays, models are really good at images and so on, but text is still the predominant data type. Notion Q and A now runs on Pinecone and they serve essentially question answering with AI to tens of thousands and probably hundreds of thousands of their own customers. Gong does the same thing with sales calls, again, serves all of their use cases for all of their customers and so on. So one of the most common patterns is companies that themselves become trailblazers and innovators

• 7:28Speaker 01

  with AI and they themselves hold a lot of their own users or customers' text and they want to search over it or generate information on top of it. That ends up being an incredibly common pattern.

• 7:42Speaker 02

  I guess earlier this month, one of the things that Pinecone announced was the serverless offering called Canopy. Could you tell us a little bit about why you decided to go down the serverless direction and how you view that in terms of either use cases or adoption or other things?

• 7:57Speaker 01

  So Canopy is actually an open source that we put out there as a framework for people to learn how to use RAG. Pinecone serverless is just going Pinecone. It's just Pinecone, but serverless. What it does is basically removes the limits from what people used to experience before. When we started Pinecone, a lot of the applications had to do with recommendation engines and anomaly detection and other problems where usually the scale was actually fairly small and the requirements had to do with super low latencies and sometimes high throughput. And as a result, you still see a lot of databases kind of.

• 8:41Speaker 01

  Play in that field. We very quickly figured out with our own customers and our own experimentation that something else is much more significant with just scale and cost. If you want to be able to answer correctly, you just have to know a lot. If you want to do that, you have to ingest hundreds of millions, billions, sometimes tens of billions of vectors into your own, into your vector database. And you want to query it efficiently in terms of cost. You just don't want, you know, you don't want that to explode in terms of, again, spend. And finally, you want to do that easily. So you don't want to spend weeks and months setting things up and getting it to work.

• 9:24Speaker 01

  And doing that in our old architecture and frankly, with any other architecture today that's not serverless is, is very difficult. And serverless is here to basically resolve those main problems. It's incredibly easy to operate. It scales massively. I mean, again, there's no theoretical limit to how much it can scale. We've tested it with tens and tens of billions with live customers in live traffic. And I'm not going to go into the architectural design, but it's actually designed to be incredibly efficient, like asymptotically better than, than, you know, what can be done with any other architecture. It's fundamentally about removing the old limits so people can actually have all the

• 10:09Speaker 01

  information they need, ready for the foundational models.

• 10:13Sarah Guo?

  You mentioned Canopy is to help enable more people to build rag products. Like where do you, where do you see developers or your customers struggle to get embedding space to AI products generally successful? Or what were you, what were you trying to achieve with, with Canopy?

• 10:28Speaker 01

  Yeah, so vector databases and pine cones specifically are very foundational model, are very foundational pieces of technology. We're, we're very deep in the stack and to build a You know, a proper, full end-to-end solution, say like Notion Q and A, there's quite a lot that you have to build on top of it. You have to ingest documents and what's called chunk them. You have to figure out how to break them into like factoids and pieces of information. You have to embed everything with models. You have to ingest them into the vector database. You know, when you get a query, you have to figure out how to manipulate it and how to embed that. You have to search over it. You have to re-rank. You know, there's a lot. There's a whole system you have to build around it.

• 11:13Speaker 01

  And a lot of people told us that this is actually quite complex and they're right. Right? We put out Canopy as really an example. It is an end-to-end kind of cookbook. If you just take this, it should work. You should probably, once it works, you should figure out how to make it better for your own application. Right? Because, you know, medical data is not Jira tickets, you know, and Jira tickets are not Slack messages. And you might be building a different product, but at least you have some end-to-end starting point that already does something and you can start improving on.

• 11:49Sarah Guo?

  Two of, I think, the most common comparison points for vector databases that people use are A, like traditional databases, right? Like, why not just use Postgres, PG Vector, or some index associated within an existing database? Or B, like sort of more traditional search or incumbent search technologies or services like Elastic or Alkalia? Can you, can you talk about like, you know, why not other databases? Or like how you think about traditional search?

• 12:18Speaker 01

  Yeah, I'll just go back to the fundamentals about what are you trying to achieve? Right? And what we're trying to achieve is to give as much context and as much knowledge to foundational models as possible. Do that easily at scale, you know, on a budget. Get to a unit economics that actually works for your product, which is incredibly hard to do with AI, with like many discussions going on about that now.

• 12:47Speaker 01

  Those other products don't work. They don't work either because they don't scale in terms of the efficiency, scale, cost, the trade-offs that they can offer. Because they're not designed to do this. They're designed to do something else. They kind of thought about vector index as a bolt-on. You know, retrofitted feature. And so, yes, it works at small scale. But when you try to actually go to production with it, you understand the limitations. With other search technologies, this is again, this is the wrong search mode. If you're searching with keywords, you're just not finding the relevant information because the embeddings, the contextual space

• 13:32Speaker 01

  in which these pieces of text, documents or images live is in vector space, in high dimensional numeric space, not in keyword space. And like everyone that's ever searched their inbox for an email you know for a fact you have and not find it, knows that keyword search is a deeply flawed retrieval system.

• 13:54Sarah Guo?

  I'm just curious if customers or, you know, developers are trying to combine the existing search systems they have. I know you also are increasingly supporting hybrid search. So kind of wanted to understand that. Where are embeddings like amazing and useful and like delivering new experiences? And whether or not enough or not like the full experience that end users want?

• 14:16Speaker 01

  So it's interesting. Our research actually shows that when you do this well, you very rarely need keywords alongside embeddings. But getting embeddings to perform perfectly is actually, it could be quite intricate. And we find that it's very convenient to have keywords alongside embeddings and to score those things together. We call this hybrid search. And in fact, we made this even more general. And we said, okay, why not? You know, keywords under the hood are actually represented as sparse vectors. That's true of any keyword search, by the way. This is not, this is just kind of mathematically identical. And then we said, why don't we just make this more general?

• 15:01Speaker 01

  And just say, hey, you can give either sparse or dense vectors or both of them and kind of have the best of both worlds. And people find that very convenient. And so I'd highly encourage people to look at it and improve, you know, by boosting and all sorts of other tricks that you can bake into sparse vectors, including keywords. My guess is that that's not going to be the dominant mode of search in the very near future.

• 15:29Sarah Guo?

  You think we progress, like you think hybrid search is a more temporary convenience?

• 15:33Speaker 01

  I mean, I think it'll be used for boosting and other types of levers to control your search. I think the mode of you baking keywords into that is going away. Yes.

• 15:44Sarah Guo?

  And maybe just going back to like the traditional database market, like why not in my Postgres or my Long or whatever I'm using already?

• 15:53Speaker 01

  Again, I mean, we see this in the market. A lot of people tell me, hey, I already use tool X or database Y and why not? And frankly, oftentimes when it's some tiny workload or just learning how to use embeddings for the first time and so on, it might actually work. Okay. It's when people try to actually do something in production, they're trying to scale up, they're trying to actually push the envelope, or they're trying to launch a product that needs to have some unit economics attached to it that makes sense for the for that product. That's where people run into huge problems. And so many of them just, you know, start with us to begin with, to be honest, a lot of them are enthusiasts and they actually kind of enjoy learning how to use a

• 16:38Speaker 01

  new kind of database and are you, you know, user experience is smooth enough and, you know, so many tutorials and notebooks and examples that they actually find it exciting, but I guess some don't. And that's, that's, that's fine.

• 16:51Sarah Guo?

  So maybe one more on database dynamics. Pinecone is closed source. It's gotten great adoption, but many databases and like, mature market are open source. How'd you think about this decision? And has that has that been an issue for you?

• 17:05Speaker 01

  I'll say that most databases started before cloud was really a fully mature product or a market or platform. Okay. And so the, that was the precursor to PLG essentially, or whatever it was PLG, right? It was, you know, it was that. That was the only way to put a technically complex product at the hands of engineers was to open source it, right? And you see, I think all, I mean, maybe not all, but definitely the larger databases that are open source out there, I think that's the reason they did that.

• 17:46Speaker 01

  When we started Pinecone, we asked, you know, the very basic question of why, why do people open source the platform, right?

• 17:58Speaker 01

  One of it was to earn trust. One of them was to get contribution from the community and one of them was a channel to, you know, users. And we figured we can earn trust by being excellent while we do and providing an amazing service. We don't need external contribution. And in fact, if you look at statistics, even companies that are open source, 99% of the contributions are actually from the company itself. Not 99, but high 90s. And so that doesn't actually make a huge difference. And in terms of experience, we figured that we can actually provide a much better experience and much better access to the platform than what open source does.

• 18:41Speaker 01

  And Pinecone is a fully managed and multi-tenant service. And to be able to run that at scale and provide the cost trade, the cost scale trade-offs, we actually run a very, very complicated system. And in some sense, even if we gave it as open source to somebody, they wouldn't know what to do with it. It will be a Herculean effort to even run this thing. The right decision was basically that we should offer this as a service. We should manage it end to end. And as long as you give people a fully reliable interface and you keep doing that year after year, you earn the trust and the ease of use that open source becomes in, I hope, not, not an issue.

• 19:19Speaker 02

  It's funny because the two negatives around, along those lines, I remember talking with, I think it was Ali from Databricks, and he said that if you can avoid doing open source, you should, you know? He felt like it was an incremental challenge because you get distribution through open source, but then you have to figure out the business model. And so he viewed it as like, you know, I think that the analogy he uses is like making an open source project work is like hitting a hole in one in golf. And then you pick up a baseball bat and you have to hit a grand slam because then you have to do the second act to make sure that the thing actually works as a company.

• 19:52Speaker 01

  That's right. No, I mean, I agree 100%. I mean, this is exactly what we're experiencing. And in fact, we we already see, even though new players in the vector database space that that that basically started to try to take us down, all took the open source angle. We already see them, even young as they might be, they are already struggling with their open source strategy.

• 20:17Speaker 01

  Serverless is the fourth, almost complete rewrite of the entire database at Pankon.

• 20:23Speaker 02

  Yeah. The one other thing that's coming in terms of the LLM world, which may or may not impact you, and I'm sort of curious how you think about it, is increasing long context windows for foundation models. Does that change how people interact with embeddings in vector databases? Or does it not really impact things much? There's things people are talking about in terms of infinite context or other things like that.

• 20:42Speaker 01

  Look, I mean, I don't know what infinite context means, to be honest.

• 20:46Speaker 02

  It's like very big.

• 20:49Speaker 02

  It's infinite. It's like huge.

• 20:51Speaker 01

  Oh, oh, thank you.

• 20:53Speaker 02

  Yeah, yeah, you're welcome.

• 20:55Speaker 01

  I should take a note. First of all, those companies sell their services by the token. So the fact that they allow you to use infinite context windows is not shocking. Okay, that's good for business. The second thing is there's plenty of evidence that increasing the context size doesn't actually improve results unless, you know, you do this very carefully, right? So just what's called constant stuffing is not helping. You just pay more and don't actually get much for it. And the last thing, that even that, even if you you kind of buy in to the the the marketing, that runs its course, right? If you're, it's

• 21:39Speaker 01

  like saying, oh, I don't need Google because I can, and every time I query Google, I can send the internet along with my query, right? It's like, yeah.

• 21:51Speaker 01

  Well, theoretically, that's maybe possible, but clearly practically, that's that's not feasible, right? So at some point, the context window just becomes gigabytes and gigabytes and gigabytes of data, like terabytes. I mean, where do you stop? Right? And so already today, we have users who use not even very large models, you know, maybe a few billion parameters. And the vector database next to their model contains trillions of parameters. Right? And they get, you know, much better performance that way. Right? Just attaching all the context to everything you do, I think runs its course very, very quickly. And it's also unnecessary, to be honest.

• 22:36Speaker 02

  Yeah, I guess related to that, another place where people have been talking about embeddings and vector databases is in sort of aspects of personalization and privacy. And I'm a little bit curious how you think about that because, you know, one of the the risk people view is running an LLM over a large data corpus or fine tuning it against a specific company's data is the issue of data leakage. You know, say, for example, you're you're an HR company and you don't want different people's salaries to leak across an LLM because you're using it as like a chatbot to help you with context regarding your own personal data in an enterprise or things like that. Can you talk a bit more about how embeddings can provide personalization and in some cases, potentially other features that may be attractive to to enterprises?

• 23:19Speaker 01

  Yeah, so that's a very common and reasonable thing to be concerned about.

• 23:27Speaker 01

  Data leakage can happen in in two main ways. A, if you use a service for your foundational model that that frankly re-trains their models with your data or records it, right? Or saves it in some way that is opaque to you. Right? That is a huge problem, and I think a lot of people are, a lot of people are struggling with that. The second is if you're building an application in-house, whatever it might be, and you fine tune your models on added data, that added data might end up popping where it shouldn't in answers to, you know, other people's questions or whatever.

• 24:09Speaker 01

  What people do with vector databases is actually incredibly simple, right? You don't fine tune your model on your own proprietary data. At which point you know for a fact it doesn't contain any proprietary data because it's never seen any of it. Okay. And then at retrieval time or at, you know, whenever you, you apply the uh, the chat or the agent, you retrieve the right information from the database. Give it as context to the model, but only do inference. You don't actually retrain and you don't save that interaction. At which point that data doesn't exist anywhere. It's like an ephemeral thing.

• 24:51Speaker 01

  And the added benefit to that is by the way, that you can be GDPR compliant. You can actually delete data. So if, if, you know, so you know, if you're, if you're a company, uh, like a legal company and somebody deletes a document, you can just delete it from the vector database and that information will never be available to your foundational model again. So you don't even have to devise some complex mechanism for forgetting. You just don't know it anymore. What are the main reasons why people attach vector databases to foundational models? It's, it's gives you this operational sanity. Uh, that is almost completely impossible without it.

• 25:28Speaker 02

  Hmm. That's interesting. Yeah, I guess, um, it feels like there's three different approaches that people are using. They're not mutually exclusive for models that kind of overlap in terms of what the hope for output is. Uh, one is really changing or engineering prompts. Uh, or adding more information into the prompt. The second is fine tuning. And the third would be RAG slash different aspects of embeddings or other approaches like that. How do you think about fine tuning in this context? Like, when should you fine tune versus, you know, use some of the approaches that you've talked about earlier?

• 26:04Speaker 01

  I can answer both as a scientist and as a business owner, right? As a scientist, I'm all for fine tuning. We have all the evidence to show that done right, it helps tremendously. As a business owner, I can tell you that it's actually extremely hard to do. To do well. I mean, this is something that unless you have the research team and the AI experts that know how to fine tune, you might actually make things significantly worse. Okay. So there's nothing that says that more data is going to make your model do better. In fact, it oftentimes gets, uh, regresses to something significantly worse.

• 26:43Speaker 01

  With prompt engineering, again, I think it's necessary, especially when you build applications, you want to respond to, you know, uh, conform to some format or have some property. I think that's, that's a given, you should do that. It runs its course after a while. I mean, it's in some sense, you get what you get. It's necessary, but there's a limit to what you can do with that. And RAG, I think, is incredibly powerful, but like I said before, when we talked about Canopy, that's not, you know, that's not simple either. I mean, it's simpler than the other ones, but Silicon Valley's work and understanding, experimentation and so on.

• 27:23Speaker 01

  This is almost a hallmark of a nascent market when the simplest solution is still somewhat complex.

• 27:33Speaker 02

  Yeah, makes sense. What's next for a pine cone or what are, what are some major things coming that you'd like to talk about?

• 27:39Speaker 01

  So, I mean, there's a ton. We are, we're an infrastructure company. Uh, and so we obsess about ease of use and security and, and stability and cost and scale and performance. Also, as an engineer at heart, I'm, I'm very excited about those things. And all of that is coming. Again, serverless is becoming faster, bigger, better, uh, more secure, easier to use. And we're starting to really grapple with, uh, what very large companies and very, you know, kind of trailblazing tech companies are going through. I said that getting AI to be truly knowledgeable is still complex.

• 28:22Speaker 01

  I think we're starting to grapple with deeper issues that, that the entire information retrieval community has been dealing with for about.

• 28:32Speaker 01

  Forty, fifty years now, we're starting to see those, you know, come to the fore in in RAG and in AI in general.

• 28:40Speaker 02

  I guess, putting aside Pinecone and sort of the database world and everything else, what are you most excited about in terms of what's coming next in AI?

• 28:47Speaker 01

  It's hard to say. I really do wanna see a a distillation, in some sense, of foundational models. And by distillation, I know it's a, I don't mean what usually people say, but there's distillation of models. I don't mean that. I mean, the separation of reasoning and and and knowledge, right? Foundational models get it fundamentally wrong. When we learn how to build the subsystems of AI correctly and for each one of them to do their roles optimally, either we're gonna be able to do the, to achieve the same

• 29:29Speaker 01

  tasks, much cheaper, faster, better, or we're gonna still wanna use the same amount of resources but achieve much more. What happens today is that we have very crude tools and we try to use everything for everything. Delightfully or shockingly enough, depending on who you are, that kind of works. I mean, we found these like very, very efficient, very general-purpose tools, right? But they're still very general-purpose. They're still super blunt instruments. Again, as a technologist or somebody who cares deeply about how things are built, you kind of see the inefficiency and it hurts the brain to figure out that, you know, we take half the internet and cram it into GPU memory. I'm like, holy, why?

• 30:14Speaker 01

  This can't be the right thing to do.

• 30:19Speaker 01

  So, I'm very excited about us as a community truly understanding how the different components interact and how to build everything much more, in some sense, correctly. And I hope we get to build some exciting products, by we, I mean the community gets to build some exciting products this year. I think we're going to see a year of a lot of experimentation that when

• 30:46Speaker 01

  the people went through last year, they're going to take the production and to build cool products this year. And I can't see, I can't wait to see how that looks like. I have a feeling that this year is going to be very, very exciting for consumers of AI.

• 31:00Speaker 02

  Yeah, I totally agree. It's a very exciting year ahead. So thank you so much for joining us today.

• 31:04Sarah Guo?

  Thanks, Edo.

• 31:05Speaker 01

  Thank you, guys.

• 31:08Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.