Vectoring in on Pinecone

Daniel & Chris explore the advantages of vector databases with Roie Schwaber-Cohen of Pinecone. Roie starts with a very lucid explanation of why you need a vector database in your machine learning pipeline, and then goes on to discuss Pinecone’s vector database, designed to facilitate efficient storage, retrieval, and management of vector data. Sponsors: Plumb https://useplumb.com/ – Low-code AI pipeline builder that helps you build complex AI pipelines fast. Easily create AI pipelines using their node-based editor. Iterate and deploy faster and more reliably than coding by hand, without sacrificing control. Featuring: Roie Schwaber-Cohen – GitHub https://github.com/rschwabco, LinkedIn https://www.linkedin.com/in/roiecohen, X https://x.com/roieschwabcoChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Pinecone https://www.pinecone.ioPinecone | Blog https://www.pinecone.io/blogUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the CEO and founder at Prediction Guard, where we're enabling AI accuracy at scale. And I am joined, as always, by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris?

• 1:02Chris Benson?

  Doing great today, Daniel. How's it going? I know we're recording leading into a holiday weekend here.

• 1:07Daniel Whitenack?

  We are, and so many exciting things. Last week, I got the chance to briefly attend the AI Engineer World's Fair, which is sort of prompted in certain ways by our friends over at the Laten Space Podcast. And that was awesome to see. And of course, a big topic there was all things having to do with vector databases, RAG, all sorts of, you know, retrievals, search, sorts of topics. And to dig into a little bit of that with us today, we have Rohit Shabbar Cohen, who is a developer advocate at Pinecone. Welcome.

• 1:47Speaker 03

  Hi, guys. Thanks for having me today. Really excited to be on the show.

• 1:51Daniel Whitenack?

  Yeah, well, I mean, we were talking a little bit before the show. Pinecone is, from my perspective, one of the OGs out there in terms of coming to the vector search, semantic search, embeddings type of stuff. Not that that concept wasn't there before Pinecone, but certainly when I started hearing about vector search and retrieval and these sorts of things, Pinecone was already a name that people were saying. So could you give us a little bit of background on Pinecone? Kind of how it came about and what it is position wise in terms of the AI stack.

• 2:29Speaker 03

  So Pinecone was started about four years ago, give or take, and our founder, Edo Liberty, was one of the people who were instrumental in founding SageMaker over at Amazon and had a lot of experience in his work at Yahoo. And I think that one of the fundamental insights that he had was that the future of pulling insights out of data was going to be found, not exclusively, but predominantly in our capability to construct vectors out of that data. And that representation that was produced by neural networks was very, very useful and was going to be useful moving forward. I think he had that insight way before tools

• 3:14Speaker 03

  like ChatGPT became popular. And so that really gave Pinecone a great edge at being the first mover in this space. And we've seen the repercussions of that ever since. With the rise of LLMs, I think people very quickly came to recognize the limitations that LLMs may have. And it was clear that there needed to be a layer that bridged the gap between the semantic world and the structured world in a way that would allow LLMs to rely on structured data, but also leverage their capabilities as they are. And that is one

• 3:54Speaker 03

  of the places where vector databases play a very strong role. Vector databases are distinct from vector indices in the sense that they are databases and not indices. So like an index may have, basically is limited by the memory capacity that the machine that it's running on allows it to have. Whereas vector databases behave in the way that traditional databases behave and the way that they scale. Of course, there's a completely different set of challenges, algorithmic challenges, that come with the territory of dealing with vectors and high dimensional vectors that don't exist in the world of just simple textual

• 4:35Speaker 03

  indexing and columnar data. And that's where the secret sauce of Pinecone lies, right? It's ability to handle vector data at scale, but maintain the speed and maintainability and resiliency of a database.

• 4:52Chris Benson?

  As you were kind of comparing vector databases to indices and then kind of bringing that compared to that, one of the things that I run across still a lot are people, you know, vector databases are really incredibly helpful now, but there's still a lot of people out there who don't really understand how they fit in. You know, they don't really get it versus the NoSQL versus relational databases.

• 5:16Daniel Whitenack?

  Or fine tuning.

• 5:17Chris Benson?

  Yeah. And so, and they hear you say it does vectors and stuff like that. Could you take a moment, since we have you as an expert in this thing, and kind of like lay out the groundwork a little bit before we dive deeper into the conversation about what's different about a vector database that is storing vectors versus storing the same vectors in something else? Like, why go that way for somebody who just isn't quite, hasn't really ramped up on that yet?

• 5:46Speaker 03

  So, the basic premise is you want to use the right tool for the job, right? And the basic difference, right, between a relational database, a graph database, and a vector database, or a document database for that matter, right, is the type of content that they are optimized to index. Meaning, a relational database is meant to index a specific column and create an index that would be easily traversable, right? And in scale, it would be able to traverse it across different machines, right, and do it effectively, right? A graph database does the same thing, only its world is nodes and edges, right? And it's supposed to be able to build a optimized representation of the graph such that it

• 6:29Speaker 03

  could do traversals on the graph efficiently. In vector databases, vector databases are meant to deal with vectors, which are essentially long, high-dimensional set of numbers, meaning, like, you can think of an array with a lot of real numbers inside of that array. And you can think of this collection of vectors as being points in a high-dimensional space. And the vector database is building effective representations to find similarities or geometric similarities between those vectors in high-dimensional space. And that means that, basically, it would be very effective at Given a vector, finding a vector that is very close, quote unquote, to that vector

• 7:13Speaker 03

  in a very large space, right? So to do that, like you need to use a very specific set of algorithms that index the data in the first place and then query that data to retrieve that similar set of vectors to the query vector at a small amount of time and also being able to update or make modification to that high dimensional vector space in a way that is not cost prohibitive, right? Or time prohibitive, right? And that's like the crux of the difference between a vector database and other types of databases.

• 7:45Daniel Whitenack?

  Just to draw that out a little bit more, so from your perspective, like what would be if you were to kind of explain to someone, hey, here I've got one piece of text and I'm wanting to match to some close piece of text in this vector space? What might be advantageous about using this vector based search approach and these embeddings in terms of what they mean and what they represent versus doing like a, you know, TF IDF has been around for a long time. I can search based on keywords. I can do a full text search. There's lots of ways to search text.

• 8:27Daniel Whitenack?

  You know, that concept isn't new, but this vector search is, seems to be powerful in a certain way. From your perspective, how would you describe that?

• 8:37Speaker 03

  Yeah, I think that the linchpin here is the word embedding, right? The vector search capability itself is a pretty straightforward mathematical operation that in and of itself doesn't necessarily have value, right? It basically, it's like other mathematical operations. It's a tool, right? The question is like, where does the value come from? And I would argue that the value comes from the embeddings. And we'll talk about what exactly they are, would just point a flag and say embeddings are represented as vectors, which is why the vector database is so critical in this scenario. But why are embeddings helpful in the first place? Right? So embeddings come from a different set, right?

• 9:19Speaker 03

  Like a very wide set of neural networks that have been trained on textual data. And they create within them representations of different terms, different surface forms, sentences, paragraphs, et cetera, that map onto a certain location in vector space. The cool thing about embeddings is that it just so happens, and we can talk about why, it just so happens that terms that have semantic similarity have a closeness in vector space. And that means that if I search for the word queen and I have the word king embedded as well in my vector database, and I also have the word dog, right?

• 10:00Speaker 03

  Because the word king is more semantically similar to the word queen, I will get that as a result and not the word dog. Right? And that allows me to basically leverage the quote unquote "understanding of the world" that machine learning models and specifically neural networks have, right? Large language models have of the world, right? In a way that I can't quite leverage from other modalities like T F I D F and other, you know, B M twenty five, et cetera, that like look at a more lexical kind of perspective on the world. Right? And so when we talk about, you know, practical use cases, right? Like RAG comes up very, very frequently. And the reason for that is because we are in semantic space.

• 10:42Speaker 03

  A user interacts with the system in semantic space. So that means that they ask the system a question in natural language. We can take that natural language and basically, again, quote unquote, "understand" the user's intent, right? And map it again into our high dimensional vector space and find content that we've embedded that has some similarity to that intent. Right? And so we're not looking for a exact lexical match, but we're actually able to take a step back, right? And look at the more ambiguous intention and meaning of the query itself and match to it, things that are semantically similar, if that makes sense.

• 11:23Chris Benson?

  Would it be fair to say that you're essentially, because the output of those structures being embeddings, and those are vectors, and therefore, you're essentially storing it and operating on it in a closer representation to how they naturally would be? And so you're not doing a bunch of translation just to fit it into a storage medium and to operate on it. Therefore, it's going to be quite a bit faster since you're... Is that fair? Is that a fair way of thinking about it?

• 11:51Speaker 03

  Perhaps. We're, in a way, we're compressing the representation into something very small, in a sense, right? So you can think of an image, for example, right? An image would, that could be like a megabyte big, right? We can get a representation that in terms of like its actual size, in terms of the vector, is order of magnitude's smaller, right? And we can use that representation instead of using the entire image, right, to do our search. Now, it just so happens that, again, when we're doing embeddings for images, right, we get like that same quality where we're not looking at, you know, we're not looking at an exact match, or like pixel matching, pixel to pixel with images

• 12:34Speaker 03

  that we have. We can actually look at the semantic layer, meaning what is actually in that picture. So if it's a picture of a cat, we would get, as a result, other pictures of cat that we've embedded and saved in the database, right? And that will come out of the representation itself, of the embeddings that were a result of like, say, like a clip model that we use to embed our image. So I don't know if it necessarily means that like it simplifies things. In a lot of ways, it actually adds a lot of more oomph to the representation, right? So you can actually match on things that you wouldn't necessarily expect, right? And that's kind of like what the beauty of semantic search in that sense,

• 13:16Speaker 03

  right, is that users can write something and then get back results that don't even contain, right, anything remotely similar in terms of the surface form to their query, but semantically, right, it would be relevant.

• 13:40Speaker 04

  Hey friends, this episode of Practical AI is brought to you by our new friends over at Plumb. Plum is a low-code AI pipeline builder that helps you to build complex AI pipelines super fast. You can easily create AI pipelines using their node-based editor, iterate and deploy faster and more reliably than coding by hand without sacrificing control. Deployment is easy, pipelines are live API endpoints. Eliminate the need for constant code redeployment and debugging by deploying complex AI pipelines as API endpoints. Team collaboration is easy too. Plum's declarative node-based editor enables you to build quickly while empowering

• 14:22Speaker 04

  non-technical roles to iterate on what you've done without breaking it. You can build advanced AI features, get structured output every time, transform data and leverage validated JSON schema to create reliable, high-quality structured output. So Plumb is built for builders. Earliest-stage product teams are using Plumb to go from idea to validation in record time. To get started, go to useplumb.com. That's Plumb with a B, as in plumber, to request access today. That's U-S-E-P-L-U-M-B dot com. Again, useplumb.com.

• 15:15Daniel Whitenack?

  Well, Rowi, I really appreciate also the statement about adding oomph to your representations. I think that would be some, there's some type of good t-shirt that could be derived out of that. I see for listeners who are just listening on audio, Rowi's wearing a shirt that says "Love thy nearest neighbor", which is definitely applicable to today's conversation.

• 15:41Daniel Whitenack?

  Well, this is great. So we've kind of got a baseline, in a sense, from your perspective, what a vector database is, why it's useful in terms of what it represents in these embeddings and allows you to search through. You mentioned RAG. We've talked a lot about RAG on the show over time, but maybe for listeners, this is the first episode that they've listened to, what would be the kind of 30 seconds or some type of quick sort of remember RAG is X from Rory?

• 16:13Speaker 03

  Right. So I love quoting Andrej Karpathy with his observation on LLMs and hallucinations. So usually when people talk about RAG, they say, oh, RAGs sometimes hallucinate and that's really bad, right? And Andrej Karpathy says, actually, no, they're always hallucinate, right? That they do nothing but hallucinate, right? And that's really true, right? Because LLMs don't have any kind of tethering to real knowledge in a way that we can trust, right? We don't have a way to say, hey, I can prove to you that what the LLM said is correct or incorrect based on the LLM itself, right? We need to go out and look and search, right? And RAG to me is that opportunity where we can take the user's intent, we can tie it

• 16:57Speaker 03

  using a, for example, right, a semantic similarity search to structured data that we can point to and say, this is the data that is actually trusted, and then feed that back to the LLM to produce a more reliable and truthful answer. Now, that's not to say that RAG is going to solve all of your problems, but it's definitely going to give you at least a handle on what's real and what's not, what's trusted and what's not, and where the data is coming from, where those responses are coming from. And it shifts the role of the LLM from being your source of truth to basically being a thin natural language wrapper that takes the response and makes it palatable and easy to consume to a human being.

• 17:42Daniel Whitenack?

  Great. Yeah, I think a lot of people have done a sort of, maybe they've done even their own demo with sort of a naive RAG, maybe pulling in a chunk from a document that they've loaded into some vector database. They inject it into a prompt and they get some useful output. One of the things that I think we haven't really talked about a lot on this show, we've talked about advanced RAG methods to one degree or another, but I know Pinecone along with other vector database providers, you know, offer more than a simple just like search. That's the only function you can do. There's a lot more to it that can make things useful.

• 18:21Daniel Whitenack?

  In particular, like having, you know, you mentioned Pinecone mentions kind of name spaces that can be used, metadata filters, sort of hybridized ways of doing these searches. Could you kind of help our listeners understand a little bit? So they may understand, here's my user statement. I can search that against the database and get maybe a matched document. But for an actual application, like an application in my company that I'm building on top of this, what are some of these other key pieces of functionality that may be needed for enterprise application or for a production application that go beyond just the sort

• 19:04Daniel Whitenack?

  of naive search functionality in a vector database?

• 19:08Speaker 03

  Yeah, for sure. So we can take this one by one. So metadata is definitely like one of those capabilities that vector databases have that are above and beyond what a vector index would provide you, right? And basically what they are is the, again, the ability to perform a filtering operation after your vector search is completed. And so you could basically limit the result set to things that are applicable in the application context, right? So you can imagine different controls and selection boxes, et cetera, that come from the application that are more set in stone, so to speak. They're not just like natural language. They're categorical data, for example. And you can use those to limit

• 19:50Speaker 03

  the result set, right? So that you hit only, only what you want. That is something that is very common to see in a lot of different production scenarios.

• 19:59Daniel Whitenack?

  And could you give maybe an example of that, like in a particular use case that kind of you've run across, like what, what might be those categories or what, just to give people something concrete in their mind?

• 20:10Speaker 03

  Yeah, for example, like you can imagine a case where I'm not going to name the customer, but like you can imagine that the case where you want to perform a RAG operation, but you want to do it on a corpus of documents, but not on the entire corpus, but rather on a particular project within that corpus. So imagine that you have multiple projects that your product is handling, like finance and, you know, HR and whatever, engineering, right? And you want to perform that search and then limit it only to a particular project. And in that case, right, you would use the categorical data that is associated with the vectors that you've embedded and saved in Pinecone to only get the data for that particular project, right? That is like a kind of super simple example.

• 20:55Speaker 03

  But it can go beyond that, right? And move into like the logic of your application. So like you can imagine a case where, you know, you're looking at a movie, a movie data set, right? Like, and you want to search through different plot lines of movies, but you want to limit the results only to a particular genre, right? That's another case, right? Like we could just leverage metadata. You can think of wanting to limit the results to a time span, right? A start and end date, right? Things of that sort that kind of like have to do more with the nature of when and how and what category the vector belongs into and not specifically the contents of the vector, right? So that's one thing.

• 21:38Speaker 03

  Namespaces are another feature that we've seen as being like incredibly important for multi-tenant kind of situation. And multi-tenant RAG has become kind of like a very strong use case for us. And that's where, you know, you, you see a customer, that customer has customers of their own and not one or two, but many, many, many. And in that case, you definitely don't want to have all of the documents that these, that all of the sub-customers have to be co-located in one index. And in that case, you basically break them apart, right? So they're still in one index, so management of the index overall is maintained under one roof, but the actual content and the, the vectors themselves are separated out physically from one another in namespaces.

• 22:22Speaker 03

  They're sort of sub-indexes to that super index. And that's another feature that we've seen as being super important to our enterprise customers.

• 22:31Chris Benson?

  As you're looking at these enterprise customers and with maybe most enterprises, you know, getting into RAG at this point at some level and trying to find use cases for their business to do that, I know, you know, my company and lots of other companies are doing this. What are some of the ways that they should be thinking about these different use cases? When we're talking about RAG and semantic search and multimodal things that Pinecone does, what are good entry pathways for them to be thinking about how to do this? Because, you know, they may have come up with their own, their kind of own internal platform, it might have some open source, it might have some products already in play, but maybe they don't have a vector database in play yet.

• 23:13Chris Benson?

  And so, you know, how do they think about where they're at when you guys are talking to them and you're saying, let me, you know, because we've been talking in the show so far about kind of the value of the vector database and the kind of these use cases, but not necessarily kind of the, an easy pathway. So how do you onboard enterprise people to take advantage of the goodness on this?

• 23:34Speaker 03

  Yeah, that's an excellent question. And in fact, it's like a quite, quite a big of a challenge because it ends up being, you know, a straightforward pipelining challenge that has existed from the beginning of, you know, the big data era, right? Like is how do I, how do I leverage all the insight that is locked in my data in a beneficial way, right? And the sad part about this story is that it always depends on the specific use case and it's hard to give a silver bullet. A sort of light at the end of the tunnel is that we've recently published a tool called the RAG Planner and its purpose is to basically help you figure out what do you need to do to get from where you are to an actual RAG application and follow through all of the

• 24:18Speaker 03

  different steps that are required in between, right? And sort of like understand like from an understanding of like where your data is stored, how frequently it updates, like what the scale of your data is, et cetera, et cetera, to the point where it could give you some recommendation as to like, what are like the steps that you have to do, like in terms of, do you build a batch pipeline? Do you build a streaming pipeline? What tools should you be using to do those things? What kind of data? Data cleaning are you going to need to do? What embedding models are you going to want to use to do this, right? Like, how are you going to evaluate the results of your RAG pipeline? So all of these questions are pretty complex. So what I would say is a general rule of thumb. First of all, like you have to evaluate whether or not RAG is for you, right?

• 24:59Speaker 03

  So, for example, there are a lot of situations where, you know, RAG may be the wrong choice, right? Because the data that you have, right? And the actual capability of answering the end users' questions based on that data does not match up, right? And that's how you get to see, you know, cases where, you know, chatbots sort of spit out results that may seem ridiculous, but nobody catches it. And companies get into a lot of hot water because of it, right? There are a lot of scenarios where it's much easier to start that journey and to sort of develop the muscle memory that's required in order to set these things up. And in a lot of these use cases, you see like a lot more internal processes, definitely in bigger companies, right?

• 25:43Speaker 03

  Where like there's a very big team that just needs access to its internal knowledge base in an efficient way, but it's not a system that is going to be mission critical, right? In any way, so like if a person gets a wrong answer, it's not going to be the end of the world. Nobody's going to get sued, right? And so what I would say is there's definitely a learning curve here for big organizations for sure. It's usually recommended to develop again that that internal knowledge of what the expectation versus the realities on the ground is going to be. To have like a really good idea of how you assess risk in those situations. And most importantly, how to evaluate the results that are produced by those systems, right?

• 26:27Speaker 03

  Because a lot of people are like, okay, you build the RAG system, great, and now produces answers. I'm done, right? Like, we're, we're, everybody's happy. That's farthest from the truth that you could possibly be, right? Like, these systems need to be continuously monitored and feedback needs to be continuously collected to the point where you can understand, right? Like, how changes in your data and the way that you're interacting with it, changes in large language models that you're implying are actually affecting the end result, right? Are going to be, and how your users are actually interacting with the system overall, right? How all of these things kind of coexist and happen together? And are they working in the way that you want them to? And of course, you want to do that, you know, in a quantitative and not qualitative way, right? So like, there's a lot of instrumentation that has to go into it.

• 27:11Chris Benson?

  I'm curious, as a little follow up to that, and obviously leaving specific customers out of it, are you tending to see more internal use cases of RAG deployment to internal, you know, groups of employees and stuff, maybe from a risk reduction? Are you seeing more of an external, I'm going to get this right out to my customers and try to beat my competition to it? Like, where do you think the balance is as of today?

• 27:33Speaker 03

  I think that there's a widespread, and I think that it's a journey, right? Like, I think that like, the more tech native companies that we see that are more, I would say, forward looking, or, you know, technologically adept to kind of do these things quickly, are more ready to not only take risks, but take educated risks in this space with the evaluation that comes with it, right? So like, these are not just like, let's set and forget, but they actually know what they're doing. In those cases, you see them going out to production with very big deployments. That is our bread and butter, I would say, at the moment, right? With companies that are more traditional, that have been like, that are not necessarily getting tech native, you see a more cautious sort of progression,

• 28:17Speaker 03

  which is only to be expected, right? Like, I think that's kind of like, natural to see.

• 28:22Daniel Whitenack?

  Well, Roy, I have something that I saw on your website, which was new to my knowledge, which I think is also really interesting. One of the things that I've really liked in experimenting with vector database, RAG type of systems as an AI developer, is having the ability to run something without a lot of compute infrastructure, maybe in an embedded way, or an on disk index, something that I can spin up quickly, something that I don't have to deploy a Kubernetes cluster or something to, or set up a bunch of kind of client server architecture to set up and test out maybe a prototype that I'm doing.

• 29:03Daniel Whitenack?

  And I, and I see Pinecone is talking about Pinecone serverless now, which is really, really intriguing to me, just based on my experience in working with people. These sort of serverless implementations of this vector search, I think, can be really powerful. So could you tell us a little bit about that and how that kind of evolved? And what it is? What's the current state? And how Pinecone thinks about the serverless side of this?

• 29:31Speaker 03

  So serverless came about after we've realized that tying compute and storage together is going to limit the growth factor that our bigger customers are expecting to see. And it basically makes growth kind of prohibitive in this space. Right? And so we had to find a way to break apart these two considerations while maintaining the performance characteristics that our customers are expecting and are used to having from our previous architecture. So essentially, like, serverless has been a pretty big undertaking on our side to ensure that the quality of the database is maintained.

• 30:14Speaker 03

  But at the same time, we can reduce costs dramatically for customers. To just give you like an idea, for the same cost of storing about, I don't know, around 500,000 vectors before, you can now store 10 million, right? And that's a humongous difference, right? Like, it's an order of magnitude difference. I think that to accomplish that, right? Like, there was like a lot of very clever engineering that had to happen. Because, again, now having compute and storage separated apart means that storage can become very cheap. But on the other hand, it requires you to handle the storage strategy and retrieval in a lot cleverer way.

• 30:57Speaker 03

  We have a lot of content on the website that kind of delves deeper into how exactly technically that was achieved. And we won't be able to cover that, given the time that we have. But like, the basic premise is that you can now grow your vectors indexed to theoretically infinity, but practically to tens of billions and hundreds of billions of vectors, without the cost of the expense becoming prohibitive, which is the main drive for us with. Are bigger customers and also with smaller customers like you can start experimenting. We have like an incredibly generous free tier that allows you to start, you know, like you said, right? Like if I'm just a developer on my own, testing things and trying to understand how vector database works in my world.

• 31:40Speaker 03

  It's very unlikely that I'll be able to tap the entire free tier plan, even several months in with many, many vectors stored. Right? And it will work the same way that our pro serverless tiers work in terms of its performance. So it's not like a reduced capacity or performance in any way. So you get, you get to feel exactly what it would feel like. And the effort that's required to stand it up is minimal to negligible. Right? You just set up an account and the SDK is super, super easy to use.

• 32:10Daniel Whitenack?

  Yeah. And, and I understand it or sort of representing things, right? Like in terms of the, you know, massive, so there's a massive engineering effort. I'm sure as you mentioned to achieve this because it's not a trivial thing. But in terms of the user perspective, like if people use Pinecone before and they're using Pinecone now. You already mentioned the performance is the, is the interaction similar? It's just this sort of scaling and sort of from the user perspective, scaling and pricing. And maybe also you could touch on, so Pinecone is, people might be searching for different options out there. And some of them would require you to have your own infrastructure or some of them are hosted solutions.

• 32:55Daniel Whitenack?

  Pinecone, at least in, in its kind of most typical form would be hosted by you. And yeah, could you just talk a little bit about the user experience pre, post serverless? And then also kind of the infrastructure side, like what do people need to know? And what are the options around that?

• 33:13Speaker 03

  In terms of what happened pre and post, so before serverless, there was like a lot of possible configuration choice that you could do. Right? Like, so like there was in fact, a lot of confusion with our users, you know, like what exactly is the best configuration for me? Should I use like this performance kind of configuration? Should I use like the? Throughput optimized configuration, what exactly am I supposed to use? And like the pricing mechanism was a little bit convoluted. And I think that like serverless, the the attempt there was to simplify as much as possible. Uh, and to make it really, really dead simple, right? For people to start and use, but also grow with uh with us, right?

• 33:52Speaker 03

  So again, like I said, like the bottom line is, you know, the external view into what Pinecone offers may have looked pretty similar, right? So like, if you're just a user, you may say like, hey, like I got like a cheaper Pinecone bill this month. And, you know, like I can store a lot more. Always a good thing. Right, always a good thing, right? But not super amazing, right? Like, but the end result is, uh, the question is like, what happens when, you know, you can actually store a lot more vectors? Right, what what does that unlock for you? And and again, I think that like the end of the day, right? Like, the way that we see Pinecone, and this this may help us uh kind of talk about like what's next for Pinecone, is a place where your knowledge lives, right?

• 34:37Speaker 03

  And and it allows you to build knowledgeable AI applications, right? And having more knowledge is always net positive, right? In that context, right? So the assumption is that like, you know, as AI applications grow, they accumulate more and more and more knowledge, and they become that more powerful with any additional knowledge that you can stuff into them. And so there's like actual value beyond the fact that you can store more, right? Like, and and it's cool, right? Your application actually becomes more powerful because it can handle more types of of use cases. It has a better ability to be more accurate and respond truthfully to a user when they are interacting with it.

• 35:19Speaker 03

  And so I think that like in general, like there's like this uh blatant kind of value that is only going to be apparent once people really experience what it means to have, you know, a million documents uh that are stored in Pinecone versus 10 million documents that are stored in Pinecone. And that effect is going to be very powerful. I think that's the majority of the of the benefit that I see.

• 35:44Daniel Whitenack?

  Maybe that gets to the next thing which I was gonna ask about, which I also see the announcement around Pinecone Assistance. And I'd love to hear more about that. Like, what is from, of course, sometimes maybe that can be loaded language also for people in the AI space. But in terms of this assistance functionality for Pinecone, what are you trying to enable? And where do you see it headed?

• 36:09Speaker 03

  So that has to do with the question that Chris had before, which is like, what is the journey, right, for customers, right? And I think that like, as a general purpose that we had around Assistant was to reduce the friction between me having a bunch of documents that I want to interact with, with an LLM or an AI in some form and capacity, to the point where that actually works, right? There are a bunch of ways of going about it, right? I think Pinecone wants to bring, you know, on top of our very robust vector database, a very smooth experience that lets users really do very little and get all the value

• 36:51Speaker 03

  out of Pinecone without having to think too much about it. So for that purpose, we don't only have the ability to take your documents and then embed them and, you know, do the end-to-end process of creating that completion endpoint for you, right? We're also the ones providing the actual inference layers as well, right? And so it's not going to be, again, if you asked like me, like this question, like, how do you build a RAG pipeline, right? Like, a year ago, even, right? I'd have to tell you, hey, you have to go to like some embedding provider, you have to find someone who would do like your, you know, PDF extraction, or, you know, take the data and chunk it and do all this stuff, right? No more, right?

• 37:34Speaker 03

  Like, the reality here is you can take a set of documents, throw them at this knowledge assistant, and the rest is kind of quote-unquote magic, right? It just happens for you behind the scenes while maintaining the quality that you want to get and at the scale that Pinecone can deliver, right? Which is, again, another differentiator. So like I said before, Pinecone is built to withstand hundreds of billions of documents, right, that, of vectors that you would store with us, and still be able to produce responses in a reasonable amount of time. And that's true for knowledge assistant, because Assistant sits on top of the vector database.

• 38:10Chris Benson?

  So it sounds like that may be a really good way, especially for small organizations. You know, we talked about enterprise and they have a certain infrastructure and teams to go with that. But there are so many more small organizations out there that have very little in terms of, of train people necessarily to do that and they had, you know, they don't have all the infrastructure in place and they're looking, you know, with assistants and serverless, they're looking for simple ways to onboard and, and get utility out of it. Would you say that the combination of serverless and assistants and then maybe whatever they might have in AWS or whatever platform that they're using is kind of just made to gel easily for them so they can get to something working pretty quick?

• 38:54Speaker 03

  Yeah, I mean, at the end of the day, like if you think about it, right, like the process shouldn't be as complicated as it is. Right. It's just that there are many parts to it and nobody picked up the gauntlet of saying like, hey, well, just do it all. You know what I mean? Because all of it is quite complicated to do right. Right? And so, um, yeah, like I think that like initially we'll see smaller organizations kind of, you know, picking that up because they don't have the resources, but as time moves along, you know, you're going to have to ask yourself even as a bigger organization, do I want to own this pipeline? Right? Is it something that I need to own? Right? And what value am I getting from actually owning all of this? Right? And so, um, yeah, like it would be interesting to see. So like this is a very, very new product still on public beta.

• 39:38Speaker 03

  And it will be interesting to see how the, the market kind of reacts to it and, and, and sort of experiments with it. But my bet is that as time progresses and knowledge assistants themselves become more capable, doing things maybe beyond RAG or beyond simple RAG, quote unquote, um, that, you know, like more, more and more sophisticated organizations might want to actually give it a try.

• 40:02Daniel Whitenack?

  And that really brings us maybe to a good way that we like to end episodes, which is asking our guests to sort of look into the future a little bit and not necessarily predict it because that's always hard, but to look into the future and kind of what are you excited about? It could be related to vector databases specifically or pine cones specifically, but maybe it's more generally in terms of how the AI industry is developing, the sorts of things that you're seeing customers do that are encouraging, whatever that is, what sort of keeps you excited about where things are headed going into the rest of this year?

• 40:39Speaker 03

  I'm excited about the fact that we're seeing sort of like a resurgence of what you would call traditional AI kind of come back into the fold in the form of, for example, Graphrag. I think that like the notion here is that, you know, for the longest time, and I think that's been since like, you know, GPT-3.5, you basically saw like this, like, I think, over indexing on LLMs, right? For good reasons, right? Like, they're super exciting, they're very powerful, right? And they can do really, really cool things, right? But with that said, it's as if every other technology that has ever existed before

• 41:20Speaker 03

  just like dropped off the face of the earth and nobody, nobody has ever like talked about like, okay, wait, so what can we do with those things and LLMs, right? Like, and where do LLMs fit in the bigger picture? I think that vector databases kind of like put LLMs in their place a little bit in the sense that, you know what I mean? Like, you're not thinking of the LLM as being the end all be all, like, this is the only tool that we need. I'm very excited to think of LLMs as these operators or agents that can tap into the capabilities that exist in other systems. And I think that what we're going to see more and more and more is that people are going to figure out like, and what subset of the ecosystem does each tool belong? So what set of problems that each tool solve?

• 42:03Speaker 03

  For example, like a vector database solves like the problem of bridginging the gap between the semantic world and the structured world. A graph database can Solve problems like formal reasoning over well-structured data. Relational databases can solve a whole set of different problems that they used to be solving, like aggregation, et cetera, et cetera. And then you can imagine that LLMs and agents can sit as sort of like an orchestrating mechanism and a natural language interface mechanism on top of all those things together. And that's what I'm excited to see, like it's kind of when like the the community as a whole is going to like wake up from its like LLM fever dream and sort of realize that like there's other things out there.

• 42:44Speaker 03

  And and realize that it has so many more powers that that it could yield to make really exciting applications.

• 42:52Daniel Whitenack?

  That's awesome. Well, thanks for painting that picture for us, Roy, and and for taking time to dig into so many amazing insights about about vector databases and embeddings and and knowledge management in general. So yeah, appreciate what you all are doing at Pinecone and hope to have you on the show again to to update us on on all those things.

• 43:14Speaker 03

  Thank you so much. Thanks for having me.

• 43:24Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. And join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.