Vector databases for machine learning

Pinecone is the first vector database for machine learning. Edo Liberty explains to Chris how vector similarity search works, and its advantages over traditional database approaches for machine learning. It enables one to search through billions of vector embeddings for similar matches, in milliseconds, and Pinecone is a managed service that puts this capability at the fingertips of machine learning practitioners. Sponsors: PSSC Labs https://pssclabs.com/practicalai – Solutions from PSSC Labs provide a cost effective, highly secure, and performance guarantee that organizations need to reach their AI and Machine Learning Goals. Learn more and and get a FREE consultation today at pssclabs.com/practicalai https://pssclabs.com/practicalaiSnowplow Analytics https://snowplowanalytics.com/ – The behavioral data management platform powering your data journey. Capture and process high-quality behavioral data from all your platforms and products and deliver that data to your cloud destination of choice. Get started and experience Snowplow data for yourself at snowplowanalytics.com https://snowplowanalytics.comFastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Edo Liberty – Website https://edoliberty.github.io, GitHub https://github.com/edoliberty, LinkedIn https://www.linkedin.com/in/edo-liberty-4380164, X https://x.com/edolibertyChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Pinecone | Website https://pinecone.ioPinecone | Twitter https://twitter.com/Pinecone_ioPinecone | LinkedIn https://www.linkedin.com/company/pinecone-ioTheNextPlatform - It’s Time to Start Paying Attention to Vector Databases https://www.nextplatform.com/2021/03/11/its-time-to-start-paying-attention-to-vector-databasesTheNewStack - Pinecone: A Vector Database for Machine Learning Applications https://thenewstack.io/pinecone-a-vector-database-for-machine-learning-applicationsUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I personally try to push the boundary and try to figure out what's the next huge milestone we as a society can hit. For Pinecone specifically, for us as a community, I think that really starting to think about not only the model that I trained and how do I put it in a container and ship it to someplace in my app, but rather like, okay, we now have this complete wealth of data that we can now use and how do we integrate intelligence with compute and memory and storage? How do we make that behave a lot more like a brain and not like a neural net?

• 0:40Speaker 01

  Big thanks to our partners Leno, Fastly, and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 0:59Speaker 02

  With advancements in AI and deep learning evolving at lightning pace, it's more important now than ever to research the best options suited to your unique needs. This is particularly true when building custom systems and those systems that are GPU-heavy. Not only do the applications running on the system matter, but your AI infrastructure and budget constraints need to be front of mind as well. PSSC Labs, which is an HPC and AI custom solutions provider based in California, has been creating high-performance computing systems to meet their clients' unique enterprise computing challenges for more than 25 years. And with cloud computing costs growing at astronomical rates, plus companies increasingly losing control of their data security, it is no wonder that

• 1:43Speaker 02

  enterprises and government agencies need to continually look for ways to take back control of their data. Solutions from PSSC Labs provide a cost-effective, highly secure, and performance guarantee that organizations need to reach their AI and machine learning goals. For more information and a free consultation, please visit pssclabs.com-practicalai. Once again, that's pssclabs.com-practicalai.

• 2:19Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:45Chris Benson?

  Welcome to another edition of the Practical AI Podcast. My name is Chris Benson. I am a tech strategist at Lockheed Martin. Sadly, my co-host Daniel Weihnack is not with me today, but we have a really, really good show lined up. Today, we're going to be talking about how AI is changing the nature of data, the data that we have, and what we might need to do about that. With me today, I have Ido Liberty, who is the founder and CEO of Pinecone. Welcome to the show.

• 3:13Speaker 00

  Thank you. Happy to be here.

• 3:15Chris Benson?

  Hey, I'm glad to have you. I'm really excited about our conversation here today because I think I'm going to learn a little bit about it. I'm wondering if you will start us off, instead of doing the normal bio that we do with everyone, tell me something interesting, really cool, that kind of led you into this. What's the kind of thing that you'd want listeners to know that's kind of a cool thing about your background that's kind of helped you get where you're at today?

• 3:39Speaker 00

  You know, I can tell you in kind of what got me into like big data. I know big data is already kind of like not cool to say anymore, but it was 2005, I was working on what's called hyperspectral images. So those are images, instead of RGB, you'd have like 150 different spectra, so different wavelengths, right? And so you can analyze them much more deeply with computer vision, even though to our naked eye, of course, they look the same because we only have three spectra to, you know, consume images with in our eyes.

• 4:11Chris Benson?

  Right.

• 4:11Speaker 00

  But each image itself was about a gigabyte, and our computers had 512 megabytes of memory.

• 4:20Speaker 00

  So one image was already big data and we had like a thousand of them. And so as a PhD student, we kind of like even like just even a few images were already like big data. So you we even to do like basic computer vision, you already had to think about algorithms on how to get more from less with computers. I ended up doing my PhD in theoretical computer science and working on algorithms and numerical linear algebra. But it's funny how I got there was really just trying to figure out how to do like basic computer vision on a single image. But the machine was too small and you just kind of had to figure out better algorithms and just loading everything into memory.

• 4:58Chris Benson?

  So it sounds like that was almost if you were working on that, it sounds like the move into kind of the current deep learning, you know, phase coming out of past winters and stuff really fit well. It sounds like you were really well situated for that in terms of what you had been doing in school.

• 5:14Speaker 00

  Yes and no. So I the beginning of my PhD was maybe like or maybe unlike other PhDs was like a period of about almost two and a half or two plus years where I experimented with different topics almost every few months. Like I would dive into like functional analysis and then like really into computer vision and then NLP and then like go into like numerical linear algebra and come back to, you know, whatever. And then it felt really chaotic and very wasteful in terms of resources, to be honest. But funny enough, over the years, as I became a director at Yahoo and then at Amazon, I actually found it to be incredibly useful because I could talk to all of these people. I could talk to the computer vision folks and I could talk to the NLP folks and I could

• 5:59Speaker 00

  talk to the like algorithm missions and I could talk to the, you know, engineers. And so it was fun. So all these like time wasted somehow came back in like a different life and served me really well. So it's kind of like maybe a lesson to the ages with PhD students, you know, time wasting is not necessarily a bad thing.

• 6:15Chris Benson?

  That's true. When you mentioned AWS, do I recall that you had something to do with SageMaker while you were there?

• 6:21Speaker 00

  Correct. I was the research director in charge of SageMaker. We built that platform. I led the big creation of all the algorithms, components in that and kind of spearheaded most of the effort there.

• 6:33Chris Benson?

  Very cool. And were you doing AI stuff at Yahoo?

• 6:36Speaker 00

  Correct. I was heading the Scalable Machine Learning Algorithms group. That was the group in charge of the infrastructure that supported like spam classification, ad ranking, you know, everything like personalization, ranking, threat detection, everything, everything kind of was led by machine learning. It was done with a centralized group that was in charge of that infrastructure. Yahoo at the time was a real pioneer in machine learning. I don't know if people know that, but it used to be a real powerhouse.

• 7:06Chris Benson?

  Gotcha. I guess as you went through those experiences at First Yahoo and AWS and they kind of shaped, you know, where you were going and the way that you would be thinking about AI. I alluded at the beginning of the episode about the changing nature of data. Can you share some of your thoughts along those lines and kind of how the world is evolving and how AI and the tools that we use in and around AI are starting to affect that?

• 7:32Speaker 00

  A hundred percent. So I'll kind of give a historical tour on machine learning. I think maybe the listeners would appreciate that.

• 7:38Chris Benson?

  Absolutely.

• 7:40Speaker 00

  We have been doing machine learning basically on vectors as inputs. So this like array of numbers. And we used to think about them as features that we would handcraft. Like you would take like feature number one is like whatever the ratio between the volume and the, you know, price and the feature number two is like the number of, you know, minutes it's been on the website and so on. As time progressed and deep learning gained more and more traction, more and more centrality in the AI world, crafting those features became in and of itself a machine learning problem.

• 8:15Speaker 01

  Yes.

• 8:15Speaker 00

  And so now you have embeddings. So these embeddings are these like auto-generated features, like semantically deep features for objects, not only for like images and free text that we are kind of used to with NLP and computer vision models, but it's also kind of the lingua franca now for all data. So like user behavior and threat vectors on some network and so on. So this is how we generate features this way, like nowadays. And so suddenly, when you look at the world of just kind of what data infrastructure is able to digest, like we used to think about like databases and search engines kind of being kind of the main workhorses of like data, like as in the data infrastructure in

• 9:00Speaker 00

  companies.

• 9:01Speaker 01

  Right.

• 9:01Speaker 00

  And applications in general, but they really know how to deal with like tabular data or like text and like terms and documents, you know, but they're very limited and nothing really knew how to deal with that just blob of numbers, you know, this like embedding, this like feature vector, right? And still nothing knows how to deal with it, right? But, and here's kind of the amazing thing to me, and one of the main things that led me to start Pinecone, was that if you look at any projection and any current statistic already on data in the world, the vast majority of it is not in tables. The vast majority of this is not schematized. It doesn't fit in the database. You can't run SQL for it.

• 9:42Speaker 00

  The vast majority of it is audio and video and free flowing text and like user behavior, which is very amorphous, a lot of like unstructured data, but it's much more than unstructured. It's like really unstructured. It's like really unruly, right? But today, with these deep learning tools, we can actually featureize them. We can represent them as these high dimensional vectors and suddenly we have something to work with, like we have an object.

• 10:08Chris Benson?

  Could you dive into that a little bit in terms of like, as we made this transition out of the tabular data world, you know, kind of where you had a database at the, you know, center of every website and all that. And it was, you know, we look back, it was very quaint. And these days, as you're talking about that huge volume of unstructured data, how have we adjusted to meet that as we've moved into the modern era of machine learning?

• 10:33Speaker 00

  We haven't yet. We are at just the beginning of the journey, I think, on that front. So maybe let me maybe take you on a short detour. Okay. On basically how our visual system, our human visual system works. Sounds good. In a very, you know, simplified way. And maybe by analogy, that would explain where we are today and what systems exist. So in a very, very schematized way, when you look at the world, you, Chris, all right? Light hits your retina, activates a few neurons, a lot of neurons. That signal is sent back through a few channels and organs, like neural organs in your brain. But the main one that right in the back of your neck, just above the back of your neck, is your visual cortex.

• 11:17Speaker 00

  Six layers of neural processing that you can think about as a very, very sophisticated computer vision model. It really does look like a convolutional neural net in some sense. It was kind of the motivation or maybe the inspiration to create neural nets the way they are. It takes local neighborhoods and aggregates them and takes averages and activations. And every layer builds on the layer above. And in the output of that, you have an activation surface of a few million, of several million neurons. And that processing is identical, no matter if you read a book or drive a car or look at your children. It does the same thing. It's pre-processing, right? It's the hardware, okay?

• 12:01Speaker 00

  When I think about where computer vision is today, we are mostly focused on that layer. How do we build a better visual cortex? How do we process images such that the semantic representation, that activation, enables me to do all these different tasks, right? What nobody is really doing yet is looking at how do we do the rest of the brain? So looking at stuff doesn't end there, you know? The output of the visual cortex actually flows mostly the temporal lobes, which is kind of above your ear, both ears, where you do things like remember that when you see your children, you know them, they're your children. You know, you don't just see a... It helps.

• 12:46Speaker 00

  Yeah, it helps. They're not just a person. And you know that because you've seen them before. And so a huge part of what it means to like live in a world is memory.

• 12:56Chris Benson?

  Yes.

• 12:57Speaker 00

  It's inference, it's analogy, it's association, it's things that have to do with the storage and retrieval and analysis of those high dimensional vectors and all the billions of those that you've accumulated over your entire lifetime, right? And so it's not about processing, it's not about classification, it's about how do I store everything that I've ever seen and make it accessible to me as a person when I see something, I know what to do with it because I've seen it before. I might have seen it last time 20 years ago, but I still know what it is. I still know what it is, right?

• 13:29Chris Benson?

  Let me ask you a quick question because we tend to talk about deep learning in terms of compute all the time, you know, in terms of the processing and stuff like that. And as you're reframing it there in terms of it's a memory, it's a memory problem, it's a memory thing to be thinking about, are we looking at it the wrong way or am I mixing two things up inappropriately?

• 13:49Speaker 00

  No, 100%. I think that I kind of broke the processing into two main steps, right? There is the crunching of the raw signal to bring it to a semantic representation that your brain knows how to work with.

• 14:04Chris Benson?

  Right.

• 14:05Speaker 00

  And do it, I mean, maybe surprising or maybe unsurprising to you, by the way, everything that you see and remember and know kind of is those neural representations. Your brain doesn't have access to your retina directly. Like you never actually remember the actual image. The actual image is the vector representation of it, right? All I'm saying is that once you've done the kind of hardware processing of that, which is very like, you know, compute heavy.

• 14:32Chris Benson?

  Yeah.

• 14:32Speaker 00

  Right? But it's stateless, okay? Once you're done with that, you have the rest of the brain to care about. You have memory, you have things like analogy and retrieval and search and comparison and like things that have to do a lot more with kind of how a database works or what you would expect from a database of data of that kind. The reason why I started Pinecone was because no database like that existed, like nobody knew what the hell to do with it.

• 15:03Chris Benson?

  So it sounds like before we dive into Pinecone, though, that we're still developing the tools and infrastructure that we need to be able to handle that evolving, that changing nature of data. And as we get better at doing that, as we start accounting for the fact that we have this memory paradigm that we're trying to get used to, we need tools, infrastructure that will address that. Am I on the right track?

• 15:27Speaker 00

  Exactly.

• 15:37Speaker 01

  This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights, a managed data platform that's built on leading open source tech, leveraged by tens of thousands of users. Capture and process high-quality behavioral data from all your platforms and your products, and deliver that data to your cloud destination of choice. When marketing needs to make data-informed decisions, when product needs next-level understanding, and when analytics needs rich and accurate data, Snowplow is the solution for data teams who want to manage the collection, processing, and warehousing of data across all their platforms and products.

• 16:18Speaker 01

  Get started and experience Snowplow data for yourself at snowplowanalytics.com. Again, snowplowanalytics.com.

• 16:47Chris Benson?

  Okay, so I know that Pinecone, when I look at the very front page of your website, it talks about managed vector similarity search. And I'm assuming this is how you're approaching the issues that we were just discussing a few minutes ago. Can you start us off by telling me what is vector similarity search, and where does it apply? What does it mean to my world?

• 17:11Speaker 00

  Sure. Tying back to the end of the last segment. We painted a very grand picture, but when you need to work with infrastructure, you know, you need like an API, you need something very, very crisp, right? One of the things that we started working on was say, what's the most basic thing? Let's say I have a very large collection of those vectors, say billion embeddings of like all the, you know, pieces of content on Facebook or all of the webpages in the world, all the images that I've ever seen. What's the most basic thing you would expect to be able to do with that? Okay. And the most immediate thing is retrieval, right? Basically answering the question, have I seen this before? Right? It's a very basic question, right?

• 17:56Chris Benson?

  Sure.

• 17:57Speaker 00

  Now, unlike strings or some IDs, you know, they're not identically matching ever, right?

• 18:05Chris Benson?

  Right.

• 18:05Speaker 00

  Nothing that you've ever seen is an identical, like, you know, photon to photon match. I mean, it's not, it's a different, you know, it's slightly different. So what does kind of the same mean, you know? So when you translate to these semantic vectors, you have the semantic similarity in the world. Okay. What we humans experience as this is the same thing, right? It just translates to proximity of those vectors. They are, on average, the values are close. They're not identical, but if you sum up all the differences, the differences are not large.

• 18:41Chris Benson?

  So as an example of that, we were talking about recognizing our children earlier. So if I'm looking at my daughter at any given point and I have the array of values that represent, you know, all of those activations that are occurring. And then I look at her another time and those are not the same. And yet I still am recognizing her to be my daughter. Is that a practical example that would be?

• 19:02Speaker 00

  Yes, exactly.

• 19:03Chris Benson?

  Okay.

• 19:04Speaker 00

  A hundred percent. And if you look at the neural activation of the output of that network, whether it be a computer vision network that you trained for like, Facial recognition, whether it be your actual neural network in your brain, the output is going to be similar in the sense that the most of the neurons are going to have roughly similar activations, right? Okay. Still, some of them are going to have very different activations because maybe the background is different. You know, many of them are going to, even those that match are not going to match exactly. But on average, it's going to be very, it's going to be close-ish. Okay. And so the very first thing you have to be able to do is take a very, very large collection of vectors, billions of them maybe, right?

• 19:48Speaker 00

  And take a new vector and say, okay, give me everything that just correlates with this. Okay. Okay, has a high correlation with it or has a low, what's called Euclidean norm. So I sum up all the square differences and take the square root of that. Okay. Right. So you're normalizing it essentially. Yeah, or maybe just think about it as a ball in space. So if you think about a high dimensional vector, you can think about a two dimensional vector as two numbers, right? Okay. So two numbers I can put on an X, Y axis, I can plot them on a piece of paper, right? So a three dimensional vector would be like a point in a three dimensional space. A four dimensional vector is going to be in four dimensional space, becomes a little, kind of hurts your brain to think about it. But the math is the same, you know, and you can take four to five to a thousand, right?

• 20:32Speaker 00

  And that is exactly the same. So you can think about Euclidean distance. Thinking about a ball in two dimension is a circle. Yep. A ball in three dimensions, a sphere. And a ball in a thousand dimensions is a ball in a thousand dimensions. It's all the points in distance less than something from the center. Okay. And you can ask yourself, what are all the points that are in this sphere? Okay, which is that all the points whose distance to the image that I'm looking at right now is smaller than some threshold. Okay. Which, if you remember, proximity of those vectors translates to similarity

• 21:09Chris Benson?

  Okay.

• 21:10Speaker 00

  in the real world. And so, looking at your daughter, again, this would be converted to a neural signal. Okay, you will then go to a very large bank of everything you've seen before and say, give me everything that looks like this. That is similar to this, close to this, right? What you would retrieve are mostly your daughter's face.

• 21:31Chris Benson?

  Right, right.

• 21:32Speaker 00

  Which is the answer in some sense to what am I looking at?

• 21:35Chris Benson?

  So there's a very different architecture in the sense of, just to draw a comparison. If we look at this vector database concept that you're discussing. And for a moment, we look back for a moment at the relational databases and you know, you talked about having those strings as IDs and stuff. Clearly, we have a significantly different way of retrieving data across those. Rather than having an exact match with a specific record. You're looking at generalizing across those billions of different records in the vector database. For that kind of commonality, even if none of them are exact matches.

• 22:07Speaker 00

  Correct.

• 22:08Chris Benson?

  Right?

• 22:08Speaker 00

  Correct.

• 22:09Chris Benson?

  Okay. And how do you approach that? How do you architect such a thing?

• 22:12Speaker 00

  Good. So retrieving or solving this problem efficiently is incredibly challenging. It's also like, it's a very well-known and hard problem because there are tens of open source solutions for it. Each one of them has many different algorithms with tens of, you know, internal parameters and so on. And they all excel at different regimes and dimensionality and statistics of the data. And like they trade off, like, how often do you like miss stuff? And how fast it is and so on. It's a really gnarly problem. For me, as a scientist, that's exciting because that means we have hard problems to solve. But for practitioners, that's often daunting.

• 22:55Speaker 00

  And so, when you work with something like Pinecone, we can alleviate that. We can at the very least delegate that to the experts and just say, hey, you know, just do the right thing for my data. I just need to like build an application right now.

• 23:09Chris Benson?

  Okay.

• 23:09Speaker 00

  Maybe down the line, I would wanna choose exactly which algorithm does what. And like, I would wanna fine tune the parameters. But for now, just give me kind of the plain vanilla. Give me just something that works usually, right?

• 23:22Chris Benson?

  Uh huh.

• 23:22Speaker 00

  So yeah, the algorithms themselves are. Like I said, I'm happy to dive as deep as you want, like all the way down to the nuts and bolts. I don't know how much, you know, you want to go there.

• 23:34Chris Benson?

  Because I don't think we've said it explicitly. Is it fair to say Pinecone is a vector database for machine learning?

• 23:39Speaker 00

  It is fair. I think technically that's the right thing to say. I think it's also like a search engine. It's its own thing. I think if you call it a database, it's in some sense an analogy. It's also like a search engine.

• 23:52Chris Benson?

  Can you differentiate what those two labels, saying database or search engine? Can you talk about what the implications are for each of those and how it fits in? And I will forecast, you don't have to go kind of explain the concepts, but I'll also ask you about some kind of use cases on where you might use those in a moment. So if you can kind of conceptually address it and then let's dive into something really practical about how you might do that.

• 24:15Speaker 00

  Sure. So I think that as a whole, when you tell somebody, when just the word database is loaded usually with

• 24:21Chris Benson?

  It is.

• 24:22Speaker 00

  concepts like joins and analytical workloads and like transactivity and consistency. And we don't do any joins, but we do care about consistency and persistence and so on. So we have, you know, it requires some properties of a database, but it isn't really a database. It's definitely not an SQL engine in any way.

• 24:43Chris Benson?

  Gotcha.

• 24:43Speaker 00

  A search engine usually is more focused on retrieval of relevant items quickly. Small set of relevant items quickly, which is much more aligned with what we do right now, right? But search engines are also confusing because people really strongly associate them just with text.

• 25:02Chris Benson?

  Right.

• 25:02Speaker 00

  Right. So people assume, oh, it's a search engine. It's a text you put in like text queries and you know.

• 25:06Chris Benson?

  Another loaded term, really.

• 25:08Speaker 00

  Right, exactly. We were just like so branded to think about like Google and Bing and whatnot. So I think both of those terms, like in terms of infrastructure, we overlap a lot. And we have to provide the same primitives. In terms of just terminologies, they're both bad.

• 25:27Speaker 00

  Equally bad and equally good, by the way. You know?

• 25:29Chris Benson?

  If you're willing, actually dive into that a little bit. I'm really curious. I'm fascinated by this because it's something I don't know much about. How do you store the data? We've talked about the storage a little bit and we've talked about retrieval. What does that look like? You know, as you said, that you can kind of dive a little bit, dive for us a little bit and tell us, you know, because I think when people think databases and search engines, as you just pointed out, they have a preconceived idea in their mind. How do you implement those terms? When you're taking the phrase database and the phrase search engine and you're saying, it's not the way you were thinking before. What's different about it? Tell us a little bit about what that looks like.

• 26:06Speaker 00

  So first of all, because of the nature of the data, basically, the only way to implement this efficiently is to have everything in memory. Okay, so first of all, we're an in-memory database.

• 26:16Chris Benson?

  Okay.

• 26:17Speaker 00

  The second thing is that retrieval, okay? So there's like similarity search or nearest neighbor search. It's also called this retrieval of close vectors or correlated vectors from a very large collection. It's a very compute-heavy operation. Okay, you really have to oftentimes touch and do some basic computation on very large portions of your data. So this is a compute-heavy process. Like, of course, the goal of the infrastructure is to make it as lightweight as possible. Right? But it still is significantly more tasking. So let me kind of maybe explain very roughly how you'd even approach such a thing.

• 26:57Chris Benson?

  Okay.

• 26:57Speaker 00

  I'm going to try to convince you that you can think about a thousand dimensional space, like a three-dimensional space right now.

• 27:03Chris Benson?

  Okay, I'll buy into it. I'll try.

• 27:07Speaker 00

  You know, I'm going to put warning signs where the math doesn't work. But for most concepts, the math, kind of the intuition is actually okay. Okay? So the first thing you want to do is to be able to just carve out chunks of space and to say, oh, if I'm here, then my close neighbors are usually within the same part of the space. Right? So think about partitioning the universe into like galaxies and, you know, like solar systems and so on. So you say, oh, you know. Maybe it's good to start searching for my nearest star in my galaxy, right?

• 27:41Chris Benson?

  Sure.

• 27:42Speaker 00

  It's a good guess, right? So usually you'd have a partitioning of the whole set of points into these like clusters, right? Such that in real time, you have to figure out, oh, I'm not going to search everything. I'm just going to search through these subsets of clusters. These are the areas of space that are going to more likely contain my memories, like the thing that, like the closest neighbors, right?

• 28:08Chris Benson?

  Sure.

• 28:08Speaker 00

  Within those clusters, you have a more compute-heavy process that actually kind of combs through everything, right? And figures out what's close, but it does that very efficiently. It doesn't actually compute everything. It kind of has very, very efficient rules to kind of discard things that are not really close, right? So you kind of have to raise a flag when you see something that's suspicious. Like, oh, this is like maybe highly correlated, but you have to rule out things that are not relevant very, very quickly. So you have this like second comb of everything that rules out everything that's impossible, that's irrelevant, and just leaves a smaller, even smaller candidate set.

• 28:49Speaker 00

  And then you have a final layer that just goes over that and really fine combs through and really does compute all the distances. Oh, you know what? From those, you know, here are the actual like 10 or 100 most relevant points. Now, if that process sounds like lossy to you, that you might actually have missed something, you're absolutely right. It's not foolproof. And so that's one of those properties that people kind of are confused about or surprised about with the analytics database. What do you mean it's not always accurate?

• 29:19Chris Benson?

  Yeah, you're missing, you know, in that relational world, which is addressing a different problem. You don't have that exactness that you have with those IDs tied to something. It's a different thing you're trying to achieve here.

• 29:29Speaker 00

  Right. And so you kind of have trade-off between how quickly you need to retrieve the results versus how accurate you want them to be, right? Because I can just find, comb through everything. Like, I can just decide to just brute force, to just kind of search my entire memory. You know, front to back, that will take an hour, but it would be a hundred percent accurate.

• 29:51Chris Benson?

  Yeah.

• 29:52Speaker 00

  Or you can be very, very picky. So I'm not gonna, you know, I'm gonna set all my thresholds really high. I'm not gonna touch anything unless it's like very likely a good match. Well, you might miss some stuff, but it's going to be very fast. And so

• 30:03Chris Benson?

  Right.

• 30:04Speaker 00

  And not only not getting in the point that there are like 5,000 different ways to do each one of those three stages.

• 30:11Speaker 00

  Yeah, I think that's maybe as deep as we want to go, but you can see how that is very, very different than a like a key value lookup or a B tree. It's just a whole different thing.

• 30:21Chris Benson?

  That was a really good explanation. I truly have that now in my head as to what the difference is. I know that there's lots of different ways to apply that in the real world to get productive solutions. Could you talk a bit about different areas that you can take this technology that you have at Pinecone? The vector database technology and the retrieval and where would we put that to get value out of it in the world and to help us solve our other problems?

• 30:48Speaker 00

  So interestingly enough, we can talk about very concrete situations, very concrete solutions. Some of them built with Pinecone, some of them built with by big companies use building their own technologies. And, you know, frankly, we as consumers of technology, we use these algorithms all the time. If you use Facebook, your feed is ranked by the embedding that Facebook has of you as a user. All of your tendencies and interests and everything are encoded as a high dimensional vector. And the topics and contents of a piece of content, whatever it be, like a piece of text or an image or whatnot, is encoded the same way.

• 31:31Speaker 00

  And the ranking is decided based on the similarity of those encodings, right? The same is true for pretty much every social network out there. It's the same way in LinkedIn and so on. If you search on Google, you know, if you just search any long query, right? That doesn't match directly like word to word.

• 31:51Chris Benson?

  Yeah.

• 31:52Speaker 00

  You might be looking for an answer to something that the right page that contains the So you wrote, you typed a question, but oftentimes the answer doesn't contain the question verbatim, right? It just contains the answer.

• 32:06Chris Benson?

  Yeah.

• 32:07Speaker 00

  You know, and so like text search doesn't make any sense. Like you shouldn't be looking for the words in the query. So what should you be looking for? And what you're looking for is an encoding of the query with NLP models into a high dimensional vector. And you're searching through the web where each paragraph was also encoded into high dimensional vectors. Such that, you know, it's still close if the paragraph might have contained the right information or the answer to that query. And same in retail recommendation for shopping, same as threat detection, anomaly detection, you name it. I mean, in some sense, all of it works this way nowadays.

• 32:51Chris Benson?

  I'm going to kind of throw something in kind of randomly. Could you think of it as a proxy for identifying intent? Possibly in some of these cases by comparing high dimensional vectors to where, you know, what Facebook wants to know or what Google wants to know is what is my intent with my search? Less about the exact words I'm using and more about what is Chris Benson trying to get to? What is that thing that he's trying to unlock? Is intent one of the possible things that it's proxying for there?

• 33:19Speaker 00

  So the main question is basically what do your embeddings represent? You train those models, you know? We talked about vision a lot in this session, you know? That probably encodes like visual similarity. But when you do feed ranking, then yeah, I mean, you would encode for intent and interest. When you do for maybe search, it would be more intent, maybe. And feed ranking is more like interest or maybe also intent. I'm not even sure. Yeah. So, yeah, I mean, those are, it's, you can encode a lot of semantically deep properties of the object that you're using. Whether it be, you know, person's actions on your website or an image or a piece of text. I mean, it's the same in some sense.

• 34:01Chris Benson?

  Okay, let me ask you another question. With a lot of folks listening to us here being practitioners and in their day job, they're maybe creating convolutional neural networks. They might be involved in NLP like my co-host Daniel is deeply all the time. Are there certain algorithmic approaches within deep learning that vector databases and retrieval lend themselves to more than others? Or is it more it can be used across any of these different algorithms? Should people always be thinking about it? Or are there certain areas where it will be more productive than others?

• 34:32Speaker 00

  I think they can be applied pretty widely, but there are patterns. So you said Daniel is an NLP practitioner. I saw you interview the folks from Hugging Face, for example. So a pattern that we see is people use models like Hugging Face, like BERT and others, to transform their text to high dimensional vectors and maybe just produce semantic search on top of their corpus of documents with an engine like Pinecone, right? So they would say, I would, instead of encoding, taking the tokens and putting them in Elasticsearch, I would convert them to high dimensional vectors and put them in Pinecone and search through the vectors themselves. And then I'll be searching through the semantic meaning of the document rather than the words themselves, right?

• 35:17Speaker 00

  That is very common. Now, usually if you do just that, exactly what I just said, you don't get the best results. You probably want to tweak your algorithm or retrain it, or you might need to do a few extra steps. But even if you do just that, you already get something interesting, which is very different than traditional text search, right?

• 35:38Chris Benson?

  Can you also maybe give us what your, kind of your classical or whatever feels right to you, but kind of a classical workflow? So if somebody is out there and they're using TensorFlow or PyTorch or whatever the tool that they want to use, and they're listening to this and they're starting to think, "Oh, I definitely need to incorporate this into my workflow," how do they do that? How do people start to get Pinecone and take advantage of that in doing whatever they're doing? What does it look like in a very, you know, fingers-on-the-keyboard, practical sense?

• 36:09Speaker 00

  It's incredibly simple. So we offer Pinecone as a fully managed service. And so all it takes is like asking for an API key on the website, which is free, and you get it like a minute later in your inbox. And after that, you can literally go to our Tutorial section and run notebooks and go play with, you know, all the APIs and all capabilities are open to you. You know, after a few weeks, you can either, you'll probably decide whether that's the right tool for you to use, right? Most projects end up being like very cheap to be able to run sustained workload even.

• 36:44Chris Benson?

  Gotcha.

• 36:44Speaker 00

  Usually less than a hundred bucks a month is typical for most like non-enterprise users. And yeah, I mean, it's right there and it's available to you. You can use it as a like microservice in your application or as a part of like your developer pipeline. Yeah, all the APIs are open. They're all documented. They're pretty simple in my opinion.

• 37:04Chris Benson?

  Very cool. So I'm curious, I want to go back for a moment. You were talking about some of the social media applications, you know, with Facebook. Are there any other areas specific to call out that are really notable for using these technologies? Are there any things that people might go, oh, I hadn't thought about, you know, the fact that this makes sense to use there?

• 37:23Speaker 00

  Yep.

• 37:23Chris Benson?

  Any other use cases to call out?

• 37:25Speaker 00

  So we definitely see text embeddings a lot, especially because that the models for embedding text are quite easy to get a hold of and they're pretty good already. We do see that with images, searching for images, searching for near duplicates of images for fraud, for social networking and so on. So people, you know, upload, you know, maybe slightly morphed of images you might already have in your network and you just want to flag that, right? So think about like, I've seen this before function, right? But it's not the same file. It's just a slightly distorted or slightly intentionally changed image, right? We see that with fraud and like kind of fraud and anomaly detection

• 38:09Speaker 00

  are very heavy use cases where you encode some normal behavior and then like in real time you retrieve, you know, out of a hundred million events that I've seen, you know, does this kind of look like a good thing or a bad? You know, what does this look like, you know, and should I be looking at this now? Should I flag this to my analysts or whatnot? I think those are the main patterns, but and recommendation, by the way, and I forgot to say it, recommendation and personalization are a big thing for shopping, for social networks and so on. I think those are kind of the main four, if I had to guess, but I'm probably missing something.

• 38:48Chris Benson?

  Sounds good. As we wind up here, I want to take us back for a moment to something you said early on. And you talked about how we're early on in this stage as we're starting to move into this new way of approaching technology solutions and you know, this is early days. And as someone who is working in this field, you know, all day, every day, and building your company based on these technologies, where do you think it will go and what would you like to see? And how will this set of technologies evolve in the years ahead in your view? What are you, when you're thinking, when you're laying it bad at night before you go to sleep and you're thinking, that's the thing I'm aiming for down the road, what does that look like?

• 39:27Speaker 00

  It's a fantastic question. I think that it's hard to pinpoint exactly. Again, I was trained as a scientist, we started off by talking about my meandering trip through my, uh, my PhD. I think my professional life is sort of like a scaled out version of the same, you know? I personally try to push the boundary and try to figure out what's kind of, what's the next huge milestone we as a society can hit. For Pinecone specifically, you know, for us as a community, I think that really starting to think about not only the model that I trained and how do I put it in a container and ship it to someplace in my app?

• 40:07Speaker 00

  But rather like, okay, we now have this like just complete wealth of data that we can now use and how do we integrate intelligence with, you know, compute and memory, you know? And storage, how do we make that behave a lot more like a brain and not like a neural net? You know, and I think that we are many years away from that. One of my favorite books on AI has this quote that says that saying that humanity is getting close to a general AI is like saying a monkey climbing a tree is getting close to the moon.

• 40:45Chris Benson?

  I hadn't heard that. That's good.

• 40:46Speaker 00

  You know, it's, it's true. It's true technically. So I don't know if we're as hopeless as that monkey on climbing the tree and getting to the moon, that is. But we are very, very far away. It's far away when we look at application, when you really dig in and you try to solve even very basic problems. You understand how little we know. Like we don't even know how to train neural nets.

• 41:10Chris Benson?

  I'm going to spin that in a way that I think that that's going to make for a lot of great episodes for us to talk about. We'll have to have you back on the show to discuss some of those steps along the way there.

• 41:19Speaker 00

  I'm happy to do that.

• 41:22Chris Benson?

  Well, Edo, thank you so much for coming on Practical AI. It's been a really fascinating conversation. I have learned a lot personally, and I appreciate you answering all these questions I have for you. So thanks, and I hope to have you on again sometime.

• 41:32Speaker 00

  It's a lot of fun. Thank you, man.

• 41:37Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it, and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Brake Master Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it, and we'll talk to you again next week.