Ep 32: CEO and Founder of Pinecone Edo Liberty on Pioneering Vector Databases, Barriers to Productionalizing Models and Why What’s Happening with GPUs is Not Sustainable

Pinecone has raised over $130 million and was most recently valued at $750 million. On this week’s Unsupervised Learning, we sat down with CEO and Founder of Pinecone, Edo Liberty. Pinecone is arguably one of the most important elements in today's modern datastack. Edo shared with us the most common use cases of Pinecone, the evolving landscape of vector databases, challenges in building vector databases, the "painful" launch of serverless model, and what people get wrong the most about Pinecone.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Today, we're joined by Edo Liberty, CEO and founder of Pinecone. Pinecone is a vector database that's raised over $130 million, most recently at a $750 million valuation. And they've really become a core part of the common set of tools that folks use to build AI apps. We hit on a bunch of really interesting areas, including the vector database landscape and how it's evolving, common barriers that enterprises have to building effective AI apps. And Edo also shared his thoughts on the infrastructure and AI opportunities ahead in the application layer as well. I think folks are really going to enjoy this. Definitely stick around for Pat and my debrief here at Unsupervised Learning. Edo, thanks so much for coming on. Really appreciate it.

• 0:32Speaker 01

  Thank you, guys.

• 0:33Jacob Effron?

  You know, you were obviously working on Pinecone before this generative AI craze and the launch of ChatGPT. And what was it like for you guys when ChatGPT came out? I mean, was it like immediately clear there's going to be some spike in usage? Or when did you first kind of start observing just the insanity that I imagine followed?

• 0:49Speaker 01

  I'll start by saying that vector databases, I think, are still vastly underutilized for a lot of use cases, for semantic search, for candidate generation, for ranking, for feed ranking, for like recommendation, for a million different things, right? When I started building Pinecone, this was really, it was sort of like a well-kept secret. Well, a well-known secret, I should say, that the bigger companies like Google, Amazon, you know, whatever, like use that internally for search, for recommendation, for feed ranking, for anomaly detection and so on. But then educating the market was fairly difficult.

• 1:29Speaker 01

  And I would go to, you know, investors and say, you know, vector database. And they would kind of look at me confused and concerned and...

• 1:38Jacob Effron?

  I'm sure they emailed you in 2023 and they were like, we believed all along, right? We've always been fans of the category.

• 1:44Speaker 01

  Yeah. Actually, I'll tell you a funny, a funny thing that the actual emails I got after that. But just to make matters worse, I had just come out of AWS where I helped build a product called SageMaker, which was an ML ops thing. And like ML ops was like all the, all the rage, you know, before, you know, AI kind of took off, right? So everybody thought it was building a, like an ML ops product, right? So everybody was very happy to meet with me and then they're very disappointed that I'm building something called the vector database. What is that like? It sounds like crazy insane, right? And then with OpenAI, just kind of elevating uh the discussion to the broader

• 2:29Speaker 01

  audience, like the technology really hasn't changed significantly, to be honest. I mean, the the the for practitioners, like nothing very deep has changed, except for the fact that now there was a lot of oomph behind it, a lot of capital and a lot of energy behind it. So it started started uh being accessed by the non-AI engineer. When that started happening, we started ramping up so quickly that we started really exhausting whole kinds of machines in in GCP and AWS clouds. We started like really spending millions of dollars a month on our free tier. It was complete insanity. It was uh the entire

• 3:11Speaker 01

  team was dedicated, just like spinning up more and more environments and just being able to keep up with the demand. I think at our peak, we started having 10,000 signups every day, uh which really pushed us to really think about scale and efficiency in a completely different way. Like we we kind of, we had to go back to the drawing board and redesign the entire solution, uh which later yielded the solution we have in place called now called serverless, which is like maybe two orders of magnitude more efficient. But it kind of came out of just necessity. We just couldn't, we literally couldn't handle the load. Uh the funny thing is, if you want to talk about investors, right? I did have quite a few

• 3:55Speaker 01

  different investors write to me after like that, like vector databases have become like the, like a true category. In some sense, we've already taken the lead. And they say, they went, once they went, they wrote to me and said, hey, you know, I actually looked back at the notes that I took like three years ago, and you actually said you were gonna build this. And I'm like, yeah, uh, cause that would, that's what I was gonna build. And they're like, okay, like we just that, it's not very common that somebody says they're gonna build something and they actually build it. And that's actually succeeds like, that's just not very common. I'm like, okay.

• 4:36Jacob Effron?

  So funny. I guess they'd, they'd all thought it was a, an ML Ops platform, but then they, they read, they read back through their notes and they, they realized what it was, uh, what it was all along.

• 4:45Speaker 01

  Exactly.

• 4:46Jacob Effron?

  Was there like a particular moment that you can remember that like spiked usage, you know, early on? Was it like immediately apparent to folks as they were playing around with GPT three that they needed a vector database? Or was there like a tweet or something or, uh, in the early days of, of that usage spiking?

• 5:01Speaker 01

  I think one of the, one of the kind of moments that really just completely blew up, there was this, uh, there was this open source like gadget called AutoGPT. That was this sort of like, super like minimalistic, but sort of like precursor to something we would call an agent now. And it was sort of like a, a, a piece of nothing. It was really just like a, a, uh, but I, I think even one Python file or just a handful of Python files with some like spit and gum and you know, whatever. But it like, like agents do, it started like doing something potentially interesting. It was open source and so on. That thing just took off like crazy.

• 5:47Speaker 01

  And, you know, suddenly the people that started onboarding Pinecone were not systems engineer, building AI platforms and RAG applications and so on. They were like the dentist that remembers Python from college and wants to play with AI, right? And then like things just completely went insane, like there's, there's way more dentists.

• 6:16Speaker 02

  I guess a lot of dentists that remember their

• 6:18Speaker 01

  Yeah, luckily enough, that kind of like normalized again. And the people who come today are mostly builders. But for a while there, it just completely went insane.

• 6:29Speaker 02

  Yeah, it seems like there's a huge breadth of applications that are built on top of Pinecone. What are some of your favorite applications or some of the ones that really stand out to you? Obviously, we were talking about Notion before and it sounds like Pinecone really shines for scale as well. When people have kind of millions of vectors that they want to run workloads on top of.

• 6:51Speaker 01

  So millions is actually the small scale today. When we speak with customers and they have millions of vectors, we say, great.

• 6:59Speaker 01

  You knock yourself out, you'll be hard pressed to spend more than a hundred bucks a month. Like, you really are going to have to exert energy to try to do that, right? Where we truly shine is at the hundreds of millions, the billions scale, where companies themselves are SaaS and oftentimes sort of like software providers. Notion is a great example, Gong is a great example, many others that actually, they themselves develop AI, Q and A, semantic search, like deep AI solutions for their own customers' data. So they, like each one of their own customers might have a million documents, right?

• 7:43Speaker 01

  But they themselves have 10,000 customers. And suddenly they find themselves needing a vector database that can handle 10 billion embeddings, maybe across a thousand partitions or 10,000 partitions and do that incredibly cost effectively, right? And when you try to do that, especially nowadays where people really start understanding, hey, I'm not just playing in the lab and building a true product. So I really need to care about unit economics. They want to figure out what's like, what's that going to cost me? And then we can come in and say, because of our new architecture, because we went through this insane spike a year and a half ago, and we've re-architected everything and now we've launched it as serverless.

• 8:26Speaker 01

  We can run this multi-tenant workload such that one of your costs.

• 8:34Speaker 01

  For one of your paying customers on Pinecone might be a dollar a year, like, like sometimes fifty cents a year, right? For a paying user, right? And then the unit economics really shine, right? So you said, you know, if you have a million, you know, vectors, right? I mean, a million vectors, like, okay. If you have a million vectors 10,000 times, you really don't want that to be a hundred bucks each. You really want each one of them to be fifty cents each. And then that starts making sense, right? And that's where we truly shine, because then you really can can operate this thing at scale.

• 9:11Speaker 02

  On the application side, are most people running RAG or doing semantic search? Or I guess what types of applications do you see? I can imagine, you know, the Q and A at Notion, that's kind of one. And then I can imagine there's tons that Gong is doing as well with with vectors and turning videos into vectors.

• 9:31Speaker 01

  Yeah, Q and A and semantic search are very large applications, very well understood. Chatbots, support bots, uh, which are sort of like Q and A, if you will. Uh, but special flavor of that. Uh, people do, uh, legal discovery, uh, medical history discovery and all sorts of, you know, analytics on that. Uh, which again, you can think about them as flavors of search in in some sense, right? There's a lot more that has to do with images and audio and video and anomaly detection and security, which is starting to become like a big thing. Pharma and drug discovery are, you know, we see some applications like that.

• 10:11Speaker 01

  But the kind of meat and potato is really just the kind of text and and images and search and, uh, you know, RAG over that in some sense. And by the way, RAG in and of itself is sort of like, you know, based on semantic search. It's it's like the The magic sauce happens at the semantic search layer, not at the reformulation layer.

• 10:34Jacob Effron?

  That makes a lot of sense. And I guess, obviously, you know, I think in the beginning, folks have used, you know, embeddings for, it feels like mostly text data. And there's been, you know, some in the image world. But I'm wondering, you know, we've had different perspectives on this podcast about, you know, how much multimodal stuff we'll see this year and whether, you know, I'm curious, you obviously have a firsthand view to it. You know, are you seeing lots of folks working with kind of image and video data? And do you think that we have the underlying tools required for those use cases today? Or you know, still expected to be a text dominant 2024?

• 11:03Speaker 01

  So, look, I mean, I am, unfortunately, I've been in AI or machine learning before that, or big data before that, or all that, you know, for 20 years now. And so, I think I have a good mix of enthusiasm about the promise of tomorrow and a healthy focus on the problems of yesterday. Right? Even today, right, with the advance of foundational models and, you know, general AI and all that stuff, the average company still finds it hard to

• 11:43Speaker 01

  even like, train any deep learning model at all. Yeah, like anything, like a million parameters, not a billion, not 10 billion, not general AI, nothing, like any model. Same goes to search and so on. So, you know, I think we need to be very kind of realistic about what is possible, right, by the trailblazers, the research labs and the technology leaders, and what is an actual technology that companies actually use. Right? And those could be five years apart sometimes. Right? So, can we already see amazing things with multimodal? For sure.

• 12:24Speaker 01

  Right. Do I think multimodal is going to hit the kind of mainstream technology developer in the next year or two? Unlikely.

• 12:32Jacob Effron?

  Yeah, I mean, I think it's interesting because you are working hand in hand with these companies that are deploying a lot and many times their first generative AI products. And so I imagine you do see these barriers or the places where folks fall down and being able to do that. Like, what do you feel like the biggest barriers today in terms of getting, you know, these, training these models and getting them into production for some of the classic use cases that we're seeing around, you know, support or search or others?

• 12:55Speaker 01

  I mean, one of the biggest ones is hallucinations, right? They're just, they're not trustworthy, right? They just don't, whatever. They're compelled, they're large language models. They're designed to generate language, right? And when you are compelled to write a page of text on something you know nothing about, you will write a page of text. And that will contain stuff, you know?

• 13:22Jacob Effron?

  Pat and I do that all the time, I was gonna say.

• 13:23Speaker 01

  Yeah, we've all been in like sixth grade and I remember that experience very well, like cold sweat. I'm like, shit.

• 13:34Speaker 01

  That's the mode we put these L L M's in, right? And so like, and then we're sort of like surprised that what comes out is nonsense, right? I think that's a huge problem and I think that's something that we as a technology would have to, as a society, I mean, not we, Pinecone, I mean, we, the world, have to grapple with and really bring really good solutions to. I think you will start with AI startups and technology companies that build on Pinecone, build on models and so on, that have already know how to work with those technologies, building great vertical solutions for support,

• 14:13Speaker 01

  for medical history, for legal discovery, for, you know, you name it, like, it's a very long list of applications, right? And then I think two, three years down the line, you would see maturity in the broader market where now like every large company would use some of those tools, but would also use vector databases and models and, you know, visibility and like a bunch of other AI specific technologies, kind of natively for their own development where this has become. So commonplace, so well understood, so ironclad and so well ingrained in the developer culture that just people, there's just enough there that people just know how to use it well.

• 14:58Jacob Effron?

  I feel like hallucinations come up time and time again. I guess, have you seen, what do you feel like the best way to, for folks listening there, or they're also trying to figure this out, what are some of the best things you're seeing in terms of ways to get around this?

• 15:09Speaker 01

  There are a few different really important efforts on this front. First of all, we are only now starting to truly be able to even measure hallucinations.

• 15:22Speaker 01

  Right? To even know, like you can, for example, you can design a model. I can write a model that never hallucinates by just always saying, "I don't know."

• 15:35Speaker 01

  Right? That's not hallucinating, right? It's also completely useless, but it never makes a mistake and, you know, whatever, like it never missteps, right? Okay, is this worthwhile? Like, should we celebrate that outcome? I don't think so, right? Okay, so you really want to measure kind of usefulness and correctness and faithfulness to the data that you were trained on or the data that you have in a vector database to do RAG on and so on. So there's just these measures of even what is good is already like a very complicated question.

• 16:16Speaker 01

  And I think that there is a, another huge effort that has to do with, frankly, with RAG and with vector databases and with knowledge layers and capacities and people grappling with, okay, how do I now make my data available to the model in a way that's secure, that is well understood, that is data governed, that, you know, I can maybe delete stuff that I, whatever, like I was GDPR told me that I have to delete and somehow I need to know for a fact it's not somehow a part of the model anymore and so on.

• 16:53Speaker 01

  There's a lot of work there and that's, you know, a topic we actually made a lot of progress on already. So, by we again, I mean not only Pinecone, but the community like starts to have pretty good solutions for.

• 17:05Speaker 02

  Super cool. It seems like a Hacker News post in the making, an LLM that will never hallucinate, just respond with, I don't know, every single time.

• 17:18Speaker 01

  Like creatively avoid the question.

• 17:20Speaker 02

  Exactly.

• 17:22Speaker 01

  But we know, like, we know, I mean, jokes aside, we, we already today can load all of, what have like a, oh, but a good chunk of the internet uh into Pinecone as, as an index. And basically run, run like a language model next to it with, you know, not incredibly sophisticated RAG and already outperform GPT-4. Like today, it's, we've already ran experiments like this. So, like, I, I think there's something there. And I think we're, we're going to see some significant progress there, both on, sort of like consumer facing products and like B2B products.

• 18:04Speaker 02

  Vectors are the language of AI, uh, or kind of what AI speaks. And there's this land grab to store them. Everyone wants to store the vectors. And because of that, there's kind of been this explosion. I mean, you would know this better than, than us. But there's been this explosion of startups uh that are going after storing vectors, but then also incumbents and folks like Mongo that are adding support for vectors. So how do you think about the landscape of, you know, the startups that are kind of vying to, to store some of these workloads and then the incumbents as well?

• 18:36Speaker 01

  Don't get me started about Mongo. It's a company that fricking heard the word AI for the first time a year ago. And suddenly they're like experts. So, I mean, don't, I, whatever. Like storing vectors, I think is exactly what's happening. And I think you define it perfectly. Right? People understand that now this is a data type. Right, next to my sentences or my images, I'm gonna store a semantic representation of that object that came out of some embedding model, right? And that thing is a float array, right? It's whatever, like a thousand floats, right? And if your infrastructure, be it like a non-AI technology, say that rhymes with bongo,

• 19:20Speaker 01

  you know, you need to put that array of numbers somewhere. And so they have to have that as a type, really. And that's basically what's happening there, right? And so, you know, whatever, there's some work that has to be done to do that. That doesn't make your database natively be able to query those things and work with a data type at all, right? What people don't understand about vector databases and why they're so unique is that that numeric array becomes in some sense the key. Some conceptual way, that is the primary lookup. That is the way you organize data on blob storage. That's how you load segments.

• 20:03Speaker 01

  That's how you, you know, search through your universe. By the way, that's how your own brain does it, right? I mean, this is an incredibly unique and optimized stack that if you, you know, try to somehow bolt on this like data type and like pretend that it's now a vector database, you're gonna get incredibly poor performance and frankly run into like serious issues.

• 20:27Jacob Effron?

  No, I mean, I think it's really interesting because, you know, I feel like obviously, yeah, a lot of, a lot of other folks have, have come into the space. And I think generally there's been this question of like, you know, what's the, what's the RAG stack end up being? And I feel like you guys get thrown around all the time as a key part of that. And I'm wondering like, if you, you know, if you had to go to a company today and build a RAG product, like what would be your stack? Besides pine cone, of course.

• 20:52Speaker 01

  That's interesting. I frankly, I think a lot of people default to OpenAI. I, I wouldn't. We actually see really good performance out of smaller, cheaper, somehow easier to manage, sometimes open source models. There is no reason to spend a ton of money on that. There is.

• 21:19Speaker 01

  Sort of like an ML ETLing part of it has to do with bulk transformations and movement of data and so on. We've really started enjoying any scales product and we really like start partnering with them closer and we're really enjoying their stack. We work with Purcell and others on kind of launching applications and, you know, building the, you know, kind of app layer. You know, we partner pretty closely with Cohere and AI 21 and, and, you know, we're, you know, others like Hugging Face and others like

• 22:00Speaker 01

  we're all, whatever, they're all in the mix. We see, I think, a good combination of those models, both for embedding and for ranking and summarization and so on. I really don't have a good vendor for, or a good technology for evaluation yet. I'm, you know, I'm gonna, some people are gonna write back to me that they're angry that I didn't name them, but I think that that'd be very interesting to, for a leader to emerge in that stack. I think it's, it's very interesting and it's very hard to do right.

• 22:37Jacob Effron?

  And then as you think about kind of like Pinecone's footprint, I mean, obviously, I imagine a key part of, of getting Pinecone to work in an enterprise, just having the data ready to, to be put into a vector database, right? And I imagine there's all sorts of, you know, we've obviously met companies that are, that are focused on that. And then there's, you know, the metadata you apply to it, obviously, then the retrieval methods you use to, to, to take it from a vector database. Like, how do you think about ultimately, like, the footprint you want to have versus you're like, I would love, I would love there to be a best-in-class vendor that's, that's great at the ETL side and never have to touch that?

• 23:07Speaker 01

  Yeah, so you're right. I mean, I, and by, by the way, I apologize to the, the audience. I'm like, I'm like an infrastructure guy, so I, I kind of look at the raw machines and like very basic infrastructure. A lot of companies. That are stepping into the embedding generation and data generation into vector databases. They're actually doing really well, right?

• 23:36Speaker 01

  Unstructured comes to mind, but that is becomes a viable option like very soon. And especially for specific channels, right? So if you have a fixed, so if you have a common data channel, say you import your data, say with FiveTran from Snowflake. And now you have that data in a, in a table that has a unified structure. Because whatever, because it's the same table that every FiveTran customer would have. Then there are companies who would take that and, and transform it and feed it into a vector database.

• 24:18Speaker 01

  So you can rag over your, you know, say, you know, Salesforce, you know, notes on customers and so on, right?

• 24:26Speaker 01

  And so if people, if you understand that the data source that you have is like standard, then it's very likely you will find somebody who will have already built that pipeline and be very happy to sell it to you.

• 24:38Speaker 02

  And you were talking about how, how effective some of the smaller models can be relative to an OpenAI.

• 24:45Speaker 01

  Yeah.

• 24:46Speaker 02

  How much intelligence do you think will happen at the vector DB retrieval step versus at the generation step? It sounds like, you know, maybe over time, people are, you know, they want to run on, on their own data and, and,

• 25:03Speaker 02

  you know, maybe you guys are seeing good results with, with that as well.

• 25:07Speaker 01

  Like I'll give an economics answer to this, right? The market's going to have to find like the stable point where there's a good trade off between cost, compute and infrastructure that you're willing to run and the quality of the output, right? We're already at the point where making models bigger on GPUs is not reasonable. You can run an experiment, you know, you can flex and say, I have, you know, whatever, a hundred billion parameter model and it's whatever, it's like does XYZ, but.

• 25:45Speaker 01

  Whatever. You're not gonna run a hundred billion parameter model for every API call on your platform. That's you're gonna just, it's just, you're gonna go bankrupt, right? And you're gonna make the planet hotter. So like, that's not gonna work long term. And so people really have to find better solutions, right? Again, what we see is that with those model models, if you do things right, sometimes the results could even get better. And if they're, even if they're not better, they're not noticeably worse. And you might be operating on like, you know, cents on the dollar in terms of operating costs, right?

• 26:24Jacob Effron?

  Yeah.

• 26:25Speaker 01

  I have no doubt that we would move there. That doesn't mean the arms raised to whatever, like bigger, better models not gonna exist. I just don't think that's what the market's gonna operate with, the kind of, most of the market's gonna operate with.

• 26:40Jacob Effron?

  It is interesting. It feels like as these context windows were getting longer, you know, these last few months, people were like, oh, well, you know, is there a future for rad? You just stuff, you know, everything you'd want in a prompt. But I mean, I'm curious your thoughts on that. It seems like from a cost perspective, obviously, that, you know, if you're charging per token, that can get pricey quickly.

• 26:58Speaker 01

  Yeah, I mean, I think the fact that model companies allow you to bring in a bigger and a bigger context, I mean, I think this is the least surprising thing in the world. That's their pricing model, right? Like, shockingly enough, they're willing for you to pay them more. What a, what a, you know, what a shocker. There are multiple problems with it. Not only is it slow and expensive, it doesn't actually help more often than not, right? And, and finally, it just runs out of steam, right? You can say, hey, you know, whatever, like, write me a job description and here's an example job description to go off of. Great, that's great for a context window, right?

• 27:41Speaker 01

  If, like, here are like 10 job descriptions to go off of, okay, that also works. Here are a million job descriptions to go off of? Like, okay, that's not gonna work, right? Or whatever, like, write something and then get information from the web or get information from all of my documentation ever, right? That just doesn't work. I mean, this, think about that, like, sending all your company data to OpenAI.

• 28:14Speaker 01

  On every query, like that clearly is not the right thing.

• 28:19Jacob Effron?

  Totally. I mean, it makes it makes all the more sense as you focus on these perfect use cases for Pinecone that are hundreds of millions of vectors. It's like those are exactly the ones that, you know, would would never make any sense to stuff in a context window.

• 28:30Speaker 01

  Yeah, but but even even even at the small scale, I can I'm, you know, I want to paint the extreme because like that's where it becomes like almost like farcical, like obviously impossible, right? But even when you can create like, whatever, put a hundred thousand tokens in a in a context window. I mean, putting those documents in a vector database, you might be able to send instead of a hundred thousand, you'd be able to send 10,000. And that'd be just as good, if not better. What about 3,000? And you'd be paying whatever, like 10% and not lose any performance. So why

• 29:09Speaker 01

  not? Right? And so, again, even small cases where the context engine is theoretically workable, even then, it's very rarely the right, the best solution in terms of cost and performance.

• 29:24Jacob Effron?

  Totally. And I guess just generally, as you think about how folks are using Pinecone today, does it feel like there's, you know, folks that relatively standardized around the embeddings models they use or retrieval methods? Or one thing I'm always curious about is whether, you know, it feels like in the limit, there will be, you know, different retrieval methods for different use cases and different embedding models that are particularly good for, you know, different industries. And it's hard to tell from the outside how much that is happening today versus like, hey, let's just try and get this thing to work, you know, and use. But I'm curious what you're seeing.

• 29:52Speaker 01

  Frankly, most companies are trying to get this to work, right? Very few companies have done it long enough and have like really gotten the experience with it that they really start to now iterate over the embedding models and the retrieval and the reranking and the filtering and like really get very deep in the application stack and and figure out how to build this. And I think we're getting there. It's a part of the maturity in the market, like in the same way that any search or recommendation stack is is not just the retrieval. There's like a lot that goes before it and a lot that goes after it to really get

• 30:36Speaker 01

  the results that you need. And this is one of the main discussions that I have with. Larger companies that I work with that, you know, I tell them, hey, you know, this is going to be able to do amazing things, but it's not magic. I mean, it's not, you're not going to just hook it up and everything's going to be perfect. This is, you know, this is work. You know, we're going to have scientists and engineers work on this thing for multiple quarters and we're going to learn and iterate and improve and add stuff and remove stuff and we'll get there. Eventually we'll be able to do Q and A incredibly well and, you know, chat support and legal discovery and all that stuff. It is possible because we've been

• 31:17Speaker 01

  building these things for years now and we know it's possible. We also know it's not

• 31:24Speaker 01

  trivial, like and every use case is slightly different and every application has different preferences and different comfort zones on the ratio between speed and cost and accuracy and, and format and length of the input and the output and like freshness of the data and like you, you name it. There's, you just really have to build out the whole solution around it and that takes time.

• 31:45Jacob Effron?

  And I imagine, you know, one thing that must be interesting for you as a business is, and I've, I've heard this from other folks that sell AI tools in enterprises is you kind of land in and I'm sure they're like, oh, this is amazing. We have, you know, you own this awesome pine cone team, like build out our Q and A, you know, product for us or something. And, and, you know, they're so excited to have you there. And obviously, you know, I think at, at scale, you, you certainly want to be empowering the developers in these companies to, you know, uh, to, to build these things and to tweak things themselves. Like, how have you thought about, you know, where you're just, hey, it's the early days of the space. Like, we might as well just come in and help them tune the knobs and, and make this implementation work, uh, versus, you know, hey, we really want to, um, we're kind of going to be a little more religious about them just, you know, being, playing around with, with the different things that pine cone has to offer.

• 32:27Speaker 01

  So for, uh, the first three years of the company's existence, uh, my, uh, board was, uh, very unhappy with me because we were almost religiously not doing that. Like, we would get questions and we would, whatever, everything that we Understood was needed, we either documented very clearly that we don't do, or we made it very, very easy, or maybe completely transparent and just completely something you don't have to do at all, or something that we don't do at all. Like either we take care of it a hundred percent, or we do zero percent of it, right?

• 33:10Speaker 01

  And then we were very explicit about that, right? We had to oftentimes have very uncomfortable conversations with potentially large customers that need help with X, Y, Z, and we know how to do X, Y, Z, and we tell them, no, we can't help you because we're really busy building a vector database. But that really panned out in the growth phase of the company, where you really started having literally thousands of customers who not only don't like seek our help, they're not willing to meet with us. They're like, they're just happy and successful and they don't want to waste their time. It's like, hey, this shit works. I don't, I have better things to do.

• 33:53Speaker 01

  I don't need to talk to you guys. I'm like, okay. And so my sales team is agonizing about like, hey, like this is whatever. We need to speak with them. We don't know who they are. I'm like, I know, but they don't want to talk to us. What do you want to do?

• 34:04Jacob Effron?

  For the folks that like, you know, maybe are not talking to you and can't get it to work, like, is there one like thing you wish that people tried more like, oh, if only you would like use a different embeddings model? Or I don't know, whatever it is, like, is there one kind of common failure mode you see for people that are that are trying this and can't get it to work?

• 34:21Speaker 01

  Yeah, so I will say that as a company also, we've, we've grown and evolved and companies that do want to help, do want our help, we engage with them. We were not like a professional services team, right? We won't go and build a whole solution, but we are very, very happy to help and, and consult people on their journey. And we've started doing that a lot more now that we have more capacity, obviously, in the team.

• 34:50Speaker 01

  I think what people really get wrong is is cost estimations, right? I mean, this is one of those things that just, you know, people come to me as like, oh, we did the napkin math. And like, we would pay, you know, fifty thousand dollars a month to Pinecone, and that doesn't make sense to us. Like, how did you compute that? And then they would tell you, and you're like, oh, my God, like, you know, you just kind of like, just walk them through the the cost calculator as like five hundred bucks a month. I'm like, oh, OK, like, great. I'm really good. I'm really happy we had this conversation, right? So, yeah, designing their AI solution, even if it's,

• 35:34Speaker 01

  it could like seem like a small change, but that might make a huge difference in the cost of the infrastructure and latencies and freshness and a bunch of other stuff. And so we're very happy to help with that. People get that wrong on a regular basis. And for me, again, as a technologist, if you use the wrong embedding models, sure, your results might be slightly worse, slightly better, you can fix that. If you calculated the costs and you now don't embark on the journey because you figured it was going to be too expensive, and that calculation was wrong, you've just like, you're not even building something that you should be building.

• 36:16Speaker 01

  And that, for me, is a is a kind of sort of more final failure mode, which is much more painful and frankly annoying because, you know, I hear about some of those things in hindsight. Well, some really large customer would say, oh, we thought about you guys, but then like we feel we can't run this project. And then you're like, wow.

• 36:38Speaker 02

  On the cost side, one thing that we see across a lot of our infrastructure companies is they ship an improvement that Makes the database or whatever it is much more efficient and then people actually end up storing less or there's less compute or cost customers less. My guess is with the serverless announcement that customers could potentially be paying you less than they otherwise would have because you know they're now it's a totally serverless model like was that something? What was it like shipping that change and you know ultimately the best thing for your customers but potentially at the expense of some short-term revenue like how did you think about kind of moving to the serverless model?

• 37:18Speaker 01

  No, it's very painful for us as a company and like for founders listening, this is it's a hard transition and you know for investors to see your revenue kind of go up into the right in exactly the right exponent and suddenly that thing starting to flatten out and you tell them hey look at workloads actually grow faster now, right? But revenue is impacted by the fact that our product is so much cheaper now, right? That is a complicated discussion and that's like you know investors don't necessarily love to see that and even though they sort of like

• 38:01Speaker 01

  they understand it, of course they can rationalize it, but they still emotionally react badly to it. Right? My advice to founders is that kind of transition is much easier done earlier than later and so you should really move and do that as fast as you can and take the hit as soon as possible, right? You know, for us, we think we're leaving on the table more than half of our revenue.

• 38:39Speaker 01

  For some of our customers it's going to be 70, 80, 90% reduction in cost. For extreme cases in pharma and others where their workload is like super, super, super like storage heavy, this could be as little as 1 or 2% of the spend. Like we have companies that used to pay us a hundred K, they now pay us $2,000 a month, a year, sorry. It's like literally almost nothing, right? So they're very happy, I mean investors are not very happy about that, but at the end of the day, it's

• 39:18Speaker 01

  It's the right thing for the customer and it's the right thing for people building solutions on top of our platform. And so if we can't fit really nicely into their own cost structure of their own products, we're not going to be a part of the stack. Right. And, you know, when I think about the trajectory of the AI stack, we're really in the very first innings of that. Like every sign shows that the adoption curve is such that like we're very early adopters, you know, uh, you know, trailblazers, just

• 40:01Speaker 01

  getting to production and so on. At the end of the day, like almost every company is going to have a vector base doing something, either small or medium or large. Uh, we're not there yet. And so, you know, if that's where we aim, if we need to fit snugly into that cost structure for like tens of thousands of, or hundreds of thousands of workloads, we just have to do that. Even if, you know, there's a lot of, you know, pain, uh, in the medium term.

• 40:33Jacob Effron?

  Well, yeah, one thing, you know, I was struck by is you were kind of talking about how we're in the early days of, you know, AI infrastructure. Um, you know, a question that every investor is always, and folks in ecosystem are always asking is what are the kind of net new categories for startups versus, you know, natural places for incumbents to build product? Obviously, it seems like factory databases have been one of the space where there's been a lot of net new companies, you know, yourselves as at the forefront. As you look at the other parts of AI infrastructure, like what parts do you think makes sense for new players? What parts are like the existing infrastructure player is going to provide? Well, curious how you think about that?

• 41:05Speaker 01

  You know, frankly, I am much more excited about the application and the, the solution space more than the, uh, infrastructure layer, right? I mean, for new companies, right? This is, uh,

• 41:20Speaker 01

  In infrastructure, there's a winner take all

• 41:27Speaker 01

  you know phenomenon that clearly works great for us, but but that also just means that the you know, the the

• 41:38Speaker 01

  we're just the space for for new companies is is very limited, right? On the solution application side, I think there is so much energy and so much excitement and so many different uh problems that can be solved in creative new ways now. If that uh I frankly think every you know, every digital native has

• 42:09Speaker 01

  whatever, like 20 different startups that like say now, oh, like we are the new blah with AI. Right? And uh now they, you know, so you have 20 startups trying to uh usurp uh some some some long term player and you have that long term player trying to reassert themselves as an AI uh native solution. Uh and at the same time, you have enterprises both buying those technologies and learning how to use them. Uh and so this it becomes like this conveyor belt of innovation from tiny startups all the way up to enterprises.

• 42:51Speaker 01

  Uh that is just teeming with with innovation and and great ideas and a ton of value. So I I, you know, again, I literally see hundreds of customers, hundreds of like uh companies a week that that work with us and so on. And like each and every one of them does something unique and interesting. And you're like, holy shit, there's yet another application. Wow, there's more value. There's I didn't think about this. Okay, amazing. That's I guess so. Wow. It would have sunk it, right?

• 43:24Jacob Effron?

  I mean, I know you're an infrastructure guy, but if you uh if if you couldn't do Poncon and had and were instead building an AI application, what would you build?

• 43:32Speaker 01

  Uh it's gonna sound uh weird to you, but I I just really love uh Communication like like email and Slack and Twitter and like I just love that kind of data, you know, meetings, like transcriptions and like conversations. And so there's something about human communication, written or spoken, that is, I feel like just fascinating. And and and there's like a ton of knowledge to be extracted. It's all like super messy and

• 44:15Speaker 01

  exciting and complicated and and very rich. And, you know, it will be something like that, whether it's medical records, or emails, or JIRA tickets, or what have you, like it could be something like that.

• 44:30Jacob Effron?

  I thought you were gonna say something around surfing, but communications makes sense, too.

• 44:34Speaker 01

  I wish I wish there was like some, yeah, this, I thought we were limiting the scope to serve to tech companies if it was. Probably harder to build a really strong software business than surfing. Yes, exactly.

• 44:47Jacob Effron?

  I was like wondering, I was imagining like embedding surfing videos or something, you know, for technique coaching.

• 44:52Speaker 01

  Yeah, you can have a tech company where I surf every day and then pretend to have worked. I'm talking in ecology. There's actually AI that's running the whole time you're surfing. There's an agent that's doing all the work. Yeah, exactly. This is like my software company in the Maldives on.

• 45:12Jacob Effron?

  Yeah, I'm honestly convinced with some of Pat's hallucinations, that's what's happening on on his end. I guess, you know, also, I mean, you obviously were previous to, you know, interesting places. I mean, curious, like your reflection on maybe we'll go back to the most recent one in Amazon, like, how do you kind of think they're positioned right now? What would you be building if you were still there? Obviously, you did a lot of the stage maker work. Any reflections on that?

• 45:38Speaker 01

  Sure, you know, I.

• 45:42Speaker 01

  Look, I mean, different companies just need to have the benefits and also like the restriction of needing to focus on different parts of the stack, right? When you run at AWS scale, if your product doesn't bring in hundreds of millions of dollars a year, it's a failed product. It doesn't actually move the needle at all, right? Startups operate in a different mode and so on, and you know, your horizon for innovation and your appetite for risk is different and so on. So, you know, I don't know that Amazon, frankly, Google or Microsoft are even working on the things that, you know, we're working on.

• 46:24Speaker 01

  I can tell you that we are laser focused on stuff that we know for a fact is going to be a part of the stack and a part of the solution and is realistically going to hit the market in at least, you know, a year and a half or two from now because it's that early, right? In the same way that, you know, five years ago, we, people know, didn't know what the hell we were saying when we said Victor Rabe says, you know, we are now already working on the stuff that's going to hit the market in two or three years from now. So that's, that's, that's the most exciting thing for me, to be honest. This is where like the really cool stuff is, uh, kind of happens behind the scenes.

• 47:07Speaker 02

  Yeah, one thing that I think is interesting, especially around, you know, how company, how far in advanced companies and infrastructure companies like you guys need to plan is, you know, being able to skate to where the puck is going and, you know, kind of predict the future in some extents, especially in a market that is moving so fast where infrastructure is changing really, really quickly and context windows are changing, what the models can do are changing. Are there specific kind of technologies or architectures or, or maybe even companies that you're watching and you're like, okay, that is going to be a really big thing in one year or in two years or anything like that that's on your radar?

• 47:45Speaker 01

  It's, it's a complicated question because again, there's, there's, um, I, I think there's going to be a significant shift in the kinds of hardware that we use, right? What is happening now with GPUs, I think, is

• 48:03Speaker 01

  Not sustainable and it might still go up before it goes down, but I don't think this is uh what AI will look like going forward, right? So that's that's on on hardware, we're gonna have to see some something change there.

• 48:22Jacob Effron?

  And do you think that means more CPU workloads? Or people adapting models to run on GPUs or or maybe it's something else?

• 48:31Speaker 01

  It's um it's gonna be GPUs and GPUs and maybe specialized, you know, spec servers that are really optimized for either training or serving or maybe distributed infrastructure for that. You know, I'm I'm sure vector databases are gonna pay up, play a part in that, but it's it's not, it's not even, I'm not even suggesting that right now, it's really just, there has to be a change, it's like micro economically, it just doesn't make sense what's happening now. In terms of data pipelines and and kind of data management and and so on, again, something's gonna have to change, it's it's the tools that we had five, 10 years ago, just don't cut it anymore.

• 49:15Speaker 01

  Uh and both the operational headache and the actual costs and the time to activate and so on are just not reasonable. Something's gonna have to change there.

• 49:28Speaker 01

  And again, finally, I I think we're gonna have to start seeing those moderating systems where they, even if you don't own the model or whatever, like you have to somehow be able to have some kind of governance and and visibility and ability to control what's happening with your stack that today runs open loop for for most companies. So we're gonna see, I think, meaningful changes in the stack and in the in the software we use for it.

• 50:01Jacob Effron?

  I'm too intrigued by what you said earlier about things you're working on that, you know, will will blow our minds in two, three years. All right, I don't know if we're allowed to ask, but I'll ask anyway, any hints you can give us on some of the research directions or or where things are going in in your world?

• 50:14Speaker 01

  I wish.

• 50:16Jacob Effron?

  I had to shamelessly, you know, give it a go. We'll have to just reschedule you for 18 months from now. So you can announce it with us.

• 50:23Speaker 01

  I promise, I promise to share.

• 50:26Speaker 02

  We can be the VCs that then are saying, wow, you actually said, you actually did exactly what we were gonna do.

• 50:37Jacob Effron?

  Exactly. Well, you know, we always like to end our conversations with a quick fire round where we get your thoughts on a set of common questions we ask folks. And so maybe to kick it off, would love your thoughts on one thing that's overhyped and one thing that's underhyped in AI today.

• 50:51Speaker 01

  One thing that's overhyped, one thing that's underhyped. Overhyped, I think it's gonna sound nuts, but I think foundational models are overhyped. We know exactly what they can and cannot do, and I don't think we're actually making a lot of progress there, to be honest, for quite a long time. Not any qualitative progress, I mean, not any significant progress. You know, I don't know if it's underhyped, but I find code generation and coding assistance to be exceedingly useful. It's one of probably the most exciting use cases of this technology. So, underhyped.

• 51:30Speaker 02

  What's been the biggest surprise in building Pinecone? Maybe something that you thought would work but didn't, or something that you didn't think was gonna work that ended up working really well?

• 51:39Speaker 01

  We had a, at some point, pretty early, I mean, not very early on, maybe a year and a half into the company,

• 51:48Speaker 01

  maybe two, I might be off, but what about a year and a half or two into the life of the company? We had a complete rewrite of the entire database in Rust. Everything from like, we had C and Python, Go and Control, and this, that, like a million different things. We just rewrote everything in Rust because we figured, like, the kind of operational issues that we're running into, we're not gonna be, we're not gonna be able to scale with them. I thought that was like borderline suicide, and it would take us six months to even like, get to the point where we can even, you know, be on par with our existing system.

• 52:27Speaker 01

  And at that point, we'll be far behind. And our CTO, Ram Sriharja, was, you know, I'm gonna, we're gonna, we're gonna complete this turnaround in like a month. We're like, no way.

• 52:41Speaker 01

  And I was shocked. It didn't take a month. It took, it took, you know, two or three months. But, you know, the result was so much better than, than what we had before that. Again, like rewrites, I don't know if like anybody here, whatever, rewrites are always like, uh, like, whatever, this is like, companies literally die because of rewrites. This was like a huge bit. Uh, and oftentimes they just give you fresh and exciting new problems, oftentimes worse than the ones they started with. This was one of those like as advertised rewrites. It actually did what we were supposed to do. And I was just like, holy shit, it actually worked.

• 53:21Speaker 02

  That's awesome. Um, I guess besides Pinecone, what other company do you think would be most interesting to build AI features at right now?

• 53:29Speaker 01

  I would probably go build like, uh, whatever, join either a research lab or high performance optimization like company that really does. Sort of like, uh, compiling for, for these models. I think the, the fact that the execution layer and the training, the software for training and execution is one and the same is sort of like insane. Um, so I think there's, there's a lot of fun engineering and science problems to be solved there.

• 54:02Speaker 02

  When do you think we'll have agents that really work?

• 54:05Speaker 01

  Don't they work already?

• 54:06Speaker 02

  I mean, to the point where I can say, hey, AI agent, book me a trip or, uh, something like that. It, I don't know. It seems like they sometimes work, but sometimes not so much.

• 54:17Speaker 01

  Well, I have human assistants that they, they don't work all the time either.

• 54:21Speaker 02

  Yeah.

• 54:22Speaker 01

  I'm not, I'm not sure where you set the bar on, on how often they need to work. As much as like we expect from humans to work, they're already doing a decent job. I mean, is it kind of? I will say that when they make a mistake, it's more embarrassing. Like it's more like it's silly because it's not a mistake a human would make. Uh, but I think just like probability of completing the task, I think we're getting close to human levels.

• 54:52Jacob Effron?

  Awesome. Well, I feel like there's a ton of threads in this conversation that folks will want to turn on. It's been a fascinating discussion. I'll leave the last word to you. Like where can folks go to learn more about you, about Pinecone? Feel free to point folks in any direction you want.

• 55:05Speaker 01

  Um, look, our website is always a good place to start. Uh, there's plenty of material on how to build stuff, examples, docs, notebooks, like, uh, you know, reference architectures, integrations with partners and so on. Yeah, I mean, it's always a good place to start. I, I will, you know, as a meta recommendation, I would say, don't try to learn about Pinecone. Try to go build something exciting. And if that leads you to needing Pinecone, great. And if it doesn't, great, you still have, you know, built something else that was exciting nonetheless. Right? But I think like going back to the kind of somewhere in the middle of our conversation,

• 55:48Speaker 01

  the most common mode of failure is doing nothing. If you can overcome that, you've already like outdone most of the field.

• 55:57Jacob Effron?

  Right. Then you just have to make sure you don't do the cost estimating correctly, right?

• 56:00Speaker 01

  Yes, correct. And then if you think you're going to pay Pinecone too much, uh, drop us a line and we'll come talk to you.

• 56:09Jacob Effron?

  Amazing. Well, you know, this has been a ton of fun. Seriously. Thank you so much for coming on.

• 56:12Speaker 01

  My pleasure, guys.

• 56:14Jacob Effron?

  Well, Pat, that was a, that was a fun one.

• 56:16Speaker 02

  Yeah, super fun. He's, he's a really interesting guest, very thoughtful and, you know, he kind of has this lens at the infrastructure level, but then they see so much by virtue of everyone building on top of Pinecone. So it was, uh, it was cool to hear his perspective.

• 56:32Jacob Effron?

  Yeah. I mean, what an interesting seat. I feel like he really does see everything, right? He sees like the developers messing around with new applications. He sees the really sophisticated enterprises and what's required for them to get RAG live. Uh, I feel like it's a pretty cool, uh, cool position to have.

• 56:46Speaker 02

  Yeah, and he's even more bullish on the applications and infrastructure, which that was funny.

• 56:50Jacob Effron?

  Yeah, I thought that was that was that was interesting. And I loved the the subtle jibe at VCs who, you know, poo-pooed vector databases and then came back in 2023 saying they had years of conviction. Nothing, nothing we would ever do, obviously.

• 57:04Speaker 02

  Yeah, of course, not us, but you know, the other VCs.

• 57:07Jacob Effron?

  You know, one thing that's just so interesting about this space as a whole is I feel like we've seen folks, you know, pitching, oh, we have this sophisticated embeddings model that works really well in this context or this retrieval approach. And, you know, it's very clear that we're still in such the early innings where it's just like, folks are just trying to find ways to make things work, the basics work. And then, you know, I mean, to Edo's point, like, you really, you just got to start building, right? And I think if you can make the basics work, I do wonder when we'll move to the chapter of maybe more sophisticated trade-offs and decisions around some of these, you know, embeddings models and approaches to retrieval. But it certainly seems like, you know, it was interesting to hear him reiterate or reaffirm that, that, yeah, the case today is basically just can you can you get something up and running? And he just, you know, it's clearly he's blown away by the stuff that that folks have been able to get up and running.

• 57:52Speaker 02

  He's obviously biased, given where he sits as the CEO of a vector DB company, but he definitely believes that more of the intelligence will move towards the retrieval and, you know, towards that part of the stack versus purely at the kind of generation and or model stage.

• 58:11Jacob Effron?

  I loved his like cynicism of like, what a surprise. The people that charge per token have increased the context window.

• 58:19Jacob Effron?

  You know, I think he's the first person we've ever had on that says foundation models were overhyped. You know, I don't know if he's not a scaling laws believer or whatnot, but it's always fun to, to, you know, to get a, I mean, that's, I think what's so fun about the space is there's just, you know, there's so many, every time we do overhyped, underhyped, there's like a different set of things. Whereas, you know, as you know, I run a healthcare podcast and that space is a little bit more stagnant and it's like every time the same things that are that are over and under hyped.

• 58:45Speaker 02

  Yeah, I liked his underhyped answer too of coding assistants. And there's obviously a ton of hype around them, but I do think that is an incredible application of AI and one we're going to see a lot more of. Shout out Poolside, I guess.

• 58:59Jacob Effron?

  Exactly. I always got to shout out the portfolio.

• 59:02Speaker 02

  I thought it was interesting when he was talking about people not defaulting to OpenAI. And that is kind of the default now, but he sees smaller models that are potentially cheaper doing just as well. I thought that was interesting because I think that's not really something that we see very much where it kind of, most people seem to start with OpenAI or maybe Mistral, whatever, and then go from there.

• 59:28Jacob Effron?

  Totally. I mean, I think he's laser focused on the economics of these use cases. And if I'm being honest, I think a lot of what we see today is still folks in like this exploration mode where it's like, well, what can you do with these models? And it's like, you might as well start with OpenAI because if you can't do it with that, you're not going to be able to do it with too much more. Maybe a smaller fine tune model. But I think that it'll be interesting to see. One thing I'm super curious to see is just will folks end up in this kind of optimization phase as they deploy this at scale, probably. But at the same time, right when you're ready to optimize, GPT-5 comes out or the next model comes out and it's like you're probably right back to just exploring again, like, what are the capabilities that you can do with these models? And so I do wonder, I'm sure we'll see plenty of cost optimization, but this exploration phase, I think is going to be continuous.

• 1:00:14Speaker 02

  Yeah, he seemed a little bit more bearish on scale than I think some other folks in the ecosystem where we hear a lot of bigger, bigger, more GPUs, bigger models, all of this. And his take was it's not sustainable and the pendulum may swing back towards GPUs and specialized hardware and potentially specialized models and smaller models. And I thought that was pretty interesting.

• 1:00:43Jacob Effron?

  Well, Pat, that was another fun one. Folks should definitely stay tuned and like, subscribe. We are always trying to get the word out. Share with a friend that you think might enjoy this and stay tuned next week for another great AI founder.