Ep 71: CEO of TurboPuffer Simon Eskildsen on Building Smarter Retrieval, AI App Must-Have Features & Current State of Vector DBs

In this episode, Simon Eskildsen, co-founder and CEO of TurboPuffer, lays out a compelling vision for how AI-native infrastructure needs to evolve in an era where every application wants to connect massive amounts of context to large language models. He breaks down why traditional databases and even large context windows fall short—especially at scale—and why object-storage-native search is the inevitable next step. Drawing on his experience from Shopify and Readwise, Simon introduces the SCRAP framework to explain the limits of context stuffing and makes a clear case for why cost, recall, performance, and access control drive the need for smarter retrieval systems. From practical lessons in building highly reliable infra to hard technical problems in vector indexing, this conversation distills the future of AI infra into first principles—with clarity and depth.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  TurboPuffer is an incredibly fast-growing vector database and search engine powering some of the leading AI applications out there like Cursor, Notion, Linear, and more. Today on Unsupervised Learning, we sat down with their CEO, Simon, who had a fascinating career prior to TurboPuffer, spending a decade at Shopify, working on their hardest infrastructure challenges. We touched on a lot of things in today's conversation, including how builders should think about using different databases. We talked about the evolution of the vector database space over the last few years, and why now is a particularly exciting moment. We hit on Simon's takes on the general AI infra landscape, and what areas will persist. And we also talked about what he's learned from building so close to the forefront of AI apps today. This is an episode I think folks are really going to enjoy. Without further ado, here's Simon. Well, Simon, thanks so much for coming on the podcast. Really excited to have you.

• 0:41Speaker 01

  Yeah, thanks for having me.

• 0:42Jacob Effron?

  And get to do it in Ottawa. It's a treat to take the show on the road.

• 0:45Speaker 01

  Yeah, thanks for coming.

• 0:47Jacob Effron?

  I was thinking about a few ways we could start. One is, it feels like in building AI applications, everyone is incredibly focused on connecting their, you know, business-specific context to their applications. And I feel like throughout the last few years, there's been a bunch of different ways folks have started to think about doing this. One has been, let's just stuff everything we can into a context window as they get bigger and bigger. Another has been using kind of NetNew databases. Talk about like just the origin of what made you convinced there was a need for like, a new search paradigm here?

• 1:17Speaker 01

  Yeah, so I think when, when I started working on TurboPuffer, the context windows were very small. And so that really got the initial wind in the sails of the first few vector indexes, right? The context windows were maybe 8K, very small. And so a lot of the applications that started needed rag, basically, right away. Like, most, even most articles are larger than 8K, right, tokens. So that really got early wind in the sails. But then the context windows got very large, very quickly, right? They came up with these windowed mechanisms and stuff like that to try to make the context more useful. It's still quadratic. There's still a lot of tricks. It's not perfect recall. At the end of the day, there's very few data sets that actually do any kind

• 2:01Speaker 01

  of reasonable question answering on, you know, something the size of a Harry Potter book. So it's actually very difficult to train a good model to do this, but. Either way, that sort of got the initial wind of the sails. But then it felt like the wind kind of got out of the sails a little bit because the context windows got really large. And the first few applications that were coming online were able to just abuse and stuff the context windows and run for PMF. And that's exactly what I would have done too.

• 2:29Speaker 01

  But what we're seeing now is that even with a million context or 10 million context, let alone the latency, companies want to connect all of their data to L L Ms, right? That is the new workload that we see today. And in order to do that, you often have more than a million tokens. You have tens of millions of tokens. You have tokens that have permissions attached to them. You want very high recall, which even in a very large context window, you can't necessarily guarantee that it can be there again back to. It's very difficult to train a model on this kind of thing, right? So I think now what we see is that companies want to search large amounts of data as they have found PMF with the large context windows. But they demand economics that made sense for them.

• 3:13Speaker 01

  What counts for them is that the per user economics of storing all of this data have to make sense for the value that they're delivering on the other end. And on a traditional storage architecture where you're storing every byte on three disks,

• 3:28Speaker 01

  the the the per gigabyte costs end up like north of a dollar, right? Per logical gigabyte. And we can do it at a much, much lower price with obx storage and with a set of trade offs that make really good sense for search that allows searching on, you know, not a million tokens, but hundreds of million or billions of tokens for a particular user or application. I have a little acronym that I throw around when people, when we talk about context. We love acronyms on the podcast. I call it scrap. And it's basically like the the couple of different. And I just use this when I, you know, otherwise I forget something. The first S stands for scale, right? So at a particular. Point, you will outgrow the side of the window of the window.

• 4:10Speaker 01

  Even if the context window gets to 10 million or 100 million, right? People are just going to demand more. That's what tends to happen with computers, right? But there is some sense where, okay, if I'm doing an analytical query, should I load a billion rows into a quadratic attention window? Or should we have some auxiliary system, right? It feels like even an AGI level model would build a system to do some of this stuff, right? On what's already there. So that's the first one. The second one is C, so that's cost. So the cost of executing a very large context window is substantial. You have to store that in VRAM, and VRAM is one of the hottest commodities on earth right now. It makes a lot of sense if you're doing a lot of queries against something, right?

• 4:53Speaker 01

  Some places will make sense to just have everything in a context windows and just execute against it. The economics of VRAM make that work. But in a lot of cases, you can't afford to keep all of this in VRAM. And you can't even really afford to keep it in DRAM, let alone even on disk, right? Which is why this architecture of storing things on obic storage can make sense. The next one is recall. So recall in a very large context window can be difficult. Yeah. There's a lot of benchmarks that are sort of needle in the haystack. And they do fine on that because it's easy to train. But actually doing a do having data sets that are good at reasoning over very large corpuses and distilling an answer. There's not a lot of them, at least that I know of. Maybe maybe they're behind paywalls I don't have access to. But there's not a lot of data sets for that.

• 5:36Speaker 01

  And you have to play a lot of tricks to make this work. The next one are ACLs, right? I don't think anyone trusts the context window enough that you can stuff in and say, well, you know, Jordan and Jacob have access to this document, but Simon doesn't, right? Can you just like, you know, pinky promise that if Simon is the one asking that you won't do it? There'll probably be a point where we trust them enough. But at present, we don't, right? We sort of like put some UIDs on the document and some UIDs in the query and make sure there's at least, you know, one overlap in the set intersect to make sure that the user has access to it. And then the last one is performance, right? It's difficult on these frontier models to load in a very large context window and get a response in less than a second, which is what an engaged consumer would want.

• 6:19Speaker 01

  So those are some of the reasons that we see among our customers of why they engage with. Allowing their LLM to access the data in a different way than just putting it all in there. But of course, it's some, there's some optimal function between how much goes into context and how much is retrieved into it that's idiosyncratic to the company.

• 6:38Jacob Effron?

  It feels like so much of this is connecting massive amounts of data. And it feels like in the early days of people looking at some of these new databases, you know, they were really making up for the fact that the context windows were small. And I remember when we were looking at some of these businesses two years ago, there was just a desire to find like, who are the customers that are connecting massive amounts of data? And maybe two years ago, there weren't that many, honestly, that were. And now it feels like that is actually starting to change. And, you know, you're seeing a bunch of different use cases on top of your platform of folks that are bringing tons and tons of data to actually connect into the models where you need something like this.

• 7:07Speaker 01

  I think that's exactly right. In the beginning, the first generation of these databases and most of the research was really around doing vector search or on in-memory. And when you have an 8K context window and you're trying to make up for that with maybe, you know, accessing 32K tokens, the economics are not so bad, you're not going to go for it. Of course, we've done inverted indexes, which are what you use for keyword search on disk for a very long time, but none of this has been tried on something like obic storage. But I think that now, yeah, we're seeing that the frontier really is to try to reason over very large data sets. And I think I only developed more conviction as I saw the frontier models start to use so much time querying data, right?

• 7:50Speaker 01

  They do use a lot of time searching the public web and increasingly, the models also have access to private data. And I think every company is going to, there's going to be a baseline expectation that you can fire off some research function, which I think are the agents that people use today that are the most useful, right? Hey, go out and compile a report on this. And every SaaS company is going to be expected to have a feature like that over all of their data and potentially even data that is tangentially related to the data that is all that's in that first-party SaaS application.

• 8:23Speaker 02

  I'm curious, though, on the object-oriented architecture side, we're big fans of it, by the way. We just released Infrared Report and highlighted TurboPuffer as an example. What are some of the trade-offs of actually using that architecture and what are AI workloads that maybe it wouldn't make as much sense for?

• 8:37Speaker 01

  To back up a second here, I think it's important to take the history lesson of why this is now possible, because I think there's a lot of- Love why now as VCs. That's a classic. So I'll entertain that for a second because I think that there's a new storage architecture here, right? Richie and team did it for Warpstream, right? Where we started using this for streaming. The observability providers have done this for a long time. And of course, the OLAP companies that came out in the 2010s have also done this for a while. So what's changed in the past five years that makes something like this possible? So I think the first thing are NVMe SSDs being so good. I think they got good very

• 9:19Speaker 01

  quickly. And I don't think that any new database engine has fully been built to take advantage of how much bandwidth you can drive from these disks, right? If you go to a cloud and you look at the cost of these NVMe SSDs based on the instance, you can get them for somewhere around 5 to 8 cents per gigabyte, okay? But you can drive you can drive often to these disks within spitting distance of DRAM bandwidth, right? So you might be able to drive like 10 gigabytes per second to these disks. DRAM on that machine might be 100 gigabytes per second, but they cost two orders of magnitude less, right? A gigabyte of RAM in the cloud costs somewhere around $5, right? It's like two

• 10:01Speaker 01

  orders of magnitude less in cost, but the bandwidth difference is only 10x. This is a very interesting trade-off, right? And that wasn't really available on the network storage that's always been available in the cloud. So this performance is a very interesting characteristic. And you have to write your storage engine to really take advantage of this, right? IOU ring is only something that's come out recently. You can't even use the Linux page cache because it's too slow to keep up with these drives. So these only became GA and AWS and GCP and others in the late 2010s, like I think around mid 2017 on AWS and maybe mid 2016 on GCP, right? There was like one SKU that had these disks.

• 10:42Speaker 01

  So that's the first thing. The second thing is that S3 only became consistent in December of 2020. This is kind of a, this was a mind blowing fact to me when I learned about it. But what this means is that if you put an object and then you read it immediately afterwards in another request, that you know that what you just wrote is what you're reading back. That only became a like core primitive guarantee of S3 in December of 2020. That's a very nice thing to have if you're building a database on Obik storage. You can live without it, right? Snowflake and others, right, have built big metadata layers so that you can work around this. But it's really nice to have if you want to write a database fast. The third thing that you need is that you need compare and swap.

• 11:25Speaker 01

  And what this basically means is that when you're building a database on top of a file system, right, you, you, you have a bunch of metadata of, hey, the most recent version of this is here. And so you need this metadata and you often have multiple writers that are writing to that metadata. In our case, it would be multiple nodes that might be contending on, on a piece of metadata. So you need some synchronization primitive. So past database systems that are distributed systems will have a zookeeper or RAF layer or something like that to contend for this control for multiple writers. But S3 actually only about six months ago, we're recording this in the summer of 2025, released compare and swap at Reinvent. Now, every other Obik storage actually had it prior to that, namely GCP, which we started

• 12:10Speaker 01

  on for exactly that reason, because we had conviction that this was going to become a utility function. Any database that wasn't built this way would feel archaic in five years, but that was the last piece of the puzzle. So NVMe 2017, S3 becoming consistent in 2020, and then finally compare and swap in 2024, allowed you to now build a database where everything is on Obik storage. And so we, I, I felt that you could build a database that had these Obik storage trade off that worked very well for search. Doesn't work for everything. So back to your original question, I'll be kind of building up to the answer here of What are the trade offs of building a database on Obik Storage? Well, first, it had to be possible. It's now possible because of the three, these three primitives.

• 12:56Speaker 01

  And then we have to consider the trade offs. And so the trade off is that every time you write, we have to commit to S3. The P90 for such a write, depending on the size, is maybe around 100 to 200 milliseconds. So every time you write, it takes 100 to 200 milliseconds. Now, if I'm writing a checkout system at Shopify scale, then that's too long, right? We can't wait for every transaction to commit to that long. But if you're building a search engine, it's usually fine. You update a product and it takes 100 milliseconds or 200 milliseconds for it to be updated in the search engine, well, that's an acceptable trade off in a lot of cases. Fundamentally, this is really the main trade off. Like, everything else, I think, in terms of cost, in terms of flexibility, in terms of

• 13:39Speaker 01

  scale, in terms of simplicity on the system, are some of the upside. You could say that another downside would be that occasionally you will have a cache miss and you have to read directly of Obik Storage. But there's nothing that prevents you from having it always on and hot, as you would in a traditional storage architecture, even on multiple nodes, if a node was to go away. The high write latency is really the fundamental trade off

• 14:02Jacob Effron?

  Yeah.

• 14:03Speaker 01

  of this kind of architecture.

• 14:04Jacob Effron?

  And as you think about the implications of that then for AI app builders and when they should reach for TurboPuffer and when they shouldn't, like, how do you think about the types of things people are trying to do in applications where this trade off does and doesn't make sense?

• 14:14Speaker 01

  Right, so, I mean, TurboPuffer is mainly a search engine, first of all, right? So if you're modeling all of your data and user permissions and all of that, you should probably use a relational database. Like, nothing's going to make that go away and the transactional performance of that, the flexibility and all of the know how, right? So I'm talking here from a storage engine perspective. Where something like, where something with this architecture makes sense for your search engine, right, is that if you have a million vectors and you tag on a vector search extension into your relational database, great.

• 14:49Speaker 01

  That's what I would do. But if you have tens of million, hundreds of million, you have maybe billions or hundreds of billions in the crosshairs. Well, at some point, the somewhat idiosyncratic trade off of adopting another database and ETLing into it is going to start making sense, right? Because at some point, you can't escape the economics of, well, if you use a Postgres extension, you have to replicate it to three disks. And these disks cost you, you know, 10 cents per gigabyte, 50% utilization, all in about 60 cents per gigabyte that you store. You can't really escape the economics of that. And the vectors are large, right? A kilobyte of text easily turns into 30 kilobytes of vector data. So where something like TurboPuffer makes sense is when you're searching over large

• 15:34Speaker 01

  quantities of data, right? And I mean, since the beginning of time, people have taken out the full text search workloads from the relational database. Once it reaches a certain scale, vector search is even more brutal on these transactional engines in terms of both the cost and how much it can overload the box and these other reasons why we've taken FTS out for a long time. So that will be another bottleneck, right? As an application gains success, you have to start taking out some of these pieces from the relational database before you have to shard it.

• 16:03Jacob Effron?

  What are some of the most common set of use cases you are seeing people build on top of TurboPuffer? I mean, is it some of these like deep research type use cases on top of tons of data or like as varied as the landscape is out there?

• 16:13Speaker 01

  I think, I mean, we could go through some of our customers and how they use it. So Cursor was actually the first paying customer on TurboPuffer.

• 16:23Jacob Effron?

  Pretty good first customer.

• 16:24Speaker 01

  Yeah, seriously. And they've been wonderful to work with and true, true design partners in every sense of the word. And their use case was to, they want their agents and all of their features to be able to do semantic search over a codebase, right? Or over multiple codebases. And that's what they use TurboPuffer for. So if you've opened a codebase in Cursor, then that codebase is indexed into TurboPuffer, right? Like into vectors that are completely obfuscated and encrypted. And then They can use RAG over all of that to try to draw in more context. I use it all the time because you can ask in the chat. You can ask it something like, hey, what's the function that does this thing?

• 17:09Speaker 01

  Like this morning I was asking, what's the function we have that formats a number? So it's not a million, like, you know, five zeros after or whatever. But it's like, it does some. And you just ask that in free text because you don't remember what it's called. That's the kind of thing TurboPuffer is really good at, right? And you will see the cursor agents make these kinds of queries. So that's one, right? So they're connecting code bases to AI. And sometimes very, very large code bases or multiple code bases. Notion is another one of our customers. So they have a Q and A feature. And increasingly, this is making it into more and more of their canvases. Where you can ask it, hey, what's the leave policy? Or hey, someone passed in my family, like, what are my options? If it's used for an internal wiki, it can do research type things.

• 17:53Speaker 01

  So they use it for that. And often, the way that a person thinks about something and the way that it's written in a document are different, right? Like you search for red dress and they have a burgundy skirt. Like, that's the kind of thing vectors are really good at. And so Notion uses that to draw content into the LLM to answer questions. Linear is another customer of TurboPuffer. They use it for their search and they also use it for similarities. So, oh, this might be a duplicate issue of this one. Or this might be the person who should work on this type of issue. I think they might also be using vector embeddings for that. So those would be some of the use cases that we've seen.

• 18:29Speaker 02

  Curious, you know, we talked about why traditional databases aren't a good fit for sort of the scrap model that you talked about. What about incumbents like Elastic? Sort of going after and combining vector search with traditional search like you are?

• 18:41Speaker 01

  If your per user economics allows you to store everything in memory, then, you know, fine. That would make sense, or on disks. I think traditional storage architectures, you know, they have lower write latency. And they may have maybe more features, right? Because they've been added for longer. I think it makes sense for certain scale. But if you have the ambition to search potentially trillions of documents or billions of documents. Then the cost might not make sense for your application, right? And you might have to start upping the price charges on on your users in order to have this available. And you know your market better than us, right? And we just try to price against the first principle cost to us.

• 19:24Speaker 02

  You know, I think you're one of the world's experts now in building object oriented architectures. Any tips on just building that out for folks that are looking to?

• 19:32Speaker 01

  One of the things that as an engineer continues to surprise me is how far simplicity goes. Now, we have very high conviction that keeping everything on obx stores, including the metadata, was the right decision. But really, our hand was forced a little bit early on because we had some customers that were growing very fast. And we didn't have time to introduce a separate metadata layer. And we thought that maybe people would care a lot about very low write latency. But again, we gained high conviction that this is actually fine. And the trade offs and the simplicity that we get out of committing directly to S three were there. So I think that let obx store it surprise you. I think that there's a big back of tricks that you start leveraging to build really scalable systems on top.

• 20:17Speaker 01

  The other day, for example, of course, we have, we have many millions of of prefixes on S three. And we sometimes have to go out and do various background activities on them. So then you have to go list them, right? The S three API is like you start from A and then you list and then you go and pageinate through. And it would just take forever because for S three, I think it's sort of a brutal operation for them. So it takes a really long time. And so what we started doing was just have a read through cache where every every few pages we would just put in a TXT file. What that what that prefix was. And then every time now we can sort of like start listing the buckets at all of these different keys in parallel. So there's lots of these tricks that you can use.

• 20:58Speaker 01

  But you like, I, I don't think I can explain to you how many hours I've sat and just looked at the S three API. To try to try to come up with something, right? Like another thing I'll say is like four fours or three are free. And you can use that to design systems, right? There's the compare and swap primitive. There is, there's all of these little headers that you might not have paid attention to that allows you to, to build a system like TurboPuffer with very, very low latency and very high durability. So I think we have a big bag of tricks. At some point, we should, we should lay out the, you know, 16 bag, 16 tricks in a bag. There's nothing secret there, other than just spending a lot of time looking at the S3 APIs and the GCS APIs.

• 21:43Speaker 02

  And then we talked about sort of use cases that are, are a good fit for TurboPuffer, are not as much of a good fit. What do you think are sort of unsolved problems today within vector search?

• 21:51Speaker 01

  So the hardest things about vector search is to keep the index up to date. So when you, when you do vector search, the only way that you can guarantee that the, when you do this query, that the 10 results you're getting back are exactly the 10 results that are closest to this vector, is actually by looking at every single vector, right? It's an unsolved problem to have a non like O of N algorithm that can do that. So we approximate and we approximate with a number that the, the listeners may have heard of, referred to as recall. Recall is basically, okay, these are the actual results I got back from my vector index. And here are the exact results. What's the percentage overlap?

• 22:34Speaker 01

  So if there's one result that was wrong compared to the exhaustive search, then you have 90% recall, otherwise you have 100% recall. We find that our customers feel really good at around 95% recall. That's sort of when you don't have to guess whether your evals are not performing because of your retrieval layer. And so in the background even for a percentage of queries, we run exhaustive searches against them and report them back to DataDoc. So we always have an idea of what's actually going on in production. When you incrementally maintain this index, you have to make sure that the recall is good. And if you If you built the index, you know, knowing that, okay, I have an e-commerce store and they're selling pants and they're selling dresses and they're selling shorts and suddenly they

• 23:18Speaker 01

  start selling shoes, well, which, like, how do we put this out in the vector index? Because the vector index is sort of, has these clusters around different things. And so maintaining high recall as you continuously update the data is very challenging, very, very challenging. In the beginning, in the first version of TurboPuffer, once a certain percentage of the dataset had been overwritten, we would rebuild the entire index. That's what most production implementations do today. But we found this very, very challenging and expensive to scale. So we spent a lot of time trying to, and implementing an algorithm that will incrementally maintain all of the, like, this A and N index. And this now works into the hundreds of millions of vectors and even into the billions.

• 24:01Speaker 01

  And we're trying to push as far as we can incrementally maintaining with high recall that A and N index, because sharding is the only way to scale anything. But sharding too early is coping, right? And in, on some of the traditional search engines, the shard sizes are maybe around 50 gigabytes. And, you know, looking up an operation is log N or whatever it might be on, but if you have to do log N for 1,000 shards, it's a lot more expensive than doing log N on five shards. So you want to make these shards as large as possible. And so incrementally maintaining high recall on a vector index at high ingestion performance

• 24:43Speaker 01

  into hundreds of million or billions for a single shard is a very challenging problem. The second challenging problem around vector search are is filtering. So when you, when you send a vector query, you are not just saying, hey, what's close to red dress? Oh, it's the burgundy skirt. You're often like, does it ship to Canada? Is it red? Like, you know, you have all these, like, real filters, right? I love all these Shopify examples, but. Yeah. You know, I, I.

• 25:11Speaker 02

  It's DNA at this point.

• 25:12Speaker 01

  I grew up here, right, here in Ottawa, where, where the company was founded and the infrastructure was built. And it's, it's, it's, it shaped your worldview. But anyway, so you do these, like, actual hard filters on top of this more fuzzy vector search. And that could be challenging because that's a different type of index that you need to use for that. And so you can, you use all kinds of techniques to make sure that the recall is still high in the face of these filters. Because of course, you know, if you search for something that's the color red, that might match a small percentage of the data set and you can just search all of that. So that's very easy to call the pre-filter. If something is, if you're checking for whether a product is public, well, probably

• 25:53Speaker 01

  99% of them are public, so you just overfetch a little bit and then you remove the others. But something like ships to Canada, which maybe applies to 50%, randomized around the clusters, and you're trying to get a banana that ships to Canada, there might not be any produce that will ship because that will be prohibitive. So the clusters that are closest to the banana are just completely come out. So what it ends up being is like maybe one of the clothing clusters and it's like, you know, fruit themed, I don't know, you know, it's just. And so providing high recall in the face of this really requires that the query planner that decides how to execute the query is aware of the filtering and is aware of the vectors. These are the two hardest problems and I mean like capital H hard.

• 26:35Jacob Effron?

  I mean, one thing I'm struck by, and obviously there's just so much nuance in solving this problem, is you think about this like, zooming out this general problem of connecting data to LLMs, you know, there's a bunch of parts of that process that all have their own problems, right? There's like getting the data, you know, to, you know, the database in a way that works. There's obviously the embedding models themselves and how people do that. And I'm sure some people building in this space have said, okay, like, you know, customers like to have a one-stop shop or we should, you know, build more of these things. It feels like you've been super focused on really just nailing, you know, this part of that entire process. Like, talk a little bit about how you think about that and you think about like the future of TurboPuffer across those vectors.

• 27:09Speaker 01

  We've talked about simplicity a bunch. It's a very core cultural value, I think, both of the company and of Justin and I. And it's any time Were in doubt, we go the simple path. Simplicity and focus go hand in hand. And we felt that the hardest problem that our customer was having was not choosing the embedding model, it was not running the embedding model, it was not running the reranker. It was, hey, I need to store petabytes of data and I need to search it. And so that's what we're focused on. Now, over time, what we're starting to hear from our customers is, hey, for me to ingest a hundred million vectors very quickly into TurboPuffer, it would be really great if

• 27:53Speaker 01

  you could help us with that, right? So we wouldn't, by no means not saying we would never do it, but we are very focused on just this, because as soon as you have to start running the embedding model, then it's like, okay, for low latency, you might want to run it on GPUs. Now you have to sort of get into that game and sort of the quant GPU game, right? That reigns today's world. So we want to make sure that the things that we put our name behind, we're doing a really, really good job. I take the responsibility, and Justin also takes the responsibility very seriously, of people trusting us with their data. And we know that if we fuck up, we're not the only ones that get woken up. Our customers also get woken up. And every single decision that we've made around how we've designed this has been that

• 28:39Speaker 01

  we don't want to get woken up. I was on the last resort pager of Shopify for almost a decade, right? I have sat there at 3:00 AM debugging databases completely alone in the dark so many times. We know what that feels like, and we don't want our customers to be in that position. So reliability is number one. And if you put reliability as number one, then you have to put simplicity right up there as number two. And then when you start to bring in all of these other things, you better have gotten that right on the core product. So we take that very, very seriously, and it's part of, it informs everything around how we do it, right? The only stateful dependency that TurboPuffer has is Obik storage, right? You can blow away all the nodes and all of the data is safe, right?

• 29:24Speaker 01

  You can blow away everything and things are fine. And routinely that happens in production, that things are blown away and... And auto scale down, whatever, and, and, and everything is fine, right? And no one knows, notices. So I think, I think that, that, that simplicity is really, really important to how we build software.

• 29:42Jacob Effron?

  Obviously, there's, you know, the ingestion embeddings. Is there other stuff that you think about, like, existing in this, you know, set of problems that eventually, or customers are coming to you and asking about, or a set of problems you eventually might want to add in?

• 29:53Speaker 01

  Yeah, I mean, we like to just see what our customers actually use out there, right? It's like hard enough to find PMF on one product. So let's, let's make sure that we double down on what's working. I think we see our customers asking for us advice on, um, how should I do eval on search results? Our customers ask us, uh, which embedding models should we use? And, um, we, we, we have internal reports, right? On, on which ones are fast and what we hear. And so we can pattern match, um, in a respectful way across what we see our customers doing. There are some embedding models from some of the, some of the, some of the labs that take 300 milliseconds to do a query. As that's prohibitive for some search, that's too long, right?

• 30:36Speaker 01

  Turbo power takes 10 milliseconds, it takes 300 milliseconds to create the vector. It's not acceptable. So we want people to use fast embedding models so that they don't get painted into a corner. Rerankers, the same thing, right? I mean, I, I worked on search at Shopify and we see what others do, um, at search here. And so we just help our customers. But in general, what, what I have seen and saw, um, at Shopify as well is that in the traditional search engines, you end up with a massive DSL where you're expressing, hey, multiply the title BM25 results with this and that, and this field, and then a little bit of the image. And it sort of becomes this like very finely massaged thing. And generally it's written, someone sort of loaded that context into their head,

• 31:21Speaker 01

  and then executed and got good evals, and then no one touches it for years. Because, you know, now, now, now, like, you're, you're addicted to what happens when you type in a set of characters and navigate. Have you ever tried when the search engine changes on you? And you just like understand how, how bodied you are to this? But it's very difficult to maintain. It's like thousands of lines of JSON or whatever. And so I think right now what we're seeing is that vectors make up for a lot of that. Again, the, you know, coming back to this red dress burgundy skirt example, well. We used a lot of effort and PhDs before to turn strings into things, but that's just, you know, you cut the head off of the LLM, right? And these numbers are exactly that. So we find that a lot of these features are not needed. And we find that our customers actually really like to just write a search.py or search.

• 32:06Speaker 01

  TS or whatever they're using and doing a bit more of this themselves. There might be, you know, one or two milliseconds of performance penalty, but fundamentally, there's really not much and you gain control. You can write tests, right? It's easier to write evals. So I think as we find that our customers want more of this and particular things where from first principles, it makes sense for that to be in the engine for performance reasons, then we will do that. Like something we're starting to see is that people want to do late interaction where you're searching with, searching for often 128 queries in one search query. And so that's maybe a little bit too much to funnel down over JSON. So we have to help with some APIs around that. So we're always paying very close attention to it, but we take the same stance as we always have.

• 32:51Speaker 01

  We don't, you know, European cities are beautiful because they're built incrementally and software is really the same thing. When you start to guess too much about what people might do with the software, you end up with these, you know, like we're in Ottawa's like, you know, it's not a particularly like beautiful downtown where it's just like, like a lot of things needed to happen very fast. And so you build a lot very quickly. And I think a lot of soft, bad software is written that way. You make too many assumptions about how people are using it, right? Look at like, we didn't think that people were going to love the 100 to 200 millisecond write latency, but it turned out to be fine.

• 33:24Jacob Effron?

  Does the approach to, you know, this kind of focus and gradual building, does that lead you, you know, more to then the cursors and notions of the world? Like one thing I've been struck by seeing infrastructure players trying to sell into like large enterprises is I feel like they'll come with, we do this part of, you know, your broader rag solution. And the enterprises are like, no, no, I want like one thing to do end to end, you know, for me today. And I feel like a lot of, you know, in talking to a lot of infrastructure companies, they get dragged into other parts of the stack because that's what folks want to buy.

• 33:50Speaker 01

  Yeah, I think, I think naturally, right? Everyone starts to bundle at some point and you start to, you hear, you see commonalities between your, your customers and, and you try to help them get there faster. But you can also get too greedy. And you can start to do too much of this too soon. At the cost of the focus of the team, right? And the magic had made it work in the first place. That you even have customers asking for you for this, which is a blessing. So we, we think about it and we, I, I expect us to like partner and bless and work on things to make, make the end to end process, process much easier for our customers. And yeah, we, it certainly takes a lot of discipline sometimes to say no.

• 34:37Speaker 01

  And then say yes to working on more performance, more cost reduction, whatever it might be. Right. On the core things that we want to get good at. There's a grab bag of ideas and it's, you know, it's rare that a new idea enters the grab bag that we haven't already like spent considerable time and effort thinking about. What is always difficult is to decide when to pull something out of the grab bag. And to continuously doing that Bayesian, you know, gradient descent on, okay, what matters this month. Right. And continuously changing the priors based on what the market is demanding and customers, right? And not saying no forever. And we do exactly that process internally as well.

• 35:13Jacob Effron?

  I think this has been one of the hardest parts of building AI infrastructure companies generally is that, you know, with the underlying model layer changing so fast, the way people are using these models evolve so fast. And so it feels like you're playing a bit of whack a mole of like, okay, we, we just solve for the way people are building things today. And then in three months, they're actually building things in a very different way. Now, I think, you know, arguably the space you're building in might be the most isolated from that. Cause it seems like at any capability of model, this will be required. But I'm curious, as you zoom out and think about other parts of the AI infrastructure stack, like, does that resonate? And how do you think about what other persistent parts of it will be versus, you know, things that seem more moment in time?

• 35:49Speaker 01

  I like companies that have state. And that's why we built a stateful company. I think that there, there would be a lot of commonalities, right? Of, of, of what will be in there. And I think that to build a good company as part of the AI stack, you either want to come up with a really good workflow, like something that makes, and lots of good companies have been built around workflow. Generally workflow companies eventually start capturing some state as well. And generally the stateful companies also start capturing workflow. I think that if you try to do all of that at once, you need a lot of years of R and D in a lab before you go to market. And I don't think anyone has time for that right now. So I think it's a very interesting time to be an infrastructure company because I think that.

• 36:32Speaker 01

  Among the frontier, they are picking and choosing and doing the Lego thing. And they don't, they, they, they're being very careful about what they're choosing. And they want the best in the world for every particular piece, right? And so then there will also be companies that try to bundle everything, but it will come at some quality trade off for individual pieces. This market is different, right? In that, like, I feel like there's always a, some demarcation in the generation for different companies, right? It was very clear that a new era of companies happened in late 2022, right? It was clear that during the Zerp era, there was a particular set of companies, right? Into 2010s, which is where I feel like I grew up in the soft, in, in software world,

• 37:14Speaker 01

  you had the Stripes and the Shopify's and the GitHubs, Zendesk and, and Pinterest and so on, that in some ways were also felt similar, but also, you know, wonderfully different. And now we're in yet another one. And I think that it has various attributes, but I think that we, we are seeing that some of the ones that are doing very well are very specialized on what they're doing, right? They're taking the, the, the niche that they're good at and they're, they're doubling down on it. At some point, we will see them bundle, but it might take a little longer than it did for the previous generation of companies, because all the great generational companies have a very strong finger on the pulse of what's happening and what people want right now. And it's very informed by the customers.

• 37:54Speaker 02

  One area that's getting a lot of interest now is memory on the AI emphasis side. What do you think the future of memory is? And then what is the role of turbo buffer, you think, in that space?

• 38:03Speaker 01

  Yeah, I think memory, people will always start by playing around with the simplest thing, right? And so if you, if you're using agents today that do memory, they, they're sort of, there's the memory within the context of a very long encounter, right? So if you're working with a coding agent, you might be working with it for a long time. And at some point, it sort of has to compact. And of course, right now, for that kind of thing, a lot of, you just use either a TXT file or, or you, you just ask the LLM to compact it and that's where it's going to start. Then we see them, them start to bring in memories, sort of laterally east, west to that chat itself. It's like, you know, my chat GPT is very Because I shared with my wife.

• 38:45Speaker 01

  And, and, and so there, but it's generated all these, all these, all these memories, right? Of how do you grow, how do you grow this flower? And what do you do about this pest, which is, you know, my wife? And then it will join, oh, since you're a good Rust programmer, here's like a script to get a like a weather and humidity report for your flower. Like it's like, so now we have to split the accounts. But anyway, so, but those memories are lateral and they're not, it's not a lot of data, right? And so even if you just pulled them all into memory and did some similarity or pulled them all into context and they're condensed enough, that's probably fine. Are we going to see, start seeing memory at a scale where you have to start doing a lot of rag over it? I think that we do see some of it.

• 39:26Speaker 01

  So one of our customers is this, is this company called Protola. And they built these TOLs and the TOLs have very long standing conversations with their users. And these are not just memories, right? There are long, long chats. So there's also this sliding window between, or the slider between like what's a memory versus just like searching and all prior context. I think similar to what you've heard me talk about before, we haven't seen enough patterns emerge here among our customers that, that we're gonna, that there's anything in particular that we have to ship, right? You could also just put all of this into TurboPuffer and it's just like a KV store on Abic Storage and it will work great. You don't need to use vector search or just, you could just use the keyword search as well.

• 40:08Speaker 01

  But I think, I think it's TBD exactly what this looks like. I don't think, I think, I think some implementations, it's not a lot of data and some it is a lot of data and it's more over the history. I think I would, I would assume that those that do it over the entire history will outperform those that, that just condense into memories. But the memories are sort of a higher, have a higher weight because they're condensed from a chat. I, I, I wouldn't pretend to know exactly where that's going to go. But we are seeing a lot of experiments in that area.

• 40:38Jacob Effron?

  How much time do you spend thinking about like? Or where the foundation models are going in terms of, you know, obviously, I feel like the future state of like the size of these models, the way they interact with memory, with databases, huge implications for your business, probably unknowable to some extent today.

• 40:56Speaker 01

  I do spend a lot of time thinking about it. I don't think I have a general answer to just like, this is why it's fine. I think it's easiest to think sometimes in extremes. Okay, we get AGI, well, none of this matters anyway. So, okay, right, so that's, that's fine. Great, you know, like, everything you, we have now will just compound and everything's great. Then there's the other scenario, right? Which is, I think, the one that we're rapidly entering now where the models are incredibly powerful. They're very good at generating reports over large amounts of data. And it's just very clear that even if they get very capable, it just feels like you got to yank a pipe into them and put something computationally good on the other side. And like, I think the architecture could look a lot like TurboPuffer because there's a lot of data, you don't need all of it at all times, and it needs some kind of like fuzzy search a lot of the times.

• 41:40Speaker 01

  I think that there's a, there's a real role for a database with this architecture. I think that to build a good database company, you need two things. You need a new workload, which is connecting lots, large amounts of data to LLMs. And you need a new storage architecture. And we talked all about that before, right? And why, why now for it? And so I think our hypothesis is that this will play a role in how these LLMs are going to interact with data, right? But we also see lots of people who just use TurboPuffer for traditional search, right?

• 42:08Jacob Effron?

  Before we start recording, you kind of said there would be this new set of things that would be table stakes for SaaS applications. And, you know, I imagine one thing that you think about a lot is just this flowering of tons of different applications that will need hundreds of millions of vectors and, you know, build applications on top of that. Like, how do you think about the, you know, addressable universe of what those companies might look like?

• 42:29Speaker 01

  When I think about what I want out of some of the applications I want today, I'm just like, ah, like, I just talked about, talked with someone about this over here. Like, can we just, you, you, essentially, a lot of what knowledge workers do has to do with funneling context around in different systems, right? So it'd be great if they could help with that. And I think we could help them help people with that. I think there's a bunch of features that are now going to be table stakes for SaaS in the same way that once mobile really hit, it became table stakes that everyone had a good mobile app and It felt like a huge tax at the time, right? I have to bring in another programming language. I have to do this. And there's all of these things that were happening around the time around like, do we build them natively? Do we build them as web? And now, you know, you just don't think about it.

• 43:14Speaker 01

  When you build a software company that you're serious about, you just kind of need a mobile app, right? For most of them, not all, but for most. And I think that that's also what we're seeing with AI, right? It's that there is a now a set of table stakes features that people expect there to be in your application in the same way that if they they expect to search for your application name in the App Store. Those features are things like, I think that semantic search works, right? If I search in my linear for chat, that issues around Slack also come up, right? You know, if I search in a commerce store for burgundy, you know, whatever, whatever, right? Where I search for shoe and they only have, they only have sneakers.

• 43:57Jacob Effron?

  This must have been the most important eval at Shopify.

• 43:59Speaker 01

  I think it's just one that I keep coming back to. I think semantic search is table stakes, right? And I think, I think it works great as a, as a byproduct of the LLMs. The second one is similarity. I think that people are just expecting this deduping and, oh yeah, there's something similar to this over here. You could also call it recommendations. It's, you know, rose by any other name. The third thing that you want is that you want the ability to generate a report, like sort of ask a question and then, you know, it goes and finds a bunch of information, queries the data. Then you also want some agentic workflows, right? Cleaning up things, like taking actions for you and all of that. And the agentic workflows probably want some of one and two, and maybe also three, to get them done.

• 44:42Speaker 01

  And there's probably others that are idiosyncratic to the particular application, but I think all of those are becoming table stakes AI features in SaaS. And I think we're seeing that the, the, the incumbent SaaS providers are doing a phenomenal job of building these in and really prioritizing it. And I think that the upstarts, there's a massive opportunity to try to get some of these like really, really right and build interfaces that are native around them. But that's how I think about the AI era SaaS.

• 45:10Jacob Effron?

  Yeah. What do you see with multimodal data? Is most, most of the usage of TurboPuffer today text-based and like, you know, where do you see that going?

• 45:15Speaker 01

  Yeah, I mean, it's completely possible to do something multimodal in, in, in TurboPuffer. Again, I, you know, I'm, I look at What the market is doing, not what they're saying.

• 45:25Jacob Effron?

  Yeah.

• 45:26Speaker 01

  And I think that we don't see that many companies yet who are doing multimodal, like over images, over attachment and all of that. Usually the, usually implementations lag a little bit behind, right? But I think it's great. And I think the, the economics of Obik Storage make it really, really nice that you, you embed, you know, both the picture of the product and the description of the product and, and all kinds of other attributes around, around what you're searching. So, and I think that the economics of this are great in that, you know, the economics of TurboPuffer might allow you to just embed all the PDFs and not think too much about what it's going to cost you. But that's otherwise been scary because, okay, someone just uploaded that 2,000-page PowerPoint presentation. Are we just going to embed that and like not charge them extra?

• 46:09Speaker 01

  Like you don't expect all your SaaS providers to start doing usage-based pricing, right?

• 46:12Jacob Effron?

  Yeah. Well, we always like to end our interviews with a standard set of quickfire questions where we basically just cram in all the questions that we didn't have time to hit in the regular interview.

• 46:19Speaker 02

  What company do you think would be most interesting to run AI at?

• 46:22Speaker 01

  I mean, it would be one of the frontier labs, right? It would be, it would be at like OpenAI or Anthropic or one of the ones that are like just seeing the models of three or six months out.

• 46:31Jacob Effron?

  Where's the name TurboPuffer come from?

• 46:33Speaker 01

  Do you want the real reason or do you want the, um, the, the marketing reason?

• 46:38Jacob Effron?

  Uh, definitely the real reason.

• 46:39Speaker 01

  The real reason was that, um, it, it made me happy, sounded, and sounded funny, and it had an emoji that had no other real meaning.

• 46:47Jacob Effron?

  That is a good emoji. And then how have you made, have you, how have you turned that into great marketing?

• 46:52Speaker 01

  When the puffer fish is deflated, it's on Obik Storage. And as it expands all the way into battle stance, it's in DRAM, right? And SS9.

• 46:59Jacob Effron?

  Nice. You must have been very proud of yourself when you, when you, when you came up with that.

• 47:02Speaker 01

  Yeah, you know, it's just, yeah. Maybe a little bit, but it was not the original intent of the name.

• 47:08Jacob Effron?

  Yeah.

• 47:09Speaker 02

  What's one thing you've changed your mind on in AI in the last year?

• 47:13Speaker 01

  You know, I'm getting a lot of like on AI, I spend most of my time still thinking about databases.

• 47:20Speaker 01

  And I think the biggest thing that I've changed my mind on in databases is I just keep being surprised that the simple thing continues to work and it's not a great answer because it's not a good gotcha, but that would be my answer.

• 47:33Jacob Effron?

  Yeah. What do you think is the biggest mistake you've made so far in running TurboPuffer? Or something you look back on from a few years ago, you're like, I wish, you know, we learned that lesson.

• 47:41Speaker 01

  I'm, I feel like we on the product at least haven't committed any, any, any major, any major mistakes yet. And I think people sometimes underestimate how hard it is to run product early at a startup. But the first few customers that we had used every single feature of the product. And there was not, there is not a line of code that wasn't being run in production.

• 48:05Speaker 01

  If I thought about it a bit harder, because we've definitely made a million mistakes, I could come up with a better answer for you. But I think a lot of it is the survivorship bias of getting the product right.

• 48:14Jacob Effron?

  What was something you learned as a founder?

• 48:17Speaker 01

  I get a lot of people who tell me what they think that I should do.

• 48:20Jacob Effron?

  VCs, right?

• 48:21Speaker 01

  Yeah, especially VCs, you know.

• 48:27Speaker 01

  And I've really learned to trust my, my instincts. And I think that when I, when we talk to the team and we have a feeling about something and just giving everyone the permission is like, okay, let's just, let's just try it. That's, that's worked great. Like everyone says you should do embeddings, you should do re-ranking and all of that. It's like, it doesn't quite feel right yet. The vibes have to be right.

• 48:52Jacob Effron?

  Yep, always about the vibes. I assume you think about a lot of like, you know, of questions about the future of where AI is going. If you could like, you know, talk to someone from the future and get one question answered that would, you know, help you in building TurboPuffer today, what would the question be?

• 49:07Speaker 01

  How much is the agent searching?

• 49:11Jacob Effron?

  Like the extent of the vectors that the agent has to go through?

• 49:15Speaker 01

  Not just vectors, but like how much is it utilizing a search engine, right? Like it's very clear that you're not going to do web search by loading that entire thing into context, right? How much are they searching?

• 49:25Speaker 02

  You have an interesting story around how you learned how to code. You know, first you started on the online PHP resources. Then you took a break because there wasn't any more resources. You played a lot of World of Warcraft, which I was a big fan, by the way. And then you learned how to learn English from that. And then you got back into coding. How do you sort of think LLMs will change how people learn to code? And then what do you think sort of the future of software engineering will be with LLMs?

• 49:48Speaker 01

  There's nothing more than I would have loved than an LLM to talk to when I was 11, trying to learn how to program. And you're just like the Danish web on doing web programming was too small. I just like, like I mourn for my younger self to not have had an LLM to learn with. Like it's, I think about that a lot. And I think about it in the context of like my daughter and just like how a curious child now can just get access to so much in such an accessible form. And that brings me a lot of joy.

• 50:18Jacob Effron?

  That's awesome. Well, this has been a fascinating conversation. I'm sure folks will want to pull on all sorts of different threads. I want to leave the final word to you. Where can folks go to learn more about you, TurboPuffer? The floor is yours.

• 50:28Speaker 01

  Yeah, so turbopuffer.com to learn more about the database, the trade offs of it, and what it costs and everything along those lines. I mostly post on, on X, so X.com/syrupson. TurboPuffer is also on there, but turbopuffer.com is the best way and on X, yeah.

• 50:45Jacob Effron?

  Amazing. Well, thanks so much. This was a ton of fun.

• 50:48Speaker 01

  Thank you so much for having me.