Open source, on-disk vector search with LanceDB

Prashanth Rao mentioned LanceDB as a stand out amongst the many vector DB options in episode #234 https://changelog.com/practicalai/234. Now, Chang She (co-founder and CEO of LanceDB) joins us to talk through the specifics of their open source, on-disk, embedded vector search offering. We talk about how their unique columnar database structure enables serverless deployments and drastic savings (without performance hits) at scale. This one is super practical, so don’t miss it! Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Chang She – GitHub https://github.com/changhiskhan, LinkedIn https://www.linkedin.com/in/changshe, X https://x.com/changhiskhanChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: LanceDB https://lancedb.com/Episode #234 “Vector DBs beyond the hype” https://changelog.com/practicalai/234Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am CEO and founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris? Doing good today. How's it going, Daniel? Oh, it's going great. I was just, well, we were just remarking before actually starting the recording that one of the great things about about doing these episodes is that we get the excuse to bring bring on the show the coolest open source and tooling and other projects that I'm using day to day and get the chance to interact with. And one of those

• 1:23Daniel Whitenack?

  is Lance DB. And we're really excited today to have with us Chang Shu, who is the CEO and co-founder at Lance DB. Welcome.

• 1:33Speaker 02

  Thanks. Hey, guys. Super excited to be here. Thanks for having me on.

• 1:37Daniel Whitenack?

  Yeah, yeah. Well, first off, congrats on on all your success. I was scrolling through LinkedIn and saw like a video of Lance DB up on the NASDAQ screen in in Times Square. So that that was cool to see. That must mean good things, I'm assuming.

• 1:55Speaker 02

  Yeah, this is possible for um via Brex and also Essence VC. So, big thanks goes out to them.

• 2:03Daniel Whitenack?

  Cool, cool. Yeah, well, I mentioned um I've had a chance to look through some of what you're doing and actually use it day to day. Actually, that was a result of a previous episode that was, I think, titled, you know, Vector Databases Beyond the Hype with Prashant. I think the question that we asked him was like, oh, there's all these vector databases. You've compared all of them. What are some of the things that stand out or some of the vector databases that stand out in terms of what they're doing technically? Or how they're approaching things? And one of them he called out was Lance DB. I think in particular, he was talking about kind of on disk index stuff.

• 2:44Daniel Whitenack?

  And so I'm sure we'll get into that and a little bit more, but that's how I got into it. So I recommend listeners maybe go back and get some context from that episode. But as we get into things, could you maybe give us a little bit of a picture as to how Lance DB came about? I know there's a lot of hyped vector database stuff out there and people not might not sort of realize how these things were developed, how they came about, what the motivation was. And so if you could just give us a little bit of a sense of that, at least for Lance DB.

• 3:19Speaker 02

  Yeah, absolutely. And first, I wanted to also give a big shout out to Prashant as well. As you were saying, there's a lot of hype and noise in this area. There are a lot of different choices. And for users and developers who are building generative AI tooling and applications, it's always kind of confusing. Like which one is good? And should you listen to the marketing from one tool versus another? So it's great to see someone with an engineering background who can write so well to actually take the time and just try out a ton of different tools and interview a bunch of different companies and come to his own conclusions. I'm super happy and excited that he's a fan of Lance DB and we hope to make that better for him and also all of our users.

• 4:04Speaker 02

  So, you know, back to Lance DB, I think, so we started the company two years ago at this point. And we didn't start it out as a vector database company, actually. Because I think if you kind of remember, ChatGPT is barely one year old.

• 4:20Daniel Whitenack?

  Yeah, the dawn of AI.

• 4:22Speaker 02

  Yes, exactly.

• 4:24Speaker 02

  And so the the original motivation was actually serving companies building computer vision and building new data infrastructure for a computer vision. So I had been working in this space for a long time. I'd been building data and machine learning tooling for about almost two decades at this point. I started out my career as a financial quant and then I became involved in Python open source. It was one of the original co-authors, the pandas library. And that really got me sort of excited about open source, about Python and building tools for data scientists and machine learning engineers. And so at the time, this was in twenty twenty and twenty twenty one, what I observed was at the company I was working for, Tubi TV, so it was a streaming company.

• 5:09Speaker 02

  So we dealt with both machine learning problems for tabular data and also for unstructured data, like images and the video assets and things like that. And what I had noticed was that anytime a project touched this multimodal data for AI, from images to like the text for, you know, so let's say subtitles or summaries to the poster images, these projects always took a lot longer, they were much harder to maintain, and it was difficult to actually put into production. At the same time, so my co-founder, Lei, who I had met during my days at Cloudera, he was working at Cruise and sort of dealing with the same issues.

• 5:51Speaker 02

  And so we put our heads together and our conclusion was that, hey, it's not the sort of top application or workflow layer or orchestration layer that's the problem, it's the underlying data infrastructure. If you look at sort of what's been out there, like, you know, Parquet and Org has been around, and they've been great for tabular data, but they really, they really suck for managing unstructured data. And so we essentially said, hey, what would it take to build a single source of truth where we can toss in the tabular data plus the unstructured data and give much better performance at a much lower cost, a total cost of ownership?

• 6:32Speaker 02

  An easier foundation to build on top of for companies dealing with a lot of vision data. And so this comes in handy when you want to explore your large vision data sets for, you know, let's say all time is driving. This comes in really handy for things like recommender systems and things like that. So we started out building out that layer of that storage layer in in the open source. And that took about a year's worth of effort to really get to a shape that is usable, kind of like Parquet or ORC and other formats in these tools. And that was when generative AI became really, it burst onto the scene and became sort of a revolutionary technology.

• 7:16Speaker 02

  And what happened at the time was we had originally built in vector index for our computer vision users to say, hey, let's deduplicate a bunch of images or let's find the most relevant samples for training, for active learning and things like that. And it was sort of that open source community that discovered that, hey, this can be really good for generative AI as well. That's when we sort of separated out another repo to say, hey, this is a vector database. And it's much easier to communicate with the community than to say, hey, you're looking for a vector search, use this columnar format. And so that's how we got on to this path.

• 7:57Chris Benson?

  Quick question for you. It's really a follow up to something you said in, it's been a couple of moments now as we were going through that. But I was just curious, when you were talking about kind of going through the analysis on the top workflow versus whether it was infrastructure and you said y'all concluded infrastructure. I was just wondering, you kind of went on past that into that, but I was kind of wondering, how did y'all come to that determination? For those of us who are not deeply into that thought process, I was wondering where your head was at when you were doing that.

• 8:24Speaker 02

  Yeah, it wasn't an easy decision or conclusion. Thinking back, it was kind of, you know, it's like 2022 initially seemed pretty crazy when we sort of first came up on it, right? If you think about it, it's like, why would you make a new data format like in 2022? Like Parquet has been working so well. And I think it was really observing the pain in our own teams. And also we went out and interviewed a lot of folks managing unstructured data. And so for them, it was, you know, one, data was split into many different places. Like the metadata might be managed in Parquet and the raw assets are just dumped onto local hard drives or S3.

• 9:05Speaker 02

  And then you might have, you know, other tabular data managed in other systems. And they, they would always talk about how painful it is to stitch everything together and manage at all togethers. And some of the outcomes are like, it's really hard to maintain those data sets in production. Like you have a Parquet data set that has the metadata and then links to S3 or something like that to all the images. And then somebody moves that S3 directory or something like that. And now all of your data sets are broken. Or something like, we would interview folks are like, hey, what are you doing to explore your visual data sets and things like that? And they're like, well, you know, I use MacBook and there's this app on that called Finder.

• 9:47Speaker 02

  And if you single click on a folder, it shows you a bunch of thumbnails. It was sort of this horrible way to actually work with your data. But it, it was because it was so hard to manage all of that, that machine learning engineers and researchers were stuck with these subpar tools.

• 10:04Daniel Whitenack?

  You mentioned kind of this transition of thinking from some of the original use cases that you were talking about with computer vision to this world of generative AI that we're, that we're living in now. From my impression, from an outsider's perspective, it seems like Lance DB has kind of positioned itself very well to serve this kind of generative AI use cases, which I'm sure we'll talk about in a lot more detail later on. I'm wondering, from your perspective, like, how has that overwhelming demand for the generative AI use case kind of changed your mindset and direction as a company and a project and open source tooling and all of that?

• 10:47Daniel Whitenack?

  And how do you envision the kind of, what you're targeting as the use cases moving forward, I guess?

• 10:54Speaker 02

  I think certainly generative AI has brought in a lot of different changes and new thinking. One was the sort of focus around use cases of semantic search and just retrieval in general. I think with the advent of generative AI, I think retrieval becomes much more important and ubiquitous. For us, what that means is, you know, increased investments in terms of getting the index to work really well and really scalable. And then sort of making that data management piece to work really well as well and integrating with frameworks for RAG

• 11:35Speaker 02

  and for, you know, agents and for just generative AI in particular. When we started out, inevitably, we were dealing with, you know, multi-terabyte to, you know, petabyte scale like vision data sets and things like that. And we're still dealing with a lot of that, but for generative AI, I think there was a renewed focus on ease of use. Because a lot of users are coming in who don't have, you know, years of experience in like data engineering or machine learning engineering. And what they're looking for is a easy to use and easy to install package that doesn't require, you know, you to like be an expert in any of these underlying technologies.

• 12:18Speaker 02

  We also spent some effort into, okay, that was sort of the motivation behind us making Lance DB, the vector database, one, open source, and two, embedded. Because we felt like there were lots of options on the market that required you to say, figure out, okay, like, what is the instance I need? How many instances do I need? What type of it? Okay, like now I have to shard the data and blah, blah, blah. And coming from that data background, you know, what I had been working with a lot is like SQLite or DuckDB that just runs as part of your like application code that was and just would just talk to files that live anywhere. And it was super easy to install and use.

• 13:00Speaker 02

  And so that's sort of what gave us that inspiration to make an embedded vector database.

• 13:07Daniel Whitenack?

  You had just got into this sort of idea of embeddings, or sorry, embedded databases, which, well, embeddings are related, but that's another topic. But the idea that Lance DB is embedded, you mentioned DuckDB and other things that kind of operate in the same sort of sphere. I'm wondering, for those that maybe are trying to position Lance DB's vector database tooling within a kind of wider ecosystem of vector databases and like plugins to other databases that support vector search, could you explain a little bit about like, what does it mean that Lance DB is embedded? Like, what does that mean practically for the user?

• 13:51Daniel Whitenack?

  Maybe people aren't familiar with that term quite as much. So what does that mean practically for the user? And are there other kind of like general ways that you would differentiate Lance DB's tooling and the database versus some other things out there?

• 14:06Speaker 02

  So I love sort of geeking out about these topics. So at the very bottom layer, in terms of technology, I think there's a couple of things that fundamentally sets Lance DB apart. One, as you mentioned, is the fact that it's embedded or runs in process. I think we are one of two that can run in process in Python, and we're the only one in JavaScript that runs in process. Number two is the fact that we have a totally new storage layer through Lance column or format. What this allows us to do is add data management features on top of the index. And then number three is the fact that the indices, the vector indices and others in

• 14:48Speaker 02

  Lance DB are disk-based rather than memory-based, so that it allows us to separate compute and storage, and allows us to scale up a lot better. Right? So those are kind of the big value propositions that these technological choices bring to users of Lance DB. So number one, ease of use. Number two, hyperscalability. Number three, cost effectiveness. And then number four, the ability to manage all of your data together, and not just the vectors, but also, if you think about it, the metadata and also the raw assets, whether they're images, text, or videos.

• 15:26Chris Benson?

  Could you kind of describe a typical use case of a developer doing this where you're kind of taking those features that are distinguishing Lance DB from, you know, other possibilities, other competition, but just talk about what that workflow looks like, you know, or if there is a major one or a couple and just kind of get it very grounded so somebody that's listening can kind of understand how they're going to do it from A to Z when they're integrating Lance DB into their workflow?

• 15:53Speaker 02

  So there's a couple of sort of prototypical workflows that we see from our users. I think at the smaller scale for land CB, you know, you're installing it via like pip or NPM or something like that. And in general, you get some input data that comes in as like pandas data frame or maybe a polar's data frame. And then you interface with an embedding model. You can do that yourself, or you can actually configure the land CB table and say, hey, use OpenAI embeddings or hey, use these hugging face embeddings. Lance DB can actually take care of all that. So it's a pretty quick sort of data frame to land

• 16:34Speaker 02

  CB. And then you can search it and then that comes out as, you know, data frames or Python dicks or things like that that plugs into the rest of your workflow that are likely data frame or pedantic or Python dick based. So that's number one. And then a kind of number two is really these large scale use cases where some of our users have anywhere from like 100 million to multiple billions of vectors in one table. And that's a much bigger production deployment. And typically, what makes Lance DB stands out in that area is one, it's very easy for them to process the data using a distributed

• 17:14Speaker 02

  engine like Spark. And they can write concurrently and get that done really quickly. I think we're one of the few that offers GPU acceleration in terms of indexing. So even for those really large data sets, you can index pretty quickly. And then number three is because we're able to actually separate the compute and storage, even at that large vector size, you don't really need that many query nodes. Like, you can actually just have one or two, like, fairly average and commodity query nodes that runs on your storage of choice, depending on what latency requirements you want, and then just have a very simple architecture. For these types of architectures, the query

• 17:59Speaker 02

  nodes are stateless, they don't need to talk to each other. So when you need to scale up, or when a node drops out and has to come back in, there's no sort of leader election, there's no coordination, it really lowers the complexity of that whole stack. So another great example of this kind of architecture and the benefits that it brings is Neon, the Neon database. So I think Nikita, who's the founder, recently had a good Twitter thread about the difference between Neon and other databases. And he called it, you know, shared data versus shared nothing architecture. And I think that's also what we kind of strive to deliver in Lance DB versus other vector databases.

• 18:42Daniel Whitenack?

  Yeah, I know one of the things that I really enjoyed in trying out a lot of things with Lance DB is I can pull up a Colab notebook, right, and try out, like, I can import Lance DB, I can import like a subset of the kind of database that I'm gonna be work, or the data that I'm working with. It all runs fine. I don't have to, like, set up some client-server type of scenario. And then when people ask, well, how are you going to, are you gonna push this out to a larger scale? The appeal of just saying, hey, well, we can just throw up this Lance DB database on S3 and then connect to it. That's a very appealing thing

• 19:23Daniel Whitenack?

  for people because also those storage layers are available everywhere from on-prem to cloud to whatever sort of scenarios you're working with. So it's very, very flexible for people. Could you explain a little bit? Because this is something like... I've been asked a couple of times, but so this is my selfish question because I have you on the line. So you're helping me with my own day to day work. But when I'm, when I'm talking to like some people, clients that I'm working with, I'm like, oh, we can just throw this up on, on S3 and then access it. Usually their question is something like, well, like, because they have in their mind, a database has a compute node.

• 20:04Daniel Whitenack?

  And like the somehow the performance of queries into the database is tied to the sizing of that compute node and maybe like how that's sort of clustered or sharded across the, the database. And then like this idea, oh, I'm just going to have even just a lambda function that connects to S3 and does a query, right? Like this kind of, in some ways it like breaks things in people's mind. And so a lot of times their question was like, how does that work? How can a query to this large amount of data be efficient when the data is just like sitting there and in S3 or in another place? So could you help with, help me with my answer, I guess, is what I'm asking?

• 20:45Speaker 02

  Yeah, absolutely. So this goes back to what we talked about earlier with separation of compute and storage. And if you've been sort of steeped in like data warehousing, data engineering land, this has been a big arc of data warehouse innovation in the past decade. By allowing us to scale up the storage versus the compute separately. This is the thing that makes the system seem magical where you can, you can process a huge amounts of data on what seems like, you know, pretty commodity or pretty weak compute. And so the analogy that I like to make with the situation is kind of like, a lot of us are familiar with,

• 21:25Speaker 02

  let's say like DuckDB demos or videos and you could see instance where DuckDB is processing hundreds of gigabytes of data on just a laptop and in a very fast amount of time. And they are able to spit out results and almost interactively. And there are companies, you know, from like Mother Duck to, you know, there's a new company called Bao Plan that is looking to essentially distribute DQDB queries on AWS Lambda's. It's basically the same thing. It's all about the separation of compute and storage. And that's only possible if you have the right underlying data architecture for storing vectors and the data itself.

• 22:09Daniel Whitenack?

  And just for someone that like is not a database developer, can you describe in any words like the generalities of that data structure that enables such a thing?

• 22:22Speaker 02

  Yeah, so it's two things. One is the, one is the columnar format. So typically, you know, from GAI to machine learning, you can have very wide tables. But typically, a single query only needs like a couple of columns. So columnar format allows you to only have to fetch and look at like a very small subset of that data. Number two is the, that columnar format needs to have be paired with an index, like the vector index in this particular scenario. And that vector index, in order to give this separation of compute and storage, has to be based on disk. So you have to be, store the data on disk, not force the user hold everything into memory and then be able to access that very quickly.

• 23:06Speaker 02

  And then number three is how to connect that index with a columnar format. So a columnar format like Parquet does not give you the ability to do fast random access. So even if you had that good index, using Parquet, you would not be able to get interactive performance in terms of queries. And it's only by having a new columnar format like LAMS that can give you fast random access and fast scans that you can successfully put these two together and deliver the thing. So, so those are the three sort of big pillars that I think in our data architecture that makes this possible.

• 23:42Chris Benson?

  Well, we were talking here, I'm going through GitHub on your repo and stuff, and was surprised at something that kind of prompting the next question. It looks like you're really addressing a wide range of different types of needs. And so there's obviously Python, as you would expect, but you have JavaScript. And then I was delighted to discover that there's a Rust client in there, which is when I'm not doing AI-specific things most of the time. That's my language of choice these days. Could you talk a little bit about kind of two things? The broader, like what you're trying to achieve, like how you choose what languages to support, and how you're getting there? And then if you'll scratch my itch, what is your intention with that Rust client?

• 24:27Chris Benson?

  Is it ready? What does it do? Just because I'm fascinated with that. Sorry.

• 24:30Speaker 02

  Yeah, absolutely. I love talking about Rust. The Rust package is actually not a client, but so on the core of the, both the data format and the vector database is actually in Rust. So the Rust crate that we have is actually the database or the embedded database. And so, and we actually built, for example, the JavaScript, again, the same thing with JavaScript. It's not just a client, but it's also an embedded database in JavaScript. So that is actually based on top of the Rust crate. And kind of like you have in, so like Polers or something like that. You have like a Rust core and then you connect that into JavaScript.

• 25:09Speaker 02

  So we had actually started out in 2022 writing in C++ because Parquet is written in C++, you know, like serious data people and database people write in C++.

• 25:23Chris Benson?

  Until they find Rust, of course.

• 25:25Speaker 02

  Right. And it was sort of a hack project during Christmas time in 2022, at the end of 2022, where we had to do a hack project for a customer, actually, and where we had to actually re-implement partially the re-path for LAM's format. And what we found was just, it was so good that we decided to just actually rewrite everything in Rust. I think biggest things were, we were a lot more productive. We rewrote. Roughly six months of solid C++ development in about three weeks with Rust. And we had, this was like us learning Rust as beginners as we went along.

• 26:10Speaker 02

  A lot of that initial Rust code has again been rewritten over the past year, but it just made us feel a lot more productive. And then number two is the safety that Rust offers you has been amazing. With C++ like every release, it just didn't have a good feeling. It was almost like, you know, where's that next seg bolt gonna come from? Whereas with Rust, you know, we felt very confident making multiple releases per week with, you know, major features. And we did not see anywhere near the sort of issues that we saw with C++. Right? So everything has been really great.

• 26:50Speaker 02

  And I know that like Rust has become really popular now for, actually even with vector databases, like Quadrant, I think is Rust, Pinecone. They're not open source, but in turn, they publicly said that they've written their whole stack in Rust as well. Um, so.

• 27:06Chris Benson?

  One more question from you along the same line before I let it go, because we've hit that sweet spot that I love. Do you think, and this is not specific to Lance DB, but based on what you're saying, clearly you're thinking ahead on these things. Um, as we go forward and you see both the AI applications and you see the different types of workflows and infrastructures, you know, becoming broader and more supportive, the multi-language aspect of getting out of only Python, for instance, do you, do you foresee that as a convergence where you're seeing language agnosticism developing in this space as it has in other areas of computer science? Or do you think that we're still gonna be kind of locked in

• 27:48Chris Benson?

  on the current sets of infrastructure and tooling, very Python oriented for the indefinite future? What is your thinking along those lines?

• 27:55Speaker 02

  So I think generative AI definitely changes the picture in that I think there's a very large like TypeScript JavaScript community that has been brought into the arena to build AI tools. And so I think this is also an underserved segment where, you know, it's not just vector databases, but data tooling in general lags far behind in JavaScript slash TypeScript land versus Python. And I think there's a real opportunity for the open source community to create, you know, good tools for for this part of the community as well.

• 28:34Daniel Whitenack?

  I want to hear about some of the actual use cases that you've seen people implement with LanceDB. Maybe if there's ones that stand out like, oh, this was cool because whatever it was, it's they used it at scale or it's like fits a very typical generative AI use case or whatever. And then maybe something that surprised you in terms of, oh, I didn't. Always when you put a project out into the world, there's these things where, oh, I really didn't expect people to be using it that way. But yeah, that sort of makes sense. So do you think of anything that fits into one or both of those categories?

• 29:13Speaker 02

  The use cases for LanceDB in the community that I see falls into three or four large buckets. One is of course generative AI, RAG and things like that. And I think there, I think it's not so much the use of LanceDB that I think is really cool, but it's the applications that people build with it that is really cool and amazing. And I think a lot of the applications that people build that is cool, that really takes advantage of LanceDB is things where you need RAG to be very agile and that you need it to be.

• 29:51Speaker 02

  Really sort of tightly bundled with your application. You can sort of call this RAG from anywhere and have it return pretty quickly and without too much complexity. And so this is where I see a lot of folks from like your standard like chatbots and chat with documentation to things like productivity tools where, you know, they build things that help people organize their daily schedules to, you know, much more high stakes things, you know, in production and like, you know, code generation or, you know, like healthcare and legal and things like that. And so there, I think typically you see vector data set sizes from like the tens of thousands

• 30:36Speaker 02

  up to single digit millions of vectors typically, right? And so, you know, production means you really scale up both the number of data sets that you have and then the number of vectors that you have. And one of the cool things that I've seen that takes advantage of Lance C B and Lance format uniquely is there's a code analysis tool that sort of analyzes your GitHub repository and plugs it into a RAG like customer success sort of tool. And what they want to be able to do is say query the state of the database like this today versus yesterday versus a week ago to say, hey, was this issue fixed or not?

• 31:20Speaker 02

  And like, what's still outstanding? And so Lance C B uniquely gives you this ability to version your table and also do time travel. So you can say any data vector database can do like, give me the ten most similar things to this input uniquely, but what Lance C B gives you the ability to do is say, give me the ten most similar as of yesterday or as of a week ago. And we do that sort of automatically for you. Yeah, and then I think the other big buckets are, you know, e-commerce and a search and like recommender engines. Right. This is like the traditional use case for vector databases. And there you tend to see much bigger like single data sets that are, you know, say, I want to store like item embeddings.

• 32:03Speaker 02

  Maybe that's, you know, up to a couple of million, up to 10 million. I want to store item embeddings that could get up to like hundreds of millions. And you don't have as many tables, but you have potentially have very large tables, right? And then, of course, the last bucket is this like computer vision, uh, like AI native computer vision, either generative computer vision or things like autonomous vehicles and things like that. And there's a whole sort of combination of more complicated use cases that enables active learning, deduplication, things like that. And the thing that is very unique about the use case of Lance DB in there is companies that are managing all of their training data in Lance DB and Lance format as well.

• 32:46Speaker 02

  So you can use the vector database to find the most interesting samples. And then you can actually use the tooling on top of the format to essentially keep your GPU utilization high and keep your GPU fed very quickly during training or if you're fine tuning or, you know, if you're running evals and things like that.

• 33:06Daniel Whitenack?

  Yeah, so cool. I, uh, one of the things, uh, that has been most fun for me recently is this combination of an LLM Lance DB and DuckDB where like you can create these really cool. So, um, if I'm using an open LLM that can generate like SQL queries or something, but I have like all of these different SQL tables, like what we're doing is like putting descriptions of the SQL fields and tables in Lance DB and actually on the fly like matching and pulling. Those to generate a prompt which goes to the LLM to generate the SQL code, which is executed with DuckDB. And this gives you like the kind of really nice

• 33:50Daniel Whitenack?

  natural language query to your data type of scenario, which has been really fun to play with.

• 33:56Speaker 02

  That's really good to hear. Actually, sorry to interrupt. So, because you kind of nerdsized me.

• 34:03Speaker 02

  So one of the things that's really cool about DuckDB is its extension mechanism. And so I think they've also published like a extension framework for Rust-based extensions. And so we have sort of a basic integration going there. And I think in New Year, what you can expect from us is actually, we're going to be spending a little more more time to make that integration be more rich, meaning our goal is to be for you to be able to say, to write like a DuckDB UDF to do vector search. And then the results come back as as like a DuckDB table where you can then run additional query, like DuckDB queries on top of that.

• 34:44Speaker 02

  And so, and sort of the same, and the same thing with like Polers, right? So you can, and the goal is to essentially make it so that people like vector database is no longer a thing that you even have to think about. It's people are generally more familiar with like DuckDB or Polers as the sort of that tool that stitches together the workflow. So we just want that to make it feel even smoother and more transparent.

• 35:10Chris Benson?

  A couple of moments ago, when you were talking about the use cases, you were talking about, you know, like autonomous vehicles and stuff. And I was wondering if if we could pull that thread a little bit more. It seems like it is a fantastic.

• 35:21Daniel Whitenack?

  Chris loves drones.

• 35:22Chris Benson?

  Yeah, I love drones and I love things that are not by data centers.

• 35:26Speaker 02

  Yeah.

• 35:26Chris Benson?

  I love things that are off on the edge, whether it be for inference or including training concerns that you may not have all the things that we're so spoiled with with our cloud providers out there. And it seems like, you know, there's many types of of opportunities to use that. What's your thinking around that? Have you seen any use cases? Any any ideas for the future in that kind of autonomous on the edge world?

• 35:53Speaker 02

  Yeah, definitely. So we certainly have, so some of our users are like robotics or device companies, where they either collect data and write it as LANCE on the edge, or they sort of collect data as like, let's say, Protobuff or something like that, and send it off to be converted into LANCE for like analytics, vector search, and so on and so forth. I think in this world, you're gonna know it better than me. So what, but what I see is that, one is the data is super complicated. So especially with, let's say, like, vehicles, types of use cases. You're getting visual data from the cameras, you're getting point clouds from the LiDARs, you're getting time series data from the sensor readings over time.

• 36:37Speaker 02

  And then you've got manual input data from like the auditors and the drivers that are sitting in the car. You're also getting metadata about the car, about the weather, about the geography and all of that, right? So like, being able to manage that and query all that together, I think will be super important for, you know, robotics and vehicles and in any sort of, any company that's putting things out there in the real world that's generating data in the physical world. And I think that, yeah, I mean, it's a really hard problem, but I think the, the potential is huge, right? Because I think for AI, we're, we're going from this era of like, very canned like question-answer to much more free-form question and answer.

• 37:22Speaker 02

  But it's still a little bit passive, right? You're like, you're asking it for information. But what's really exciting would be like, you know, you marry these sort of generalized AI capabilities with a drone or a robot or, you know, something that can go out and be active out in the real world.

• 37:41Daniel Whitenack?

  That gets me super excited about what's to come. I'm wondering, as we close out here, it's been a fascinating discussion. At the end here, could you just take a moment and make a few observations about what is exciting from your perspective right now in this sort of?

• 38:01Daniel Whitenack?

  Practical AI space, because that's where you're living. What excites you about, you know, whatever it is, the next six months, the next year? And what you think is kind of coming as this tooling rolls out there further and further? People learn to apply it better and better. What's exciting for you?

• 38:19Speaker 02

  That's a great question. I think there are, there are lots of things that I think holds a lot of promise in the next six to 12 months. I think we'll see, one is this explosion of retrieval, kind of information retrieval tools. So we already see a lot of companies that are adding like generative AI in like customer success management. And like documentation and things like that. And so I think we'll see a lot of applications providing value that is, you know, that can be also personalized. And, you know, not just like chat GPT style answers, but actually personalized to their own data or their own, you know, cases or, or things like that.

• 39:04Speaker 02

  And then number two is I see a lot of successes in very domain specific agents that are able to dive deep into legal or healthcare or some domain very specifically and build things that seem sort of magical, whether it's, you know, compliance or driving better outcomes or, you know, creating things that would democratize a lot of these sort of like very deep expertise type of, of domains. And then I think a little bit further out are generalized, like low code to no code tools for you to build, you know, very sophisticated applications using generative AI through code generation

• 39:48Speaker 02

  and sort of creative, let's say creative interfaces and things like that. So those are things I'm, I think we'll deliver in the short term. And then, you know, personally, like I love games and I'm actually super excited about what generative AI brings to gaming. You know, we talk about open world and things like that. And this is, this can be really open, where you could just get lost for a long, long time in a generative world.

• 40:14Daniel Whitenack?

  It's awesome. Thank you so much for taking time to talk with us. And please pass on my thanks to the Lance DB team for making me look good in my, in my day job by giving me great, great tools that work really well. Appreciate what, what you all are doing. And yeah, I just looking forward to seeing what comes over, over the coming months. And yeah, encourage our listeners to check out the show notes, follow the links to Lance DB, try it out. It only takes a few minutes and I hope to talk to you again soon. Thanks so much.

• 40:47Speaker 02

  Thank you, Daniel. Thank you, Chris. It was a super fun talking with you guys. And if you have any feedback, please let us know. We hope to make you look even better in the new year.

• 41:05Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a long time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.