Pod Insights Software Engineering Daily Vespa AI and Surpassing the Limits of Vector Search About this episode Vector search has risen to become a foundational tool in modern search and retrieval systems, including the RAG pipelines that power many AI applications. However, the demands on retrieval systems are growing more sophisticated, which is revealing the limits of relying on a single vector similarity score.
Vespa https://vespa.ai/tensor-formalism/?utm_source=software_engineering_daily&utm_medium=referral&utm_campaign=vector_search_article&utm_content=tensor_formalism_link is a popular open source search and data serving engine. Central to Vespa’s architecture is tensor-based retrieval, which is an approach that represents data as tensors rather than simple vectors. Tensor-based retrieval enables richer mathematical operations and more flexible ranking functions that can surmount the limitations of a single vector similarity score.
Radu Gheorghe https://x.com/radu0gheorghe is a software engineer at Vespa with a background spanning nearly 12 years of consulting and training on Elasticsearch and Solr. In this episode, Radu joins Sean Falconer to discuss why vector similarity alone falls short in production, how tensor-based retrieval generalizes to support richer ranking functions, the trade-offs in chunking and multi-stage re-ranking architectures, and where AI search is headed next.
Full Disclosure: This episode is sponsored by Vespa.
https://i0.wp.com/softwareengineeringdaily.com/wp-content/uploads/2023/10/Rectangle-3-2.pngSean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from AI to quantum computing. Currently, Sean is an AI Entrepreneur in Residence at Confluent where he works on AI strategy and thought leadership. You can connect with Sean on LinkedIn.
Please click here to see the transcript of this episode. http://softwareengineeringdaily.com/wp-content/uploads/2026/04/SED1928-Vespa.txt
Sponsorship inquiries: [email protected] https://cms.megaphone.fm/organizations/619b680e-d813-11ea-9750-e7ecac406436/podcasts/e60108fe-e328-11ea-b2b6-1348249f900a/episodes/3913370a-bc44-11ed-ace6-8b1f96d82bba/[email protected]
The post Vespa AI and Surpassing the Limits of Vector Search https://softwareengineeringdaily.com/podcasts/vespa-ai-and-surpassing-the-limits-of-vector-search/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.
Transcript 0:00 Speaker 00 Vector search has risen to become a foundational tool in modern search and retrieval systems, including the RAG pipelines that power many AI applications. However, the demands on retrieval systems are growing more sophisticated, which is revealing the limits of relying on a single vector similarity score. Vespa is a popular open source search and data serving engine. Central to Vespa's architecture is tensor based retrieval, which is an approach that represents data as tensors rather than simple vectors. Tensor based retrieval enables richer mathematical operations and more flexible ranking functions that can surmount the limitations of a single
0:42 Speaker 00 vector similarity score. Radu Jorgue is a software engineer at Vespa with a background spanning nearly 12 years of consulting and training on Elastic Search and Solr. In this episode, Radu joins Sean Falconer to discuss why vector similarity alone falls short in production, how tensor based retrieval generalizes to support richer ranking functions, the trade offs in chunking and multi-stage reranking architectures, and where AI search is headed next. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.
1:31 Speaker 01 Radu, welcome to the show.
1:33 Speaker 02 Hi, thanks for having me.
1:35 Speaker 01 Yeah, absolutely. I'm glad you're able to be here. I interviewed your founder, Vespa and CEO, probably a couple of years ago. So it's great to catch up again on everything that's happening over at Vespa. A lot has changed in the world of AI and I'm sure in the world of Vespa over the last couple of years.
1:50 Speaker 02 Yep.
1:51 Speaker 01 So you've been working in this space for a while. I guess like, what's your origin story? How did you end up working in search infrastructure and ultimately get involved at Vespa?
2:01 Speaker 02 Yeah, this was two jobs ago, working at an antivirus company and we needed to centralize logs and that's how I got into Elastic Search. And then I moved on to a company that was, at the time at least, doing mostly consulting on top of Elastic Search and Solar. And so I've been doing for almost twelve years at that company, consulting, training, that sort of stuff for Elastic Search and Solar, and then OpenSearch.
2:26 Speaker 01 And then what ultimately led you to Vespa?
2:28 Speaker 02 Well, I guess mostly curiosity, because Vespa not being based on Lucene and having different internals, different distributor model, different trade-offs that it makes, got me intrigued. Met a bunch of people at conferences, was more and more curious. And yeah, that's how I got into it.
2:47 Speaker 01 Yeah. I mean, Vespa is a company, at least the origins of it has been around for a long time. Over twenty years has been working on search-related problems. Can you share a little bit about some of the origins of the company? And what was their original problem that they were focused on? And how much of that originating DNA is there today?
3:08 Speaker 02 I know quite a few things, but only from other people, because I wasn't around, of course. I've only been here for like a couple of years. As far as I know, the origins are pre Yahoo. So there used to be a company called FAST, which is a recursive acronym, comes from fast search and transfer. And they've been doing web search and like search in general. I think there were a few other things. But the idea was like large-scale search. And then through a series of acquisitions, ended up in Yahoo. And in Yahoo, they were serving lots and lots of use cases. And Vespa still does serve lots and lots of use cases within Yahoo. Not sure which of them can be told publicly, but the idea is like you have a bunch of verticals
3:50 Speaker 02 that you can serve, some smaller scale, some really huge scale.
3:54 Speaker 01 Hmm.
3:54 Speaker 02 So I think this implies that a lot of the problems that Vespa needed to solve were quite generic as well as large scale. So I think you'll see this in Vespa today, like a lot of the solutions we adopt tend to be over-engineered, if you will, because we expect them to be used and pulled in all sorts of directions.
4:18 Speaker 01 What do you mean by that in terms of over-engineer? Can you give an example?
4:22 Speaker 02 Well, tensors I think are a good example of that, because it's like you don't only support vectors and distance functions and all that. You support all sorts of maths on top of all sorts of numerical structures.
4:36 Speaker 02 So then when you come up with a new use case, it's just that much easier to add it up. Because there are lots of things that are already supported and already thought through to be scalable and fast.
4:49 Speaker 01 I see. So you're talking more about having some first principles thinking around search that generalized all sorts of problems versus attacking everyone as like a narrowly brand new problem that you have to go and engineer a specific solution for that particular problem.
5:04 Speaker 02 Yeah, for me, coming with my consulting background, I'm used to solving specific issues. And at Vespa, when I look at an issue and I'm like, okay, how do we solve this? And people are like, whoa, wait. Let's make sure we don't bump into something like three months later, we have to do this all over again. So is there something generic that we can do? Will this perform at scale and all that sort of stuff? Does this align with how Vespa is used in general and things like that? So that for me is a bit of a shift.
5:32 Speaker 01 Yeah, I think there's always a trade off between moving from like a consultancy or even forward deployed engineer type of role where you're really trying to help solution specific things for a customer and unblock them versus being part of like core product and R and D where you're thinking beyond just a singular customer, but how do we generalize this thing to maybe all of our customers?
5:55 Speaker 02 Yep.
5:56 Speaker 01 So Vespa has been fairly vocal about writing about vector search and how it's reaching some of its limits. For some people, maybe that's a bold claim, vector search, vector database. This is something that's been around for quite some time. I think it's really got a lot of traction, certainly in the last few years. Can you talk a little bit about the core argument behind the dialogue come from Vespa in terms of vector search reaching its limits? Like, are there certain things that vectors are good at and where are things starting to break down?
6:25 Speaker 02 I think the general idea is that people want good relevance, right? You have a corpus, you're searching in it, you need the most relevant things to surface. And so for that, vectors are only one, like in general, vector distance is one signal, right? You may have N other signals, like is this document recent? Does this document match well? Lexical search, which chunk is more relevant? Do we care about the top chunk? Do we take care of the average chunk or average of top 10 chunks or whatever? Business rules we may have. So in practice, what we see is that a lot of people end up having
7:07 Speaker 02 really complex algorithms for measuring what ends up being a relevant score. So having flexibility around this, I think, is important. I don't think there's anything wrong with vector similarity. I just think that vector similarity in itself is not enough. And even if you look at what are traditionally called vector databases, they add stuff on. It's not just vector similarity that they care about. So I think it's just a natural trend that people, when they start to use these things, they just care about multiple signals and also how to combine them.
7:39 Speaker 01 Yeah. So if you're only looking at something like vector similarity, what are some of the things that you might end up getting wrong in some of these use cases or scenarios where you're limiting yourself to the singular signal?
7:52 Speaker 02 Well, one thing that comes to mind is lexical search. I think there's a lot of, let's say memes in the search world about BM twenty five, which is, let's say, most popular algorithm behind lexical search. And that BM twenty five actually performs really well as time goes by. It doesn't, doesn't seem to die. And we had a recent blog post where we benchmarked a lot of embedding models. And most of them, in most of their flavors, would outperform BM twenty five. And first of all, actually, let me take a step back. When I say most of those models outperform BM twenty five, I mean, those model off the shelf outperformed BM twenty five off the shelf.
8:35 Speaker 02 In reality, nobody actually uses that. Most people will tune both their embedding models and their BM twenty five implementation. But for argument's sake, okay, most of those models would outperform BM twenty five. But hybrid search, so just BM twenty five combined with all those models, would outperform the models themselves. So lexical search is a signal that I think is here to stay and has, has been proven over and over again. That is just one example. Another one that comes to mind is if you have long texts, you can have, I mean, vector similarity on the whole text becomes quite meaningless because you can't capture that meaning of a blog post or a book in one vector.
9:16 Speaker 02 So I think that's where chunking comes in. And then how do you combine chunks and stuff like that and metadata? So again, in practice, what we see is people end up adding up on more and more stuff that becomes their compound signal.
9:33 Speaker 01 Can you talk a little bit about like how this vectorization process works? So what do you lose when you turn a document into a vector? Like, what are you kind of giving up by using that representation?
9:44 Speaker 02 Compared to what?
9:46 Speaker 01 I mean, there's other ways that you could potentially represent a document. I'm sure you could do something like you could just store the text of a document, for example, and do some sort of text-based search over it.
9:57 Speaker 02 So then you do lose things like exact filtering. Like, that's one of the main complaints people have about vector search is like, where's my threshold? Where's the cutoff between a relevant and an irrelevant document? Like, with lexical search, this is usually quite easy to figure out. Like, a lot of people in the Lucene world use some sort of minimum match. It's like, okay, if you have three words, then all three need to match. But if I have 10, then seven out of 10 is good enough. And then I have a reasonable cutoff. Okay, it's not perfect, but it is somewhat intuitive, explainable, and most of the time it works. With vector search, it's like, you don't really know, because you can still have for something like cosine similarity,
10:42 Speaker 02 you can say, okay, decent similarity score is like 0.7, I'm gonna cut it off at that. But then that depends that that similarity changes when you're running different queries. So in other words, it's very hard to figure out what that cutoff point is and what that cutoff point means.
11:03 Speaker 01 Hmm.
11:03 Speaker 02 And that messes up with facetting. So if you want to analyze your result set, then you're looking at what? Because vector search, unless you have this artificial cutoff point, then you're going to match everything.
11:15 Speaker 01 Yeah, I mean, I think that would be certainly true of if you're not doing any kind of chunking, like the whole point of if you have a large document or breaking it down into smaller chunks is that you have more like sort of tightly coupled, semantically meaningful chunks. So if I take an entire book and I turn it into one singular vector, then I'm creating a single point in high dimensional space that represents this whole book. Like, there's no way I can encapsulate all the meaning of that book and have all the points in space that are similar to it in some reasonable way. I'm going to end up losing a lot of the specifics, I would think. But if I break it down by paragraphs, or sections, or chapters, or whatever it might be, then at least I have more tightly coupled dots in high dimensional space
11:59 Speaker 01 that are going to probably have a sphere of similarity around it to other dots in that space. They're probably more like semantically meaningful. So the more data I'm essentially trying to stuff into the vector, the more I'm generalizing essentially the ultimate meaning of that thing. Is that fair?
12:16 Speaker 02 Yeah, I think that's a fair thing. It's like you have a limited number of data points, effectively, the dimensionality of your vector that you can store. So the more meaning you have in something big, the more you're going to compress and the more loss it's going to be.
12:31 Speaker 01 Yeah, exactly. Yeah, so it's a very lossy format, especially as you're getting more and more text stuffed into the singular vector representation. There's also, if we look at things like the use of RAG over the last couple of years, and vector databases, typically the RAG systems are getting more and more complicated, where we're doing, we have the pipeline where we're breaking up, we're chunking these things, we can have different chunking strategies, we're indexing it in a vector database, and then when we're actually retrieving it, we're also doing multiple steps where we're maybe retrieving relevant documents, and then maybe we're using like a re-ranking model as well to re-rank the results from it. In terms of this kind of two-stage architecture, where we're decoupling the search from the re-ranking,
13:14 Speaker 01 is that problematic, or there's challenges around not tying those things together and decoupling them?
13:20 Speaker 02 I think it is problematic in the sense that if you have a lot of data to re-rank, then you're going to have a lot of traffic coming in and out. That can become bottleneck.
13:30 Speaker 01 So it's primarily a efficiency problem?
13:33 Speaker 02 Yeah, and efficiency is really important because if you're more efficient, then you can essentially afford to do fancier stuff.
13:47 Speaker 02 Right? Let's take this re-ranking example. So if you have a really good re-ranker that is, performs really badly, you can only throw a few results at it because otherwise you're going to have unacceptable latency. But if your re-ranker is super efficient, then you can throw all your results at it and you're going to have great results. But this, I think, applies all over the place, right? Like, if you can, in general, have a base relevance function that performs well and does a lot of stuff, then you're going to have a really good baseline to work with.
14:19 Speaker 01 And I guess all these problems that we're talking about are perhaps even more amplified in like the multimodal world. It's one thing where we're talking about compressing text into a vector, like what happens when we compress images and video and things like that? Do we lose too much in using this kind of lossy format, especially when we're talking about rich medium?
14:39 Speaker 02 I don't know that I have enough experience with things like audio or video, but I know for things like PDFs, it's going to be really hard to put all that information in a single vector because you can have N pages and on those N pages, I think it's enough if you extract the text. We have the problem that we talked about earlier, which is how we crumble out of text into a vector. But now if you have diagrams in it and other things, good luck.
15:08 Speaker 01 Yeah, and tables, which you're probably going to, I would suspect, like, even the approach to doing similarity measurements between those things is probably going to be quite different than you would do for traditional text.
15:20 Speaker 02 Yeah, so what I've seen with PDFs are people doing vectors per page.
15:28 Speaker 01 Hmm.
15:29 Speaker 02 Or rather, per patch per page. You have models like Colpali and so on that can do this sort of stuff.
15:34 Speaker 01 Do they handle tables and images differently, though?
15:36 Speaker 02 They don't actually. They're pretty generic. You just throw the image of a PDF page at it and they give you a vector per patch. So you'd have typically one hundred and twenty-eight patches, so thirty-two by thirty-two. And you're gonna have one vector for each of those patches, but it's all very well coordinated. So in the end, because you can throw a text query at the same model, so that they live in the same vector space. It can actually figure out whether something's on a table or on a graph. So you can have a graph of, I don't know, energy consumption by month. And you can say, what was the consumption in July? And it can highlight that for you.
16:16 Speaker 01 Okay. I want to get into a little bit of this topic around tensor-based retrieval. Essentially for the listeners that perhaps at this point know what a vector is. And I think generally because of all the, everything that's been happening in AI over the last couple of years. I think people who didn't know what a vector was three, four years ago, you know, perhaps know what a vector is today. They might not be super familiar with the concept of a tensor. Can you explain essentially what is the difference from vector to tensors and why does it matter for search?
16:46 Speaker 02 Yeah, so a vector is a list of numbers, right? The data type can differ. It can be a float. Normally it's a float natively, but we can quantize it. So basically compress the float into, let's say, a sixteen-bit float or an integer or even a bit. So that is a vector. A tensor is a more flexible way to represent numbers. So simple thing could be just to represent one number. You can have an array, which would be a vector. You can have named dimensions. So like I mentioned, patches earlier for Colpali. You can say we have a patch ID and for each patch ID, we can attach a vector. So now we're gonna have a map of vectors.
17:27 Speaker 02 Or you can have a sparse tensor where we can say, let's say for personalization. Right, so I go to a clothing store and I prefer, I don't know, black pants and blue T-shirts and stuff like that. Those could be named dimensions in my tensors and based on my preference, I can store a number. So like, let's say a heavy preference would be closer to one. And maybe if I hate them, they should be a negative number. And so I can perform all sorts of math on top of these numerical structures, these tensors. And I can get the results I want. So for example, with vectors, we can do the similarity search that we all know and love.
18:07 Speaker 02 But we can do personalization, for example, by doing some sort of dot product between my preferences and what a specific item of clothing would be, or
18:17 Speaker 01 Mm-hmm.
18:18 Speaker 02 We can do copally and we can sum up things, we can do maxims. So all sorts of things can be done on top of tensors. I'm not sure if that answers your question.
18:29 Speaker 01 Yeah, so every feature or thing that you want to describe needs to map into a numeric representation, right? And that could be a vector, it could be a singular value, but some numeric representation.
18:40 Speaker 02 Right. In the context of tensors, yes. I mean, with Vespa, you can do much more with ranking than just using tensor math. But tensor math is a really flexible way to represent a lot of things and then do those interactions quickly.
18:55 Speaker 01 Right, so you, by representing things as tensors versus just purely vectors, you have a whole set of tools essentially that you can use to perform these different types of searches using tensor math that you wouldn't be able to support using if you were just doing essentially cosine measurements between two different vectors.
19:14 Speaker 02 Correct. Yeah. And also, I think most importantly, we are very, I wouldn't say completely, because nothing is complete, but very future-proof. So for example, when copally models came in, we could just natively support that because you can have these patch vectors modeled in a tensor and then you can implement maxim using tensor math and there you go. You have all the maxim stuff. You don't need to come up with a whole new feature of how do we deal with this? How do we deal with multiple tensors? How do we combine them in the way that they're supposed to be combined?
19:46 Speaker 01 This was something that Vespa was already supporting?
19:49 Speaker 02 Yeah, I mean, not before the model and the technique came into existence. It's not like we were supporting it, but yeah, we were supporting it from day one because all the plumbing was already there. You just needed to write the correct expression and there you go.
20:03 Speaker 01 I see.
20:03 Speaker 02 So another good example is Bayesian BM25. There's a new technique to normalize BM25 scores because one of the main problems with BM25 is that you don't have a predictable score that you can use to then combine with other kind of scores. So it's like it's it's ideal if you can normalize it between zero and one and then you can treat it much more uniformly. And so when that technique came out, we were like, okay, how do we implement this in Vespa? And it turns out pretty much everything was already there. We, you know, all the sigmoid calculations we could already do in in the rank profile math. So this was impressive even for the author.
20:40 Speaker 01 Can you walk me through what is the process for doing a tensor based search in Vespa?
20:47 Speaker 02 The process would depend on exactly how the tensor looks like. So if you have a vector, you define, I mean, any type of tensor, in fact, you will define it in the schema. It's like, okay, this is the shape of the tensor. This is the data type. And then you feed the data, which should match that shape, right? So if it's a map of arrays or whatever that is. And then when you run the query, you typically also have a query tensor. You can construct tensors at query time from the signals that you may have, like, I don't know, chunk similarities, you can construct tensors from that. And then you would have something called a rank profile. So in the schema, you would
21:30 Speaker 02 say, this is my rank profile. And the rank profile expresses how the similarities, how the score of the document should be computed. So let's say we do a dot product between two tensors or we do a similarity between a bunch of vectors and you can iterate, we can take the average of that similarity or whatever we want, the top N vectors similarity and average that, whatever math you can think of should be there or a lot of the relevant things are already there. And you can construct your relevance function that way.
22:05 Speaker 01 How do I know like what relevance function to use?
22:09 Speaker 02 I think that is very much up to you and like, how do you, let's say, tweak your relevance? It would depend on the use case. I think most people would just start from something simple like lexical search. Then, okay, we can find a decent vector, like an embedder model to work with my data. Then I can think of, okay, what are other business relevant signals that I want to incorporate? All sorts of metadata. So people typically iterate. And I think it's very important to have some sort of golden set that you can evaluate and see, you know, whether my quality is going up or down. Let's say, very generic approach.
22:52 Speaker 01 It sounds like there's maybe some additional complexity involved with like getting this set up and working. But the advantage is that you're trading off, you know, some level of maybe technical investment and complexity upfront. But the trade off is that you get better results.
23:08 Speaker 02 This is valid for any system. I don't think it's particular with tensors. Uh, you know, if you add more signals and you want to combine them, it's going to be just, that engineering investment that you were talking about will happen everywhere. Maybe tensors require a little bit more understanding of some sort of math. Not crazy. I mean, my math stopped at high school and I can still grok it to some extent. So it's not too, too scary, but it's, it's a little bit more than just, you know, at least what I'm used to.
23:37 Speaker 01 How much is DASPA like abstracting away some of that math for you?
23:41 Speaker 02 There are some helper things, like for example, we talked about call poly. You have, there are aliases, like you can just multiply two tensors, for example, like X asterisk Y. And then that's going to do a dot product for you. You can also do the unfurled thing. I think more interestingly, we have a bunch of helper, let's say frameworks, if I can say that. There's something called Tensor Playground where you can go and click around and you have some examples. And you can also come up with your own and you can fiddle with tensors and see what the results are.
24:27 Speaker 02 We also did in a couple of of years in December, we had this tensor advent challenge where the idea was, okay, let's have some thematic challenges you have to solve with tensors, like how much Santa has to pack and how much the elves have to travel and stuff like that, that you would just solve with tensors, just to get a feel of that math. And then there is a quite a big repository, which is called sample apps in the Vespa GitHub, which has lots and lots of examples of use cases. And you can see the rank profiles there and you can see the schema and a lot of people will take one of these sample apps and just change it to what they need.
25:11 Speaker 02 And I think that's useful, like it's rare that you just start from scratch on a path that nobody went to before.
25:18 Speaker 01 You mentioned this a little bit earlier of this concept of like named dimensions and Vespa's Tensor Framework supports named dimensions like token and region, timestamp. What does that give you? Like, why does that design choice matter?
25:31 Speaker 02 It matters because it's a very quick to, let me step back here and try to come up with an example. So one of them is you can have attributes that you care about for ranking, right? So let's say you're searching for cars and you may have things like, is this car expensive? Is this car cheap to insure? Does this car use a lot of fuel? Is it new? Whatever, right? So things that maybe I care about when ranking. So even if you don't have tensors, right, you can still take those into account when ranking, right? You can take the mileage, you can take all those dimensions and you
26:14 Speaker 02 can come up with a formula that takes all those dimensions and comes up with a final dimension, which is the score of my document. But it is quite expensive to get, assuming that you store this in multiple fields, you can, you need to get the value from all those fields. Do whatever math you need to do in some sort of high level math. Hopefully you don't have to bring it all the way to the application because that's gonna be horrible. But even if you have to do this at some sort of high level script, like, you know, with with Elasticsearch is painless, for example, that, you know, can be very slow. By contrast, if you have this natively in a tensor, then you can simply, you know, take the user's preferences, take those car attributes and do a dot product, which is super, super fast.
26:59 Speaker 01 Yeah.
27:00 Speaker 02 So this will scale a lot better than, you know, taking those attributes manually. So I think this is what it gives you in essence, because kind of comes back to what we discussed earlier about efficiency that allows you to do fancier things. Because at some point, you will not be able to do things in other search engines, even though the capabilities are there, but you know, if at your scale, they don't make sense, you're not gonna use them, right? It doesn't help you that they're there if you can't use them. But with tensors is different because a lot of those tensor operations are super fast. And so they will scale and and people do use them at very large scale.
27:40 Speaker 01 Yeah, I mean, one of the things that I think that's seems unique about Vespa around some of this like efficiency stuff that you're speaking to is that the tensor computation is happening on the content node where the data lives. It's kind of the idea of like, do you bring the data to the computation or bring the computation to the data? Data is expensive to move around. So if you can bring the computation to the data, then it's gonna save you some cost in terms of time of like moving this data around, which then gives you probably more compute cycles that you can spend on trying to, you know, get good results out of the search.
28:15 Speaker 02 Exactly, and I think this comes at two levels. One is the computation that you do on all the documents. I think it's just unfeasible to bring all the documents somewhere outside where the data lives, right? Like you will have to take some sort of top end. Unless you have a tiny data set, you just cannot afford to take all the data out of the content nodes and into something external. And so that is one thing, like if you can do some tensor computation on the first level, then you're. Going to save a lot of cycles. But then the other thing is you can do the first reranking phase. So basically the second phase, the first phase runs on all documents. Second phase runs on top end.
28:58 Speaker 02 That second phase will run on the content nodes as well. So you can bring a more sophisticated model, you know, could be a light G B M, like a tree. It could be on a next model. It's usually not something super big, but it can be, you know, complex enough. For example, to handle multiple signals of multiple ranges, such as, you know, have similarity. And then we discussed B M twenty five and recency and all the things that, you know, maybe matter to you. And and come up with a coherent score. And that still happens on the content node without moving moving data. And only later you can maybe move a much smaller set to what we call the global reranking, which happens on a stateless layer.
29:43 Speaker 02 And that can again have its own model, maybe a bigger, more complex model can also run on G P U. That can can do the final reranking. So there's sort of stages to that.
29:55 Speaker 01 I guess one of the things I was wondering about too is, you know, there's like the concept of RAG, you know, vectors, or even if you're using some other, you know, search technique, like the tensor stuff that we're speaking about has is very, very popular a couple of years ago. And then now with A I agents, I think there's some dialogue around like, how relevant is this today? Like, can you talk a little bit about, you know, where do some of these concepts fit into the agent world?
30:20 Speaker 02 Right, so they for agents, this would be just a search. I don't think they care all that much about what happens under the hood. But if what happens under the hood gives them good results quickly, then that is, I think, even more important than it is for humans, because agents would typically run multiple searches. And so the problem, I think, would be compound with latency or with bad results. Because it's latency, definitely.
30:49 Speaker 01 If you're ninety percent accurate in isolation and then you do that ten times, then it's like, you know, point nine to the power of ten, which means that you're successful. Like, I don't know what the math is, but it's probably going to be like ten percent success in that compound factor of searches, right? So the more you can get the accuracy up on the search in isolation, the more the accuracy goes up in the aggregate as well.
31:12 Speaker 02 And I think the other thing is that models. At least to my knowledge, to this day, aren't as good as figuring out how to filter the context. So if you if you give them bad results, they will tend to hallucinate more because now they have bad context to rely their hallucinations on.
31:35 Speaker 01 Yeah, or if it's too much, right? Like all models degrade in performance, the larger the context that you give them due to context rot. I mean, just based on what the way the attention mechanism works, they can only pay attention to so many things. So they might pay attention to the thing that you don't want them to pay attention to if you give them bad results.
31:52 Speaker 02 Yep.
31:53 Speaker 01 How does Vespa handle like updates to data? So if you have like knowledge base that's changing every minute, you know, there's news, there's pricing, there's inventory. How does the index, a re-indexing of that information work?
32:06 Speaker 02 So to talk in general terms, Vespa is real time, meaning when you make an update, the moment you get the acknowledgement as the application, that thing is searchable. So most engines would be near real time, meaning there has to be some sort of commit happening, which, you know, there's always a trade off, like there's no free lunch, right? But this is the trade off that Vespa does. It's like it assumes that you need your data to be available right now. So you won't have some some caches that you have with other engines, but the upside is
32:45 Speaker 02 that you can, for things that are moving quickly, such as pricing for e-commerce, that is a very frequent example, or you know, how much you have in stock, that you can change a lot. And if the data you're changing is an attribute, so effectively the price or the, you know, in stock thing that is kept in memory, that is super, super quick. This contrasts with other systems where you have a commit and then you would effectively need to kind of re-index the document in order to change one value from it, which can be prohibitive. But in Vespa, that's the advantage that you can you can quickly update things.
33:26 Speaker 01 How does that like, you know, technically work? I'm not sure I'm I'm following like if I have a new update, how does Vespa handle that in real time? Like a continuously flow of new information, like how does Vespa handle making that available in real time?
33:39 Speaker 02 So if you have an in-memory attribute, like a price, and you want to change it, I mean, it will be backed by disk, right? So you have all the persistence, right ahead log, all that stuff. But you send the update, it's changed in memory, it's also replicated to all the other nodes. And when all the other nodes got the update request, you get the acknowledgement from the client. And also, this happens at the operation level. So if you want to update three, let's say, products' prices in one go, the way you typically do this with Vespa is with HTTP two. You're going to send, we have libraries that do this.
34:20 Speaker 02 You're going to send effectively ten updates individually and they respond individually. And each of them, at the moment they respond, you know, they're already kind of flipped in memory. So you see the new price, every searcher that runs after that will see the new price or whatever you updated.
34:38 Speaker 01 Vespa has been in the search world for a long time, like we've talked about it. So, you know, over a twenty year journey. What's next for search? Like if we fast forward ahead three, five years, like what are some of the problems that need to be solved? That haven't been solved today?
34:53 Speaker 02 I don't know, to be honest. There's so much work in the short term that I find it hard to look because things are moving so quickly. It's it's hard to tell. I do have a feeling that multimodal search should become more important. Like it would have visual cues here and there that will become more important depending on the use case. I would think that the ability to explore data in real time. Would also be increasingly important. I think people are. And even agents are not necessarily happy with seeing top end results. They may want to know what else is in that result set.
35:34 Speaker 01 Um.
35:34 Speaker 02 And that brings yet again, the question of what is the result set, where, what do we consider, where's that threshold between relevant and irrelevant results? And yeah, I think there are also problems that have been there since before I got into search, which was like fifteen years ago. And are still not really solved, which is like, how do we get a good golden set? How do we, you know, measure search effectively? How do we get that feedback loop going? How do we improve performance? Not performance in sense of latency, but relevance without breaking other things. Yeah, I think if those have been around for more than fifteen years, I would assume they will be around for the next five years as well.
36:19 Speaker 01 I think the golden data set problem is a huge one, even outside of search, just in like A I in general. If whatever A I system I'm building, like if I don't have a good data set to essentially like test against. How do I know that the investments I'm making are moving in the right direction? And I I I see this, a lot of, you know, companies and projects like skip that step. Probably because it's hard. But it's really, really hard to know whether the things that you're doing are actually useful. If you don't have any way to test against it, but people skip that step because there's not like an easy way to achieve it essentially right now.
36:54 Speaker 02 Yeah, and I feel like it's also a chicken and egg problem. Like even if you do it, which as you said, not everyone does it. But even if you do it, it's like, how do you know your testing thing is good? How do you?
37:05 Speaker 01 Yeah.
37:06 Speaker 02 Make sure that, because that's, I think, the main difference between what we see on the internet when people publish, oh, this is the new state of the art model. This is the new state of the art technique, this and that, or academia. They have a golden set. That golden set is the benchmark. So the assumption is that the golden set works. But if you're starting your, I don't know, e-commerce shop or book search website, whatever. Search use case and you start from scratch, like now what? Would, how do you know?
37:37 Speaker 01 I mean, I think that's the advantage that some of the stuff around like coding has is that typically companies have, you know, a history of things that they can kind of build like benchmark data sets around. Like there's issue trackers, there's prior code that engineers have built. Like there's been essentially a history of creating stuff that they can mine for creating these like golden test sets. But if you're starting brand new in a new field where the measurement of what good is is far more subjective than just like compiling and running something against unit tests. It's like really, really hard to create those data sets. And even if you put the work into creating it, to your point, like, how do you know whether it's good or not? Radu, thanks to you so much for being here. It was a great conversation.
38:19 Speaker 02 You're welcome. Thanks for having me.
Previous SED News: Anthropic’s Mythos, Supply Chain Hacks, and the AI Spending Surge Next Open Source Sustainability Page updated November 6, 2026 · transcript and summary are machine-generated