Building Brave: Private Search, One AI Layer at a Time with Josep M. Pujol

In this episode of the Cognitive Revolution, join us as we dive into a compelling conversation with Josep M. Pujol, Chief of Search at Brave, about the complexities of developing a privacy-focused search engine. Explore how Brave maintains user data privacy while managing over 1 million searches per hour with an AI-powered system. Gain insights into the significance of human evaluation in AI and learn about the potential of the Brave Search API. Don't miss the shared Google collab notebook link in the show notes! 🔥 Apply to join over 400 founders and Execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj Google collab notebook : https://colab.research.google.com/drive/1AxQD-bDePl1l90A_gwvfAGXBIl92ERKy#scrollTo=WrhGZsvbxZy8 SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ Recommended Podcast: Byrne Hobart, the writer of The Diff, is revered in Silicon Valley. You can get an hour with him each week. See for yourself how his thinking can upgrade yours. Spotify: https://open.spotify.com/show/6rANlV54GCARLgMOtpkzKt Apple: https://podcasts.apple.com/us/podcast/the-riff-with-byrne-hobart-and-erik-torenberg/id1716646486 CHAPTERS: (00:00:00) Introduction (00:05:46) Brave (00:07:52) How Brave collects data (00:10:45) The problem is not the data (00:14:02) How Brave builds its index (00:16:36) PageRank (00:21:07) Sponsors: Oracle | Brave (00:23:14) Brave Search (00:26:26) Hardware improvements (00:28:33) Language models (00:34:17) Noise reduction in vertical search (00:36:50) Fine-tuning models (00:43:09) Sponsors: Squad | Omneky (00:44:55) Ensemble approach (00:58:00) The future of the web (01:03:30) The business side of this (01:09:22) Brave Search users (01:14:21) Brave Search API vs. Google (01:20:00) Brave Search API Pricing (01:23:26) Brave Search API Use Cases

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Josep Pujol, Chief of Search at Brave. As regular listeners will know, Brave has been a sponsor of The Cognitive Revolution for the last several months, and that makes this our first sponsored episode. I think it's important to mention that right at the top for the sake of transparency, though I did approach this episode

• 0:43Nathan Labenz?

  in exactly the same way that I always do, with a deep dive into the company's products and an earnest search for insights that will be worth your precious time. And I can sincerely say that I believe this episode maintains the quality standard that we always strive for. As you'll hear, Josep was very open and shared a number of fascinating details about how Brave has built their search product and infrastructure over time, including the trade-offs that they accept to maintain the highest standard for user data privacy. Some of the most interesting details center around the challenges of iteratively improving an AI-powered product while operating at real scale. Brave handles over 1 million searches per hour, or roughly 300 searches per second. They do not want to take any steps backward,

• 1:27Nathan Labenz?

  and so their approach is generally to add to their system, making their ensemble of models and techniques more effective, but also more complicated with each generation. I found this approach really fascinating to learn about as it's a perspective not often found in younger startups. I also appreciated Josep's candid perspective on the challenges they face in evaluating search results. Brave has developed some scalable systems to assist in measuring quality, but Josep emphasized that there still is no fully automated substitute for hands-on human evaluation, good taste, and judgment. I think this too is an important lesson for all AI builders to keep in mind. Coming away from this conversation, I genuinely believe that the Brave Search API is an excellent product. I've spent more time testing it since the recording and for the

• 2:10Nathan Labenz?

  Waymark Small Business Profile Builder, which today relies primarily on scraping small business websites, I did find it to be notably better than both Bing and Google. As such, I have recommended it to the Waymark product team. My expectation is that like Brave, we will augment rather than replace our existing system. We'll use the Brave Search API to help us return results faster, to cover URLs that our scraper fails on, and to surface valuable bonus context from discussion and review sites that we simply aren't tapping into today at all. Overall, I think it will be a really nice product win and given the low cost and latency, I think it's something that more AI builders should consider using. If you'd like to check it out for yourself, I've created a simple Google Collab Notebook, which makes Search API

• 2:53Nathan Labenz?

  calls to Brave, Google, and Bing. It's nothing fancy, but this is exactly what I've used to evaluate it for Waymark. We'll put the link to that notebook in the show notes. Finally, before we get started, if your company is interested in sponsoring the Cognitive Revolution and potentially being featured in an episode, I invite you to reach out. We'll always be fully transparent about sponsorships and we will only produce sponsored episodes with companies that I personally judge to be worthwhile on their merits. So my hope is that this will be a win-win for the show, for the sponsor, and most importantly, for you, the audience. As always, we hope you'll share the show with friends and we invite your feedback. And with that, I hope you enjoyed this inside look at Brave Search and the Brave Search API with Brave's

• 3:34Nathan Labenz?

  Chief of Search, Joseph Boujol. Joseph Boujol, Chief of Search at Brave, welcome to the Cognitive Revolution.

• 3:42Speaker 01

  Nice, nice being here and thank you for having me.

• 3:46Nathan Labenz?

  My pleasure. I'm excited to get deep into all the exciting developments and AI technologies that you have assembled into the Brave Search experience. I guess just as a quick prelude, regular listeners to the show will know that Brave is a sponsor of the Cognitive Revolution, so we thank you for your patronage in supporting us. I've obviously listened to the ad copy a few times and I know the audience members have as well. Do you want to start off by just giving us a little bit of a general background on Brave, now my default mobile browser, by the way? Just tell us a little bit about the company at a super high level, you know, what the kind of motivations are, the ethos of the company, and then we'll dive after that more deeply into the technology.

• 4:27Speaker 01

  Yeah, Brave, first of all, is a great company founded by Brian Bondi and Brandon Eygh. And the aim, they started as a browser, especially with a focus on privacy. And myself, I joined Brave like three years and a half ago, out of a company that was called Clicks or Tailcut, Spinoff, and we were basically were developing a search engine from scratch. And when we, like when a previous company went belly up because of lack of funding, COVID and whatnot, well, we had multiple potential places to go, and the one that was more appealing was Brave, because we share like the same values, privacy preserving, and also like the aim to be an alternative to big tech, right?

• 5:12Speaker 01

  It's not so much that what we tried to achieve is that to do something totally different, right? Because at the end of the day, it's a browser, it's a search engine, it's a talk software, like a video conferencing. But the point, the whole point is that, well, you can actually like have the full technology suit, right? Without having to rely on big tech with all the compromises that it entails. So it's very appealing. It's very appealing to me personally and to the team. That's why we joined Brave, and I think that's the best definition there is. Of course, there's a bunch of features that are pretty unique, but to me, and the biggest value proposition of Brave is that while you use Google, one you can use Brave.

• 5:56Nathan Labenz?

  I've been impressed actually by just how big the product line is. When I started to get into it more, the mobile browser experience is really good, and the farther I got, it was like, oh, there is a VPN that you can enable here, and there's a chatbot language, but language model powered experience, question answering, all that stuff. It really is a pretty, and an ad network as well. There's a lot going on. How many people at the company? It's not that big of a company. I'm impressed by how many different products.

• 6:25Speaker 01

  But people, Brave as a whole, and the people working on search, like engineering and quality raters, like the core of search is about 20.

• 6:35Nathan Labenz?

  Twenty, wow.

• 6:36Speaker 01

  On search, right. There is a lot of, not just that search is around by twenty people, right. There is marketing, communications, HR, there is a lot of shared resources, right. But, but anyway, like the search is like twenty people, Brave as a whole, two hundred. So it's a very small company and yes, we do have a lot of reach. But anyway, that's the proof that you can, I mean, you don't have to be big to provide value.

• 7:01Nathan Labenz?

  Yeah, it's impressive. We'll get into some of the head-to-head comparisons as we get a little bit deeper into this. So as head of search, we're going to spend most of our time doing a deep dive into the search technology, how it works. As I've said, our audience is very interested in technical details. Many people that are listening are building stuff and they are looking for the, the hard one lesson. They were looking for shortcuts to the, the lessons that you've learned the hard way. So maybe just for kind of starters, like you want to give a kind of a, a general overview of how search works. One thing that definitely stood out to me again from the ad copy

• 7:39Nathan Labenz?

  is that the data is at least to some degree sourced based on pages that people are actually visiting. And that was, for starters, an interesting data point.

• 7:52Speaker 01

  Yes, actually that project, the use of browsing data, if you want, that's not a start from Brave, it starts from the previous company. That's how we can actually like gather enough data to start developing the search engine. But it's a very controversial topic, right? Because there is like certain tendency to, to have this dichotomy with either data or privacy. And this dichotomy is actually false, right? It's as simple as that. I wrote extensively about it. People can check it out. But basically, the, the whole point is not so much that data is a problem. It's like how you collect that data, right? I'm not talking about opt-in, opt-outs, or like a legal issue, right? Those aside, it's more like.

• 8:38Speaker 01

  What kind of purpose the data will have and how the data is collected, which makes it dangerous or not. For instance, like Brave, if you, people who want to contribute, they can opt in to something is called a web discovery project, right? And in there, they will send us browsing data, right? That sounds dangerous, but what do you mean, right? It's not that we collect your browsing history or your search history. No, that would be dangerous, that would be like privacy problematic. What you will contribute is like individual data elements that are anonymous, right? So someone has visited this site and has engaged so much with it. Then, what, any of those data elements that people send us,

• 9:23Speaker 01

  we have absolutely no way to know whether they come from the same person or not. So we do not learn anything about you or about me, right? What we learn is that someone is anonymous. There is a lot of technical details here. It's not that, just, we do not use user ID's, neither implicit or explicit. It's just like the data is like basically sent using a mixing network to remove like network fingerprinting. The messages are homogenized so that they actually have to pass a quantum. So more than X people need to send the same amount of data, the same data in order to be actually received by us. That like a good approval to have. So there is like a lot of like a technical merit.

• 10:06Speaker 01

  But at the end of the day, what is important is that the data that we receive is anonymous, is not anonymized, right? Or is pseudonymous, no, it's, those are dangerous, right? Those are the things that they can later be anonymized. The data that we got is just anonymous and it serves a single purpose, right? In this case, to know that this particular page is popular or not, right? Or that this particular page was clicked after a particular query, to build query logs. So that's the kind of data that we collect. And without that, would be actually very difficult to build a search engine. Google and Microsoft do the same thing, whether they admit it or not.

• 10:49Speaker 01

  Actually, there is like a nice release of some documents that happened yesterday or like a couple of days ago, where there is like some private APIs of Google. That's natural, this data is collected. And again, the problem is not the data. The same way that the problem is not the advertisement, right? The problem is like if by doing advertisement, you actually put your privacy at risk. Because your tracking is dangerous, right? Knowing your browsing history or your search history is extremely problematic. And that's something that we do not want to know, and we do not collect. One of the attack vectors that we consider is that if we actually had to release our data

• 11:30Speaker 01

  either to a hacker or to a government agency of some sort, we could release it knowingly that no particular person could be identified or profiled.

• 11:42Nathan Labenz?

  Yeah, that's really interesting. So if you're in a country where you have reason to be concerned about this sort of stuff, then this becomes an extremely valuable option?

• 11:51Speaker 01

  Yeah, first, it's an option. Even though it's privacy preserving by design, still we do it an option because it's something that people have to be aware of. But they could enable it, we would not be able to. It's something that's basically technically it's called like record unlinkability, right? So any data element that we receive, we have no way to know if those two elements come from the same person or not. That means that we only have individual data elements. Because even if the individual data elements, if we had a way to link them, we could actually create a profile. And based on that profile, it can always be anonymized, right? Because there will always be one of the data elements that somehow probabilistically

• 12:35Speaker 01

  or deterministicly can link to you. And then basically, the whole session is compromised. The whole point is that we have no technical means to actually build a session so that, you know, like there is like, yes, somebody visited this page, but we cannot know what happened afterwards. But of course, that data is much less useful than profile data.

• 13:00Speaker 01

  Because profile data, it's not that profile data is evil. Let's put names on it just for the fun of it, right? It's not that the Google engineers are evil and say, I want to collect everything or not. But basically, what you want is that, once you collect data, you want to be reusable. You want the same data that you collect to be able to count how popular the page is, but also like how pages relate to each other, or like how popular a particular restaurant is at what time. And you want the same dataset to be able to answer all those questions. And that's very powerful, but that's very problematic, right? Because it can answer all these legitimate questions, but it can also answer illegitimate ones. Like, give me the search history of whoever was here at this particular time.

• 13:43Speaker 01

  So what we cannot do is answer these generic questions, right? So for every question that we have, we collect one particular data element that only answers this particular question. So that's very different. The how is important. It's not so much about data or no data, it's like the purpose of the data.

• 14:02Nathan Labenz?

  So how does this compare? Or maybe it's, maybe you combine. I guess you have a crawler too, right? So there's, it's not that the index is entirely built from the page visits, but more that it is informed by the page visits?

• 14:16Speaker 01

  Yeah, to give specifics on that. We may learn about our URL through web discovery process, right? And we know this URL is popular, right? Because it has to pass a form. So more than end people have to see that URL in order for us to receive that URL. So that means that URL does not belong to you. It's not your login URL. Because like, you know, a hundred people across the world that access that URL. So that when we receive that URL, we only receive the URL. We don't receive the content. Because the content, there is like no guarantees that the content does not contain private information that belongs to you. So then we receive the URL, which is public information, and then we actually go and fetch it.

• 14:59Speaker 01

  So that allows us to, basically, that our index to be like small enough, but still, because it's small because we do not crawl blindly. We don't do like a blind crawl of everything. What we do is that we crawl what people basically visit.

• 15:18Speaker 01

  And from there, I'm from trusted sites, so we actually do some additional crawling. But most of the crawling we do is not crawling, it's called fetching, right? Because we actually fetched the content. It's not that we proactively crawl the whole web, because the whole web is full of noise, right? There is like at least 10 different sites that are clones of GitHub. As odd as it sounds, it is. There are like sites that are clones of GitHub. So if you crawl blindly, you just like increase by 10 your size, but only by adding noise. So that's like one of the, one of the things that we use people for, to become more efficient

• 16:00Speaker 01

  and to reduce the noise, which is like the ultimate goal of any machine learning or AI.

• 16:07Nathan Labenz?

  Yeah, I was just gonna say, the cleaning the dataset is often where the majority of the work goes.

• 16:13Speaker 01

  Absolutely, that's like the main, that's like the main realization. Noise reduction is the single most important thing that you have to do ever. Because noise does not only affect the quality of the output of your models, but it does affect scalability of those models and also the cost of running those models. Noise needs to be avoided as much as possible.

• 16:36Nathan Labenz?

  So do you have a sense for the relative size of the index compared to what either Google is doing or the whole web?

• 16:45Speaker 01

  It's a tenth of what Google has. It's a tenth of what Google has, and we can achieve like 99.9% of the work that Google has.

• 16:54Nathan Labenz?

  Yeah, interesting. Yeah, power law is in action. So a little bit about the signals that you're looking at beyond the behavioral. Obviously, we've got from traditional search projects, we've got like keyword-based indexing, PageRank, I assume you're still getting some value from like that sort of LinkedIn hierarchy? No?

• 17:21Speaker 01

  No, page rank, no, no, how polemic can it be?

• 17:24Nathan Labenz?

  As polemic as you want.

• 17:25Speaker 01

  Page rank is a myth, you know that.

• 17:28Nathan Labenz?

  Yeah, let me tell you what, let me think about what I know, and then you can correct me. I thought it worked well at the beginning, and then it became subject to a lot of adversarial attacks with SEO of a million kinds. And now I would still assume that it has some power, but that it has had to be updated a lot of times in a lot of ways to counteract all these SEO optimizations.

• 17:54Speaker 01

  Okay, I already was an adult at university, I almost started to PhD when Google came out. I started using Google when it was not even google.com, it was Stanford. They released two papers. One was the PageRank, which was like a proper scientific paper, and the other was like a technical report. Like the anatomy of a large search engine. Where it had the meat of why Google was so good. Two was not good because of PageRank. PageRank was a gimmick. It was a gimmick that worked, of course, had a contribution. But was good because it had a, it was an algorithm. And it had a paper on it that was published. And it was something that journalists love to, to stress, right?

• 18:36Speaker 01

  The algorithm that does, that changes everything. That's what people want to hear, like people want to hear a story that there's like an invention that does, that there is like a straight line, there is like something happened and we went from here to here. And that's not what happens. In reality, things are like a little bit more. The reason why Google was so good from the get-go is because of Ankor text. Larry and Sergey through Stanford were the first ones who, using Stanford resources, were able to crawl the whole web, crawl the whole web brute force. And that's why you will see like there are some pictures having like hard drives on Lego structures.

• 19:19Speaker 01

  Right, because they had to buy so many hard drives to be able to store the whole web. That they did not have like iron to put it, so they actually had to create like Lego stands for it, right? So the project was like that, where they were able to like crawl the whole web. What does it mean to crawl the whole web? That suddenly, unlike AltaVista, right? You had a much better representation of what the page is, right? Because you have the backlinks. And the backlinks, what does it take? It really has the anchor text. It's basically like everybody else's description of your site. As opposed to what we had before, which was like your own description of what your site does. Remember, noise is everything.

• 20:04Speaker 01

  Noise reduction is everything. What is less noisy? The content of a webpage? Or the summary that thousand different people do about your page? But it's it's much less noisy, the anchor text. And that's why they were able to actually have much better recall with much better precision. And then, you know, like, well, they want to have the anchor text, you have the backlinks, and the backlinks have an adjacency matrix. And with the adjacency matrix, you can calculate the first day vector, which is like a stationary state of random walk, yala, yala, yala, very nice paper. Out. But that's like cherry on top of the cake. The real meat was their ability to brute force via something that was not possible

• 20:49Speaker 01

  a couple of years before, to actually brute force and crawl the whole web in a single file system. And then the realization, very smart, okay, we have a much cleaner representation of what the page is than the page content. That's what made Google.

• 21:06Nathan Labenz?

  That's fascinating.

• 21:08Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:12Nathan Labenz?

  How does that play into the Brave search today? My sense is still that it still matters if a reputable site links to you versus an unreputable site. Seems like there's some sort of authority notion.

• 21:25Speaker 01

  There is there is authority, right? But it's not done on page rank. It's rank, you can actually write it out of the authority of the pages. There's no popularity.

• 21:33Nathan Labenz?

  Just on traffic.

• 21:35Speaker 01

  Just on just on traffic, that's that's that's enough. Then, yeah, you might do a one or two levels deep of like calculations. But in any case, again, that's a very small contribution to the end goal. What the takeaway message here is not so much the history. It's a little bit more like that. Very often, like the real innovation happens on something that is not as fashionable as an algorithm. It's more like on the methodology, right? And actually, that happens on Brave Search too. Brave Search. Is as good as it can get. It's because, for example, we rely very heavily on query logs. Right? A query log is, in a way, is like, is a representation of what the page is about.

• 22:19Speaker 01

  That is not done on the anchor text, but on the queries of another person. In a way, like, search started being a recommender system. And that allows us to, right, to keep building query logs, which are pretty close in. And as you grow, you have more query logs, and then you're gonna start to do semantic relations with your queries. You have, you've never seen, like, what is the age of Lady Gaga, right? But you have seen Lady Gaga's age. And then, if you've seen the other one, and you know that which is good for this one, it's gonna be good for the other query, because semantically it's very, it's very. So you keep expanding the query semantically to do semantic search on queries. Not on content, but on queries, right?

• 23:03Speaker 01

  And then you start to be able to use, like, models to generate semantic queries. Not to try to index the content as a whole, but to create what would be queries that would be answers to this page. Right? And then you do this, and then you add it, and then you start, you become a little bit better. Then you start to index content of the page, so you start to use Angrams. Right? Which is more conventional approach. And more expensive than the other ones. So like, the cost to benefit ratio is worse. But now it makes sense, right? Because it's the next step that you have to take. So you keep adding, and what you end up is having something that actually works. Doesn't make a nice history. No, because the story is boring.

• 23:45Speaker 01

  Your story is, no, you just keep building and keep iterating until there's a, you keep, like, doing 1% increments until you have something that works. But that's like the only way that things typically work again. And that's why I put so much emphasis on the example of the page rank. Because that's like the, the story. And you're like, every step. You keep adding and every step, there are things that happen that were not possible before. For instance, back before before Brave, the Black We, like the same concept that we started was not possible five years before we started. And why not? Because they were like not because doing a recommender system means that you have to have everything on.

• 24:28Speaker 01

  And that's a big cache, you know, a big memory map. So at a time where like the servers had sixteen gigs of RAM, max, to host something that has two terabytes, it takes a lot of machines, a lot of machines means a lot of latency, it's not feasible. However, at that time, at Clicks, Amazon came, Amazon released a machine that had one terabyte of RAM. Suddenly, it was possible to create a cluster of four machines to actually put everything on memory. And everything became very easy, right? Where before, one year before that, it would have required like two thousand smaller servers. Unfeasible from the G N perspective. And those kind of things happen all the time, right?

• 25:12Speaker 01

  Another big hardware improvement that without this, Brave Search wouldn't exist. NVMe's hard drives. Without NVMe's hard drive, Brave Search wouldn't exist. It would not be cost effective. And NVMe's, I guess you know what it is, right? It's like hard drives that are super low latency on lookups, that's faster.

• 25:33Nathan Labenz?

  Is that the solid state disk as opposed to spinning disk? Or is it even a more fine distinction than that?

• 25:40Speaker 01

  No, it's actually faster than solid state. There is spinning disk, impossible to do what we do. Flash, still not fast enough, but at least it's like an order of magnitude slower than RAM.

• 25:52Nathan Labenz?

  Gotcha, interesting.

• 25:54Speaker 01

  So suddenly you have what, hundred terabytes, a petabyte of NVMe's. Now you can actually do what you want to do. Before, it was even the, basically like the way of doing it would have to be done differently. That would actually like increase costs by an order of magnitude, increase everything by an order of magnitude, right? And it's not doable. So again, that's the thing like, I believe that's how Brave Search is, was built, is like incremental. Always taking advantage of what the world had to offer in a way.

• 26:26Nathan Labenz?

  So that history of the hardware enablement is really interesting. Can we do the same and equivalent history for the progression of language models or semantically enabling models? Because going back to like early days, you had literally just keywords and then N-grams. And then now, obviously, we've got pretty amazing language models. But I would imagine that like the best language models are maybe an overkill for some of the use cases, if only because of cost and latency. So what's that history like? And where are you guys today in terms of the models that you use for semantic purposes?

• 27:02Speaker 01

  Yeah, we might example I told you before, Lady Gaga age versus How old is Lady Gaga? That's a semantic magic, right? We started to use embeddings, semantic embeddings, based on a model that was called Star Space. I don't know if it's familiar, it sounds familiar.

• 27:20Nathan Labenz?

  I don't know it specifically.

• 27:22Speaker 01

  Two thousand nineteen, right? So a long time ago. And basically, this relationship of being able to go from Lady Gaga's age to How old is Lady Gaga? It was done via embeddings, two hundred dimensions, two bytes, vector search. But that back at the time was not called vector search. It was called like nearest neighbors approximations. And that's what actually what we built. So we actually took advantage of these recent technologies. Again, there was like no only semantic embeddings. Actually, there was also like N-grams based on queries. Remember, like we were basically very heavily based on queries. Just to give you a number, a nice number.

• 28:04Speaker 01

  Our query similarity system has more than nine billion unique queries. Not so like the query hotmail is only one, so nine billion. So whenever you actually get a new query, we are able to return a set of like similar queries that we have seen in the past out of these nine billion. In twenty milliseconds, right? And that is done both using embeddings, but also using angles based on queries. Right? Because we need both. Since you cannot be semantic hundred percent, right? And that's, that was true back then and it's true now.

• 28:40Nathan Labenz?

  Why not? What's the limitation there? Or where does the purely semantic approach fall short?

• 28:44Speaker 01

  Because purely semantic is, is purely semantic. Sometimes the queries that people do are not said, doesn't have a semantic meaning. It's like a particular model or a particular product. Period. And that's what you want, or like an error on a stack trace. Or like a name of someone. My name is Jose Pujol, right? It's not Jose Pujol. Semantically, it's very similar. Spanish and Catalan will give you a very close, a very close distance in any semantic model, but it's not the same. It's a different person. You need both. The takeaway message, if there is any takeaway message that I could give people is that there is like no single bullet, that there is no single anything that solve a problem that is worth solving, right? It's always like a combination of multiple techniques and models,

• 29:29Speaker 01

  and then you do an ensemble on top of it. So can you do only embeddings, semantic embeddings? No, you need to have both. So we do have everything on the ground. So for example, on, on page content, right? So we have page content. We do have embeddings based on transformers, dark light transformers. Yes, we, we have those, right? But at the same time, those are like what we call dense embeddings. Right? Because the dimensionality that we have is the three hundred eighty-four dimensions with two bytes per dimension. And each page have about one to five embeddings, right? Of those, using different part of the content, et cetera. But that's like the, of the semantic matching,

• 30:13Speaker 01

  but we still have what we call sparse embeddings, which is sixteen thousand dimensions, one bit per dimension, which are for literal matching. And when we search, we use both, right? And it's good to have both approaches. Because you want to, sometimes when you search for, I don't know, Trump Tower, you just want Trump Tower, right? You don't want Trump, something else. A lot of noise can come out of Trump Tower, right? So you want to combine both approaches, and that's what we have. So this mentality, I guess, is shared by many people, including any person who comes to work for us out of university,

• 30:54Speaker 01

  that, oh, yeah, I just like doing embeddings out of, I don't know, OpenAI, put them on a vector database and do a search, and that's it. That doesn't work. It only works for very narrow use cases, or like for vertical searches, when there is like one, only one domain. But it would not work on a general purpose, because it will bring up noise. Statistically, you can think of it the following way. The embedding is not free of errors, right? So let's say that once every 10,000, the embedding has an issue, right? Doesn't provide the right embedding, for some, for whatever reason, right? If you are working on a database of 100,000 elements,

• 31:39Speaker 01

  that would not affect you. You will see that it's working fantastically well, right? But if you have 20 billion, the chances that those errors correlate to your initial recall set are very large, right? And then you're blind, right? Because you have a recall set of 2,000 documents, and 10% of them are noise. What else do I do? I need other things, I need other techniques, I need other models to be able to deal with the noise, right? And that's always there. So there is no single thing that works. It's always like a combination.

• 32:15Nathan Labenz?

  Yeah, that's probably a really good takeaway for a lot of listeners who are building all manner of rag apps in all kinds of different contexts. I've got a project that I'm involved with that's trying to take old plumbing catalog parts manuals and make that searchable. This is for a company that has like a proprietary database of all these old manuals. They literally have them like in paper form in a giant bookshelf, and they've embarked on this.

• 32:45Speaker 01

  But this actually, I can tell you another story that we... That might actually work, because in a way, like the level of noise that you have on vertical search... Is artificially reduced. I don't know if it makes sense.

• 32:59Nathan Labenz?

  Yeah.

• 33:00Speaker 01

  Because the problem is always the noise, right? The uncertainty. So when you're like doing a vertical something, vertical search, or you're working a particular vertical, noise is artificially reduced. Right? Because in a particular domain, a particular word has only one particular meaning. Or a particular token has only one particular meaning. And so when you work only on a vertical, it's very easy to work on. And actually, we, one of the things that when we first started to tackle problem of search, the initial thing to to try was like, hey, my search can be split into verticals, right? So if we are like plenty of people, why don't we just do vertical? You do like weather search, I do city search,

• 33:45Speaker 01

  you do famous people search, car search. Like you come up with like hundred different verticals. So what ended up happening is that all the hundred verticals worked perfectly, right? But there was like no way to put them together in a in a way that made sense, right? Because what we actually did unintentionally is like we pushed the problem, the real problem, we push it under the rug, right? On the module that, the mixing module that will come in the future. And the mixing model is a problem, right? Because on the movie vertical, Ted has only one meaning, right? The movie Ted could be Ted one, Ted two, Ted three, right? But now, forget about that you're on a movie vertical. Think of the whole web, but Ted, what does it mean?

• 34:28Speaker 01

  The movie, name of someone, the conference, a part, right? Which one it is? That's like the, that's why search is complicated. Because it's general purpose. And AI has the same issue, right? The narrower the domain, the easier it is because there is less noise.

• 34:48Nathan Labenz?

  Does that assume, though, that you are working with enough data? Going back to this plumbing example, and I think trying to generalize from that to What I think a lot of people who listen to this show are working on, I think in a lot of scenarios, there's like a proprietary data set that is like too big for people to like page through, right? So they do need some search to access it effectively. But then at the same time, it's not so much data that they could train a model from scratch on it. Maybe they could fine tune. And so I feel like a lot of people find themselves in that in-between zone where they're like, I need to use some sort of pre-trained open

• 35:28Nathan Labenz?

  source model, at least as a base. And those models, like in this plumbing scenario, everything is in a very narrow section of the like broad general purpose embeddings models. And then things like parts numbers, as you said, are not even really semantic at all in a lot of cases. So we are finding that we do need a hybrid search, if only to handle these things like parts models.

• 35:54Speaker 01

  That's correct. And I think that's why, in a way, it links back to the point, right? And then you're like on this particular use case, and the first temptation of if you want to be like pure AI, let's say, is now I need to fine tune, right? Really? Perhaps you just can use like this genetic model of the shelf, and then have another system that does exact matching, and then you combine it. So that is the right approach. Perhaps you need to find to do fine tuning, right? It depends on the use case. But at the but at the end of the day, what is important on engineering is not so much the tool. The important thing is the problem. And you just use the tool that is most convenient, and you combine different tools. That's the most important thing, right? Do not fall in love

• 36:38Speaker 01

  on a particular framework or a particular methodology, right? What you have to fall in love is with the problem.

• 36:44Nathan Labenz?

  Yeah, totally. With that in mind, how do you guys approach?

• 36:50Nathan Labenz?

  Upgrading technology, I think this is another thing that I haven't, I'm like only starting to get to the point where I'm encountering this in live apps. I have a company called Waymark, which does video generation. And there we've been through like several rounds of upgrade of generative models. I haven't got to this point yet on a search or a RAG type product where, oh, look, there is a new embedding, or there's a new vector database that could support that embedding. How, I'm sure you've been through several rounds of this stuff over time. How do you approach that challenge and try to tame it?

• 37:23Speaker 01

  It depends, right? For instance, on the summarizer, right? We have this AI summarizer that people can check where we try to provide an answer to a query, figures like about like for 25%, 30% of the queries. Those basically that's of course there is like we do RAG against ourselves, as anybody would. We do put a lot of emphasis, we can discuss later on that, on the data selection, which which snippets we pick and we run narrower QA models on the snippets to like to fit. But ultimately, we do have a prompt that we send to a large language model, right? This large language model has changed many times, right? We had

• 38:06Speaker 01

  summarizer B1, that was released one year ago. I think we started with Plantify, and then we applied it to another model after a while. Then we did some fine tune version of it. So we did three changes in the course of like few months. And summarizer B2 was even more extreme, because when we released, the final model was mixture of 7 bit, 7, 8 billion. Lama 3 came out two weeks after. We tested, we saw that it's actually, we had tests, tests were like comparable quality with using much less resources. We switched it. It took us a week to switch. And so on those components are it's very easy to switch because you have a test

• 38:50Speaker 01

  set and you basically change one component for another one. But it doesn't mean that it's generic because for instance like embeddings, can we change the embeddings? No, not at all. The embeddings is like forever. We have embeddings that the star space embeddings were trained in two thousand and nineteen and they are still in place and they will never be replaced.

• 39:10Speaker 03

  Wow.

• 39:11Speaker 01

  What we could, what we can do is to actually have more embeddings added to it, right? And we can have the old system and the new system using new embeddings and yet another system using new embeddings and then do an ensemble of all. But something that's fundamental of the embedding, you cannot change it. First of all, because the embedding depends on the data and the data is massive, right? To, to give a concrete number, for example, the content embedding, the dense vectors, which are based on, on transformers per, it takes us, we can do about like five hundred pages per second. It means that we pass a page and we get the embedding in two milliseconds. But then if you, if you work with embeddings or with transformers, you know that actually

• 39:56Speaker 01

  that's pretty impressive, right?

• 39:57Speaker 03

  Yeah, that's fast.

• 39:58Speaker 01

  That's fast. So in two milliseconds, like half fully optimized single GPU, eight, ten G, I guess, that takes one year and a half to do it for the whole data set. So even being extremely fast is, is linked to the data. So you cannot really do it, right? You have, you do it once and then you build narrower models that are based on those embeddings, on the output of those embeddings and you build heuristics that combine like your part numbers of your planting things. You have narrower models plus heuristics is you cannot change that, right? What you end up doing is you create like versions of it. You have embeddings V one, embeddings V two, embedding V two, and you keep adding. And adding is never a problem, right?

• 40:41Speaker 01

  Because at the end of the day, adding more embeddings is adding more features, adding more distances for the final G V D T that will do the final grinding. So anything that is linked to the data, at least at the scale of a search engine, you don't touch it. Anything that is stateless, like the N L M, that does a summary or that does a paraphrasing, then change it then, one day for the other.

• 41:06Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 41:12Nathan Labenz?

  So I get a couple of different follow-ups I want to chase down here. When you say like it can never change, do you mean that like outright literally? It seems if nothing else, as the world changes, you would need like a new embedding model to understand new developments, right?

• 41:29Speaker 01

  Yes, but when you say that you never change it, doesn't mean that you don't add, you can add new things. What I would then, what I tried to say is that why do you need to replace one embedding for another one? You actually want to have both, okay? Because most likely they are uncorrelated, right? And you want to actually build an ensemble of it. Why do you want to end up having like a query document distance? Why do you want to have a distance when you can have two? And then use both of them to actually do the kind of ranking. And best features is better, it's like an additive you don't replace, you just add, right? And then the cost of adding is just a matter of resources, right? Because I told you before, like it might take us like a month with like many GPUs and

• 42:13Speaker 01

  actually paying a lot of money to run one embedding second data set. That's not something that we might actually want to do it every three months, right? Perhaps it will take another year until we have yet another embedding type into the mix. Different products have different prospects, that's very important too. In your case, or like the case of the audience, if they have a thousand documents, or like 10,000, or a million documents, that's different prospects require different solutions.

• 42:46Nathan Labenz?

  Yeah, definitely. In the plumbing case, I can rerun the whole thing without too much trouble.

• 42:51Speaker 01

  But in any case, for, let me reiterate, it's better to have, to try 10 embeddings, different models, and use an ensemble than to try to find only one embedding.

• 43:07Speaker 01

  That does it perfectly.

• 43:08Nathan Labenz?

  And your ensemble approach is heuristic, right? It's not, or is that also a learned approach?

• 43:15Speaker 01

  I can learn that because at the end of the day, you have, it's like features. At the end of the day.

• 43:20Nathan Labenz?

  So it's essentially a mixture of experts, in a way.

• 43:23Speaker 01

  Of course, a mixture of experts is like another way. It's like another, like, is everybody that, mixture of experts is because ensemble is already like a, it's already used, but the mixture of experts is just an ensemble of eight different models, where two are selected at a time. It's exactly the same, the same way on your case. We did the best embedding that worked for my use case. I would use like five of them. And if they are uncorrelated, which is like the critical factor here, because if they are correlated, they are useful. They just, they don't add anything. But if they are uncorrelated, and if some of them are very semantic, others are more literal, others are like multilingual, others are English, all combined, probably will give you a better ranking than using only one.

• 44:08Nathan Labenz?

  So if I'm understanding the architecture you have today accurately, you have a number of different embeddings that have been built up over time, plus a learned combination of those that either, does it like choose which ones to do? Or does it weight them?

• 44:26Speaker 01

  No, weight them because.

• 44:28Nathan Labenz?

  Each one gets a scaler?

• 44:30Speaker 01

  We end up like having, it depends on which search engine, because we have multiple search engines, depending on what stage you are in. But you cannot have features, two hundred features, three hundred, and then you learn, boost to the addition three, and now you go.

• 44:46Nathan Labenz?

  Cool, okay. That's definitely interesting and good food for thought for mine and probably a lot of other projects too. On the, going back to the topic of upgrading the language model, you said it's like easy in the sense that you can easily swap one in for another. I imagine this is true for both search and the summarizer. It seems much harder to evaluate and actually determine which is the better result. So we've got obviously things like LMsys doing head to head, Elo style, Scale AI just came out with a new private thing that they're offering. How do you guys determine which, if you're going to make a change to search, how do you determine if it's an improvement or not?

• 45:27Nathan Labenz?

  And similarly for the summarizer, how do you go about assessing which is better?

• 45:34Speaker 01

  We do have a team of human assessors that their work is basically assessed manually. It's not the majority of the search team, but it's a significant chunk of it. And their job is basically like to have a homogeneous methodology and homogeneous criteria to assess. Because what we, what, one thing that we don't want to have is a heterogeneous assessment, right? So this team work is basically assessed of course on the either summaries, outputs, or rankings, or F Y G. So that's kind of the, our, in a way, our ground truth, right? Of course, like the productivity is what it is. They cannot produce a lot of assessments, but they are high, very high quality.

• 46:18Speaker 01

  And then, on the other hand, we have query logs. Our query logs are what we do collect from what discovery project so that allows us to compare what Bing or what Google or what Yandex does, right? So we use comparison with our competitors, right? And in the sense of N D C G or like just like comparing cardinality of intersection of the result set, right? So if for a certain type of queries, we do not match what Google does, then what we do? Like we check those results and then we manually assess those results, a sample of it, right? To see what happened. Either are we screening up? Or we just have a different criteria for this particular set of queries.

• 47:02Speaker 01

  So this is like the, the process. It's a combination of a lot of data from ourselves and competitors. Not a lot type, but in a way, watching what the others do. That's why also gens can resemble each other in a way, because we, we copy each other. It's not that we copy literally, but we end up using.

• 47:22Nathan Labenz?

  You're cross-referencing at least.

• 47:24Speaker 01

  Yeah, oh yes, we are cross-referencing, right? And if, if we do not return the same as, for example, if we have seen a query empirically in Google on the what discovery project, we also compare with ours. If we see like the intersection, that was in the flag. It's odd that this particular query happened, right? And then those queries are analyzed, it's added to the test set. And you keep building over time with a lot of patience and with a lot of effort. You end up like building data sets that have like thousands, hundreds of thousands of like high quality human assessment and hundreds of thousands, millions of automatic assessments, right? And those is what you use as as ground truth, like to evaluate, to see if we are going

• 48:08Speaker 01

  on the right direction or not. And last and not least, what we also have is like some of these queries, we're gonna keep keep them aside from the engineering team in a way. Uh, only the quality team knows them, they're to avoid overfitting, because it's like natural thing. E- even, fitting is actually a very big problem on on machine learning, because you can actually have overfitting even without knowing, just by like by keep fine tuning the the meta parameter that what you have. Yeah. The cross relationship and and all of it. So we do have also reset, and at the end of the day, it's uh it it's best effort. We are not that many people and we we try to do this like this gradient descent.

• 48:49Speaker 01

  If the results are not worse and performance is not worse, we just move ahead. And if the performance is worse, then we do, we have to do a judgment call, whether like we actually go for it and optimize later, or we we just cancel the whole release.

• 49:05Nathan Labenz?

  How does that differ, if at all, for the summarizer type experience that comes out of the, and just to set up the context there, that the summarizer is, and this is something that's both, you can experience it in the UI of search, but then also, we're gonna shift in a second and talk a little bit about the search API, and it's available via both. So this is basically, as you said, your rag style experience where, okay, there's all these results, but now can we deliver a a synthesis of these results that kind of gets delivered in paragraph explanatory form?

• 49:37Nathan Labenz?

  The challenge, another, an additional challenge I would imagine there is that there's probably a lot of like taste elements to what would even be considered good. Do you have a sort of principle? It's interesting, we've seen like the OpenAI models spec recently where they, for the first time, have really started to articulate, this is what we want the model to do. So now we can at least know if it's like doing what it's supposed to be doing, or if this is by design or not. But I imagine you have to make a bunch of similar choices, right? Do we want it to be short? Do we want it to be long? That probably varies on context. So do you have a constitution for the summarizer?

• 50:15Speaker 01

  Not, yes, but it's a constitution of one. It's like the person who actually does this and he tries to do the best. Of course, there is some guidelines that are like of good taste, right? We should not be like biased. We should be, try to be as concise as possible. There are like certain things that are like a common sense. But other than that, we do not have time enough to define guardrails or guidelines. We believe that it's more also an iterative process. The fact that we do certain fine tuning from time to time would help us that there are like certain results that we think that they are very good. The results will be used to like to fine tune the model. And the model should automatically learn what is the style that we want, right? And who decides what is good or not? The politics has been team.

• 51:00Speaker 01

  Again, we don't have any editorial criteria, that goes without saying, right? We don't play politics. We do not do any censorship or anything like this. But at the end of the day, it's a matter of personal taste of whoever does a rating. But that's, it goes for us, it goes for anyone else. It goes for you, right? Do you prepare the results from perplexity or from us or from Google? The information is actually the same. It's just a style.

• 51:27Nathan Labenz?

  Yeah, as a power user of all of these things, I have to say, I, I do find myself going to different services depending on the query. It seems like I've noticed, and I would say, I would, I think Brave is in the middle, I would say, right now between like perplexity gives me a lot of times the shortest, most to the point answer. You.com research mode is on the other extreme where it will give me like the two-page, thousand-word. Obviously, that is not suitable for every query. And I would, I don't know how you would have qualitatively described where Brave sits in relation to those, but for me, it seems like it would be in the, certainly in the middle of those two.

• 52:07Speaker 01

  But it's on the, it's on the middle because, because Brave has an extra complication compared to Perplicity, and we can discuss about that later. But it's a little bit like, it's all, it might sound very like unthoughtful, what I'm gonna say, but it's a little bit by chance. Lama 3 is much more verbose than Mixtron. It comes out of being more verbose, period. When we switched to Lama 3, we became more verbose than Mixtron. We, of course, we can turn it down a little bit, X, Y, Z, but it's certain inherent things that come out of the tools that you use. And in a way, we do not have the time or the will or know what is right in order for us to actually act upon. And then on those cases, the best thing to

• 52:50Speaker 01

  do is not try to do nothing. But it sounds stupid, but it's true. If you're gonna know what is the right thing to do, you just, I believe it as a model, as simple as that. Once I do have more data or I have more resources to do proper assessments and proper judgment, I will try to influence the model. But influencing just because someone on Twitter says something, which is very tempting. So that's one thing. And then on the, it's very interesting what you said about that you use a different system for different queries, because that's not what normal people do. Search is one-stop shop, right? And Brave Search is search. And when people search, they don't want to discover. People search because they have

• 53:35Speaker 01

  a problem, right? They want to go to Hotmail, or they want to go to porn, or they want to know like a recipe of mojito, because they are doing the mojito right now. They don't want to discover the two-page history of mojitos, or like a nice AI summary of mojito, right? They just want to know the ingredients and how to put it together quickly because you are in the middle of something, right? And that's important to realize that whenever we think of search, we always think of fancy questions or like complex questions, like unthoughtful. But most of the queries are not like this. And because we are like the default search engine on many countries on the browser, we need to cater To the common denominator, in a way. And that's actually something that is tricky, right?

• 54:20Speaker 01

  Because, but we could be much fancier if we had no traffic.

• 54:24Nathan Labenz?

  That mojito example reminds me of, it's almost a meme at this point, that all the recipe sites have like a long story at the top of the page now. And I'm sure you're familiar with this phenomenon, right? Why did that happen? Was that, is that something that, because I would agree, if I'm looking for a mojito recipe, I'm not really looking to read like your diary entry of the time that you made these mojitos and how much fun you had. And everybody seems to agree. So where did that come from?

• 54:55Speaker 01

  SEO, SEO. My random walk, that actually worked better. But once somebody learned that, the others copy. It's mostly SEO.

• 55:04Nathan Labenz?

  But why did it work for SEO? Is that a, is that sort of a misgeneralization on Google's part? There must be something where like more content of this sort is useful. And then maybe they overdid it and started to return recipes based on that criteria, even though it's worse.

• 55:22Speaker 01

  Now you, how to say that? Do not, never underestimate the, that things can actually happen by accident. So if you do a page that has very good information, but you do it in a format that the parser of Google doesn't like, your content is not going to be indexed in the same way that if you do it on free text, where the parser for sure gets it, right? So what you end up doing, again, notice the writer of things that want to be SEO optimized, what they do is an ensemble approach. They try many different things to see what is works better against the algorithm. And the algorithm might not be something that is really, has a strong rationale behind it.

• 56:05Speaker 01

  It could be that certain types of structure is penalized by accident. Just by accident, because that particular thing is not properly parsed. Or because that other thing is like between this stack and this stack, made it to the list of things that has an extra boost. And those, according to the leak, Google has 14,000 features.

• 56:28Speaker 01

  Per page, I find it a little bit too much, but we have hundreds, so I wouldn't say they have thousand.

• 56:36Nathan Labenz?

  So how do you think this will evolve as we go into a future of just language model generated content everywhere? There's a lot of talk right now about the web is sort of, the hyperbolic take would be like, the web is over, it's all going to be overrun by L L M spam, and also people don't want to necessarily be included in training data, so walls are going up in some places. How do you expect the web to change in general and specifically due to those factors?

• 57:09Speaker 01

  I mean, the web has been changing for a long time, right? The upcoming of advertisement was a change, right? The ad blocking has also a change, so everything that is like a change of incentives on the web, right? And the fact that with A I, not only the economics, which are always like pretty, even the reputational aspect of the web might actually go away, right? Because if nobody visits your site, you have your podcast, I guess you make money out of it, but also there's some reputational factor to it, right? If you lose both money and reputation, is what, why should I do, why should I do condom for, right? It's worrisome to say the least.

• 57:50Speaker 01

  I didn't know what will happen, but certainly the web will change. And I'm a little pessimistic on, on, on it, not only because I like the web, right? I like the web, I like, I'm old enough that I prefer YouTube over Netflix. Even YouTube is full of crap, but it's also full of gems, right? And I like more YouTube than Netflix, I like more MySpace than Facebook, even though, because I like this variety, and I think that's like, the A I is not the one who's gonna kill it, but is yet another turn towards, towards like a version of the web that is not what I had when I was younger. That doesn't mean that we are worse off as collectively, not necessarily,

• 58:35Speaker 01

  but I mean, A I, the, the, A I answers, in a way, they are very convenient. And that's why Google is forced to implement them. And in a way, that's why also we are forced to implement them. Remember that for us, it's not cheap to run summarizer. Right? Because we do have, we serve about twenty seven million per day, thirty million. Ten percent, thirty percent of that to be run through an NLM. That's not, that's actually fair, a lot of price, right? So we are like not forced, but there is pressure to it because people find it very convenient. But anything that is convenient has a, has a price. You sacrifice something for convenience, that's given.

• 59:20Speaker 01

  On the other extreme, I don't want to like, I don't want to go to the river to fetch water, right? I want water to come out of my tub. So, but how much, but that's kind of, what the threshold is, I do not know. But anyway, convenience is is king and, and answers with AI are here to stay. They are actually very useful and very powerful and, and very convenient too. But of course, it will pay a price. Not only in the ecosystem, but on, also the variety of, of answers that we get. Because I think that some of our competitors like that answers are knowledge. No, answers are not knowledge. Questions are knowledge.

• 1:00:05Speaker 01

  Answers are just answers, right? So the fact that you have an oracle kind of type of thing that for anything you ask, it, it will give you this. That basically means that everything else does not exist. And that's dangerous.

• 1:00:20Nathan Labenz?

  Yeah, it does seem like it's going to be a tough time for a lot of the content creators. I'll continue to do the podcast just so I get to talk to leading thinkers like yourself, regardless of any, whether anyone listens, but it does seem like there's a lot of people out there who are.

• 1:00:36Speaker 01

  Oh, and I don't know, like OpenAI has, because they have like big pockets they try to pay, but I don't know if the economics actually are because they are kind of economics of the advertisement, right? Yes, Google makes a lot of money, right? But if you have to split that money equally among all the parts, everybody gets, gets something that is not enough to live.

• 1:00:56Nathan Labenz?

  Yeah.

• 1:00:57Speaker 01

  We've been, by searching for endless convenience, we commoditized ourselves.

• 1:01:05Speaker 01

  I don't know what will happen, honestly.

• 1:01:07Nathan Labenz?

  The business side of this work for Brave. I'm just doing a little back of the envelope math and I'm imagining, okay, geez, if you're doing thirty million queries a day, and I don't know how much, how many tokens you're putting in per query, but we can get into a little bit more specifically like how much data is returned. But it's not an insignificant amount. It's got to be thousands of tokens per query.

• 1:01:32Speaker 01

  It's four thousand tokens per query usually. We usually max out our context window, which is like four thousand. But it's also, it's all self-hosted. We do not use any third party. Not only, not because of the independence part, which would also be because we value that we are independent, but also like cost, right? We would not be able to use OpenAI or do this cost wise. So everything that we do is like self-hosted. And we tell me that we have to pay machines. I think that nobody will complain if I say that. But for instance, right now, I think that we have twenty eight hundreds out of Amazon. So that's our infrastructure for the big LLM,

• 1:02:14Speaker 01

  plus a lot of like GPUs for a small model of Q and As and classifiers. So it's a big infrastructure, but it's not as massive as many people think.

• 1:02:26Nathan Labenz?

  So I guess for now, it's not so expensive that it forces a business model change. Because this is like both, this is something Google's obviously going to be facing in a big way too, right? Like, it's very unclear how, even if you're doing like pay per click advertising within search, it's very unclear how that fits in with these AI answers. And then more broadly, like, it's just very unclear how to, do you guys think of like a subscription model in the future for something like this? Or do you have any idea for how this evolves business-wise?

• 1:02:57Speaker 01

  It depends. If you enter into the chatbot domain, what you have for back and forth that is transcript. Then the economics might work differently, right? But if it's on the context of search, where like the this query, it may be query reformulation, but there is only one, one trip. The costs, let's say that worst scenario is out of the costs of infrastructure. But people tend to overdesert the new things over the old ones. Yes, running an L L M is very expensive, no question.

• 1:03:27Nathan Labenz?

  But it's not like a search engine is super cheap to be good at.

• 1:03:30Speaker 01

  That's my point, right? If Google doubled infrastructure, I'm sure they can do any N L M that they want without any issue. And they suddenly have the margins to double infrastructure. I think that's a cost problem, it's more like a, well, we want to do that problem. Whether people actually want to have that, it's very unclear that the majority of people want to have that. With search is very, again, let me go back to the point I said before, like that once you have traffic and you like serve all use cases, right? People are not, when they search is they have a problem, they don't want to be stopped, right? Because they are in the middle of something, right? And people do not usually like remember good cases, they only remember bad ones.

• 1:04:11Speaker 01

  So you, you don't go to that particular shop because they treat you wrong once. It's not, you go to the other one because they, on average, they treat you right most of the times. You avoid, you don't reward the place, you basically punish the one that, that you don't like, right? So if you start to do certain experiments, if you have a lot of users, you risk beating a lot of users because they don't like what you're offering, right? Because it's in the way, right? You're annoying them. And that's why, again, without traffic, you can do many things. Or with a very small traffic, or with a traffic that is of a very particular segment of population, you can do many things.

• 1:04:56Speaker 01

  But once you have the general public and, and your service is like running water, you open it up and please, you cannot, you don't have the luxury of doing a lot of innovation there. And actually, Google is suffering, I pity them, honestly, in the sense that it's a very tricky thing what they have to, to do. And we have a much easier problem than Google has to react to L L Ms. And you dot com or perplexity or, or taggy have a much better position than we, where we are, right? Because they have. Less commitment. So like a blanket slate for the win. It's always a game, right?

• 1:05:39Nathan Labenz?

  What's your user base look like? I know that obviously we're speaking in English, but you're from Spain and others are, the company's headquartered in Europe, correct?

• 1:05:50Speaker 01

  No, Brave is headquartered in the US. Search is 100% Europe. There is like only, there's people, only people in Europe and some people in Africa. All 10, 10 times them. But all search is EU, not EU, Europe based. But Brave is an American company, but it's like heavily, it's heavily remote, like how the people.

• 1:06:13Nathan Labenz?

  Maybe a more general purpose question would be, again, where are Brave users? And I wonder if you see differences cross culturally in how people are responding to the AI moment? Like, how would you characterize what, how, how consumers are reacting to AI experiences in Europe as opposed to in the United States?

• 1:06:36Speaker 01

  No, we don't, we have not done this analysis. Brave search users is actually, is a good sample of the global population. About like 30% of the people are in the US. The next big market is already in Europe. So we are like pretty well spread across all continents. There's no, there's no odd countries or odd behaviors. It's large enough to be statistically,

• 1:07:03Nathan Labenz?

  My intuition would be that I'd maybe be more interested in the summary, just because less, fewer of the sites are mobile optimized and it's just more hassle to browse around.

• 1:07:14Speaker 01

  Could be. But at the same, at the same time, when you're on desktop, you also would expect to have more world-related difficult queries. On mobile queries, usually they are very instrumental. They are very like fun number of restaurant.

• 1:07:29Nathan Labenz?

  Yeah.

• 1:07:30Speaker 01

  Or like a fun fact, again, Age of Lady Gaga. If you're like playing a trivia or something with friends.

• 1:07:38Nathan Labenz?

  Let's maybe turn to the search API. I mean, this is an area of the business where there is a very natural business model. And this has been like the main subject of, of the ad as well that we've been running.

• 1:07:51Nathan Labenz?

  Went ahead and created a little Google Collab Notebook that allows a user to easily show up and try the Brave API as compared to the Google Custom Search API and then also the Bing Search API. And a few observations that I would be happy to share, but maybe you should start by just telling us like, how do you see the product as being differentiated from the other options that probably a lot of people are more familiar with?

• 1:08:20Speaker 01

  Yeah, the reason why we had, we released a search API to provide an alternative to big tech. As far as we know, there is only two APIs out there that do search engine, like Google and Microsoft. And Google is not public. I think you, you use it, the custom search from Google, that's not meant to actually be used in production. I think they have a cap of 10,000 queries. So if you actually want to use Google API for something real, you actually have to go to talk to them and they might give you access or not. It's their own, it's their own decision. On the other hand, it's a little bit better in that respect because you have a public API, of course you pay, but it's powerful.

• 1:09:04Speaker 01

  You go there, you accept the terms of service, which is a bit draconian, but that's a different topic. But you pay and then you use, that's it. And at Brave Search, what we try to do is like, same as Bing API, it's a public, public API. You pay, you can use it, no questions asked. We try to be less draconian on the terms of service, right? So we do not require attribution. We allow for you to mix your results the way you want, things that like Brave would not, sorry, that Bing would not allow. But other than that, again, there is not, there is like nothing else other than before you only had Microsoft, now you have.

• 1:09:46Speaker 01

  Brave and Microsoft. That's not a small thing. And we did that like the very moment we, the very moment we actually could run the API, we did, right? Because we're not like a hundred percent independent all the way, right? When we launched Brave Search, we were able to answer like eighty seven percent of the queries, right? For the remainder thirty percent, we actually used Bing. And we skipped that over the month, right? So we went like from eighty seven. A few months after was ninety, ninety-two, ninety-three, ninety-four. And a little more like a year ago, we like went from the ninety-four, no, ninety-six or ninety-seven.

• 1:10:27Speaker 01

  We went back to there to be totally independent from Bing. And at that moment, we could offer the API, right? Because before it was not possible because of terms of service of Bing.

• 1:10:38Nathan Labenz?

  Yeah, that's interesting. I would agree definitely with the idea that the Google Custom Search API is not really a competitor.

• 1:10:48Speaker 01

  It's meant to do side search.

• 1:10:51Nathan Labenz?

  Like your own site.

• 1:10:51Speaker 01

  Yeah, actually, I'm pretty sure that if they, you can do the trick and not pass domains. So that it doesn't do side search, it does global search. But I'm pretty sure that it goes again to terms of service. But in any case, it's limited to ten, to ten thousand queries per day, which is.

• 1:11:07Nathan Labenz?

  The thing that stood out to me the most, and folks can go judge this for themselves with our collab notebook, which I'll put in the show notes, is really, more than anything else, just more information being returned. I, in my case with Waymark, for example, we have this flow where a new user shows up. And I'm always amazed like that so few products implement something like this because it's worked really well for us. New user shows up, they want to make a video for their business. They typically don't have a ready to go folder of all their assets, let alone like a good profile of their business, right? So now we're going to use generative AI to.

• 1:11:51Nathan Labenz?

  Put things together for them, but we need to know who are you? Tell us about your business. What do you offer? What makes you special? Do you have any images that we can feature? And people just don't have that stuff. So most products just require you to make a profile and load your images and whatever. We have worked pretty hard over the years to create a good experience where we go grab that content off the web for you. And as you'd imagine, the generative AI era here has really improved how well that works. We have used Google custom search for that. Our volume is not super high and we're mostly just trying to get like their homepage or kind of then we'll maybe bounce a few links off of that. But I am going to look more deeply into using Brave for this use case in particular because

• 1:12:36Nathan Labenz?

  just more information really jumps out at me. We get like a snippet from kind of the Google custom search and with the Brave results, you get like often a much deeper cut with multiple snippets and even like some structured data for some results. I looked at this like a few different ways, but in just raw characters returned for a 10 result query, the volume of data was often like a multiple higher than what I was getting with Bing and Google. And that was, that also extends down. You could fill out characters with anything, right? But that, a lot of that seems to come from these additional snippets, which for us are really useful.

• 1:13:21Nathan Labenz?

  We right now first run that query and then we go crawl ourselves and then we pull all that stuff in and then we run our own language model to process it and turn it into a profile that we can then feed downstream into the actual generation task. But I could definitely see us simplifying a lot of things and that's always nice to take multiple steps and condense them into one. If we just. Could get that multiple snippets alone, I think, would be a pretty notable enhancement. That's before the summarizer too, which is something I also could see us doing. Because again, we're doing that ourselves today. And you've implemented that whole thing. I'll have to check on like the the coverage for our small businesses because some of our

• 1:14:06Nathan Labenz?

  customers are pretty long tail. They're generally not super famous businesses, but.

• 1:14:11Speaker 01

  You have to think that you get the right business at the end.

• 1:14:15Nathan Labenz?

  But for me, the a volume of information was the biggest thing that jumped out. Would you expand on that anymore?

• 1:14:23Speaker 01

  Yeah, but basically the main motivation to provide, we did not have the motivation to provide more volume per se, right? Because again, remember that the amount of data is meaningless. Sorry, the amount of data is very important as long as there is no noise. Because if you actually just the risk of adding noise precludes any amount of data that you can provide. So our rationale was the following, it's like we, because we do the dynamic snippet generation pretty depending, right? We end up having multiple candidates that we run through a Q and A, a Q and A model. There's a cost that we already paid. So as long as that is above a certain threshold, which means that in principle, there should

• 1:15:04Speaker 01

  not be noise, I feel why we should not send it to the user so that they can, they can have the choice to decide which snippet they want to show. Or in the case that they want to do LLM inference, to have a little bit more data to, to fit the prompt or to cherry pick. Because remember, cherry picking is actually very important. And on the topic of summarizer, the secret sauce of our summarizer is not the model, the LLM. Again, we changed it from MIXTRAL to LAMA in a heartbeat, is cherry picking of which data we fit the prompt. And like, that's what we spent the time on. Like only feeding like non-repeated, non-prolonged information.

• 1:15:48Speaker 01

  So we do a lot of preprocessing on that. Anyway, so hopefully like on the API, what we try to offer is we give you like all the elements so that you can innovate on top of, on top of it. That's like the, our goal. Of course, we want to make money out of it too, right? So if it has more value than being, also, it's also good because like we will have more users. But anyways, like. It's not like one or the other. They have both reasons of the same back, right?

• 1:16:17Nathan Labenz?

  It is notably significantly cheaper than Bing as well. You have multiple different tiers of service for different amounts of data that get returned and somewhat different like price points for different licensing as well. And there's whether you're going to be able to store the data or not. So folks can get into the different price points. But Bing also has a lot of different price points. And at least for me, when I was looking at what I would naturally want to use, including the summarizer, that is a $9 per 1000 queries price point with Brave. And with Bing, I would say the most comparable thing, they don't even really have that summarizer feature at all, but the most comparable price point that returns like the varied

• 1:17:00Nathan Labenz?

  results, where you might get like video results or different kinds of discussion type pages, is a $25 per 1000 price point. So saving more than 60%. And I would say getting more, both in terms of the additional snippets, the extra information and the ability to tap into that summarizer. Anything else that I should be conscious of as I think about the right option for my app?

• 1:17:27Speaker 01

  No, the right option for your app is like whatever works. And if you can afford all data combined and do an example out of it. But no, it's like, I mean, we are very happy with the search API. And it's not just, it's one of the, it's actually, it's a satisfaction from us. When like we talk to a customer and they, and they say that, well, it's about time that there was an alternative to Bing.

• 1:17:54Speaker 01

  And that's actually for us is very, it's like very rewarding to be able to achieve that. That that goal to, you might use Bing, you might use Google on our private deal. You might use whatever, right? But like the fact that you have one more to choose, that's a big deal for us.

• 1:18:12Nathan Labenz?

  You want to shout out any customers that are using the search API that folks can go check out if they want to see a good implementation?

• 1:18:21Speaker 01

  Ah, we, I wouldn't be able, I wouldn't be able to play favorites. Because actually, there are two things here. There is, the vast majority of our customers do not allow us to mention them. It's a very competitive space. Many of the big players do not want to. Admit or let others know that they have or that they use a search or any other system. And then for the one who actually acknowledged, then it would be kind of fair to single out one. But I can tell you like some nice use cases, of course, there's like plenty of RAC use cases that are being done by names that have been mentioned before.

• 1:19:05Speaker 01

  It has also been used by to train significant foundational models. One use case that I find very interesting is that it's also being used to do grounding of LLM outputs. LLM output, then you actually run it, you submit it, you run it against a search engine to try to verify whether the output is legitimate or not. So there is like a lot of, and many other use cases that we do not know. Or because at the end of the day, we do not, unless they ping us, we do not ask questions, right? It's use it and hopefully it's been beneficial for both of us.

• 1:19:43Nathan Labenz?

  Are there any favorite technology compliments that you would? Almost like in a cookbook or demo showcase sort of way, are there things that? Patterns of development that you've seen where people have combined the Brave Search API with some other framework, technology, user experience that you think is worth sharing just to inspire people?

• 1:20:05Speaker 01

  Yeah, there is like this, I don't know how old this is. There was one guy who actually did a YouTube video and I think it is on GitHub too. They like to build a perplexity clone using Langchain, Langchain, Brave, Grok. And so he did like a whole convolution of APIs and he actually built like a very nice prototype. I think it's good to get the idea of like why the things sit together. And then people can decide what to spit out to goalkeeper.

• 1:20:36Nathan Labenz?

  Cool. You've been very generous with your time. Anything else that we haven't touched on that you want to make sure we cover?

• 1:20:43Speaker 01

  No, I think that we, I think that I really talk too much. So hopefully I did not bore your audience with my ramblings. But it's been, it's been a pleasure, Matthew. Very nice host. I appreciate the chat.

• 1:20:56Nathan Labenz?

  Cool. The feeling is mutual. Joseph Pujol, thank you for being part of the cognitive revolution.

• 1:21:03Speaker 01

  Take care. Bye bye.

• 1:21:04Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.