Write, Change, Recall, Forget: MongoDB's Pete Johnson on How Retrieval Drives Agent Performance

Nathan's guest this episode is Pete Johnson, Field CTO of AI at MongoDB, and the conversation is really two conversations woven together: a history of database architecture, and a status report on the still-unsolved problem of agent memory. Pete opens with a framing device that recurs throughout — he was born in February 1970, four months before E. F. Codd's original relational-model paper that gave rise to SQL. The relational model, he explains, was built for a world where storage was the scarce resource, so normalization — splitting data across linked tables to avoid duplication — was the rational design choice. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/write-change-recall-forget-mongodb-s-pete-johnson-on-how-retrieval-drives-agent-performance/ Sponsors: Mercury: Mercury is the banking platform loved by 300,000+ entrepreneurs, with virtual cards and Spend controls for granular budgets, receipts, and low-risk AI agent purchases. Learn more and apply in minutes at https://mercury.com Granola: Granola is an AI-powered notepad that securely transcribes meetings and turns rough notes into clean, structured action items. Try it free at https://granola.ai/tcr Diffusion: Diffusion helps organizations build custom AI software factories that scale business outcomes, not just outputs. Cognitive Revolution listeners get a 25% service credit on their first engagement at https://diffusion.io/tcr Deepgram Flux TTS: Deepgram Flux TTS brings lifelike AI voices with real personalities that handle interruptions, pauses, and natural conversation. Try all the voices free through September 12 at https://deepgram.com/keep-talking Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (03:15) Sponsor: Mercury (04:56) SQL versus NoSQL (11:24) Enterprise database choices (Part 1) (18:30) Sponsors: Granola | Diffusion (21:27) Enterprise database choices (Part 2) (21:27) Schema flexible search (33:50) Contextualized chunking tradeoffs (Part 1) (35:12) Sponsors: Deepgram Flux TTS | Claude (37:17) Contextualized chunking tradeoffs (Part 2) (46:22) Retrieval quality thresholds (53:17) Agent memory systems (01:05:51) Enterprise AI deployment (01:16:32) Voyage acquisition strategy (01:23:38) Global AI adoption (01:30:56) Episode Outro (01:34:41) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, my guest is Pete Johnson, a 30-year technology veteran now serving as Field CTO of AI at MongoDB. We start with a brief history of database technology, going back to the 1970 introduction of SQL and unpacking how the relative scarcity of disk space at that time informed data normalization as a design principle. And then how decades of Moore's Law and the resulting evolution of constraints ultimately led to the creation of MongoDB in 2007. This history puts today's acceleration into stark relief. As Pete recounts, in a remarkably short period of time, we've gone from being effectively

• 0:45Nathan Labenz?

  forced to implement RAG pipelines by the very limited context windows of GPT-4 class models to RAG is dead when we first got million token contexts, to the brief token maxing window and subsequent budgetary fallout, to today, where RAG is again a major enterprise priority as company data is increasingly available for AI systems to use. Usage itself is scaling and naively maxing out the context window costs multiple dollars each and every time. Pete's key point above all is that agent performance and especially cost-adjusted agent performance depends heavily on effective retrieval. With this in mind, MongoDB has continued to ship the improvements that developers

• 1:30Nathan Labenz?

  need to take full advantage of AI. Vector search, sorting methods which combine traditional and vector search results into a single result set, a rerank tool called Dollar Rerank, and their own embedding models powered by the acquisition of Voyage AI, which have some pretty cool features like a shared embedding space across model sizes and a Matryoshka data structure. Interestingly, while MongoDB customers are free to use whatever embedding models they like, Pete insists that embeddings are contrary to popular belief, not yet commoditized. From there, we go on to talk about how developers are using these tools to build memory systems,

• 2:13Nathan Labenz?

  the write, change, recall, forget loop that Pete sees as the general pattern, and why forgetting is currently the hardest part. For now, Pete says it's important to remember that we're only 18 months or so into building AI agents, and as such, there is no established right answer and nothing like a lamp stack for AI that enterprises can confidently buy and deploy without meaningful customization. We also touch on the lessons he's learned from talking to customers in India, Brazil, Mexico, and Europe all just this year. And to my surprise, he reports that the most advanced companies he's spoken to in 2026 were in fact outside of the United States.

• 2:56Nathan Labenz?

  Without further ado, I hope you enjoy this speed run through database history and high level AI readiness strategy for enterprises and survey of the still fast evolving frontier of agent memory with Pete Johnson, Field CTO of AI at MongoDB. The Cognitive Revolution is brought to you by Mercury, the banking platform loved by over 300,000 entrepreneurs. I use Mercury's virtual cards, which make it super easy to set limits, expiration dates, category, and even merchant-specific spending controls to give my more autonomous AI agents, Aid and Clay, the ability to buy and test products.

• 3:37Nathan Labenz?

  Recently, I asked if they could find a good way to split an AI-generated image into layers, separating the text from the background and so on. Two of the products they found were behind paywalls, but using their Mercury virtual card, which is limited to SaaS purchases only, they bought a month's subscription, tested the products, allowed me to review the results, and then canceled the stuff we didn't need, all with functionally zero risk to me. This is already really powerful. And now with Spend, Mercury is making it possible to run an entire company's spending with the same level of ease and control. With Spend, you can set granular budgets for every team, person, and all the agents you like.

• 4:20Nathan Labenz?

  Plus, you can process receipts automatically and even temporarily auto-lock people's cards if there are ever any issues. The future of spending money is dynamic but controlled. So join me in the future of banking. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. The IO card is issued by Patriot Bank, NA, member FDIC pursuant to a license from Mastercard International, Incorporated. Pete Johnson, Field CTO of AI at MongoDB. Welcome to the Cognitive Revolution.

• 5:03Speaker 01

  Thanks so much for having me, Nathan. I appreciate it.

• 5:06Nathan Labenz?

  I'm excited for this conversation. I've been a database guy in the past, and now we're all kind of everything guys these days with our agents greasing the wheels for us. But the, I think one of the big themes that we're going to get into is the quality of data is still critically important to making things work. And I'm excited to learn everything I can from you over the course of the next hour or so. Maybe just for quick starters, I remember first encountering MongoDB like 15 years ago, maybe, when a young Guillermo Rousch, who's now the CEO of Vercel, was showing it off in the context of what at that time was like seemingly like crazy futuristic

• 5:50Nathan Labenz?

  Node.js applications, like, wow, you can make JavaScript do these kinds of things. And there's a wait, there's a database that like works in this different way where I'm not constrained by the schema that I already defined. Like, this was mind-blowing stuff. I'm not sure, we've got a lot of technical people in the audience that say AI engineer is kind of our number one profile, but I'm not sure everybody knows kind of the broad lay of the land when it comes to database technology. So I thought maybe it would be helpful just to first kind of contextualize SQL versus NoSQL, schema versus schemaless. Give us kind of the high-level view so we know where MongoDB sits in this obviously bigger database space.

• 6:32Speaker 01

  Sure. I like to think of this in terms of the history, in terms of the different kinds of problems we've had to try to solve with software over the last 60 years or so. And where it really starts for me is in June of 1970 is when E.F. Cod, who is an IBM researcher, wrote the original white paper that gave birth to SQL. So that was June of 1970. I happen to have been born in February of 1970, so I am as old as SQL, Nathan. And if you think about the kinds of applications we were building at the time as an industry and the kinds of hardware we had at our disposal at the time, the applications were very departmental in their nature, right?

• 7:16Speaker 01

  Not every, not every company had a computer. And those that did, the people that were using it were often on the same floor or certainly in the same building. They were working nine to five hours. It was okay to have downtime on the weekend. And if you think about the big three pieces of hardware at the time, memory, compute, and storage, storage was by far the most expensive of those three things in 1970. So if you think about how you might organize data, keeping in mind that the scarce resource was storage, you would of course do it in such a way that it's now known as normalization. So you may have in university or in coding school, you may have learned things like first

• 7:59Speaker 01

  and third normal form, all about trying to lay out data on disk in the most efficient way so that you aren't storing any piece of data more than once. The simplest example I can give with that might be my wife and I share an address and maybe with a retailer might want to know what our address is. There's two people at that address. You might have in one table, the list of people. In another table, you might have the addresses. You might have a third table that links the people to the address so that you're only storing that address one time. And that's the most disk-efficient way to do that with, like I said, some kind of normalization.

• 8:39Speaker 01

  Well, fast forward to 2007, which in October of 2007 is when the first commit was made to MongoDB. And the world is a very different place in 2007. The world has internet, the world has cloud, the world has mobile devices. iPhone came out in the US for the first time that year. So it's a very different world. And the scarce resource in that world, because we had 47 years of Moore's Law between those two dates, the scarce resource in that ecosystem became time. Because the kinds of applications that we would write that were in the cloud or that we would write for mobile devices, you could not have downtime on the weekends. It was not people who were confined to the same building. It was not nine to five use cases.

• 9:23Speaker 01

  So it was a very different world that required a very different kind of database. And that's what gave rise to NoSQL. Now, our flavor of NoSQL, which like I said, that first commit was in October of 2007, uses it as its core JSON. So you've mentioned the stuff with JavaScript. That's why that was so popular 15 years ago or so is because what you get back from your API call is JSON. We're storing it in transit as JSON. We're storing it on disk. And a binary form of JSON we call BSON. And that's part of why we're so fast is because we're keeping that format the whole time so that when you serialize it off a disk, when it goes through our servers, when it

• 10:06Speaker 01

  ends up in your client, it's essentially staying in the same format that whole time. So if you think about the kinds of applications that we've been building since 2007, like I said, disk space is no longer the scarce resource. If you take that address example, if you denormalize that data just a little bit, if you instead of storing it across three tables, if you have a JSON structure that has my name and my address, and then a second one that's got my wife's name and the same address, you're duplicating the address, but what do you get in return? What you get in return is one disk read. So that's why it's faster. So instead of having to do three disk reads to get that same data, you can now do it in one disk read.

• 10:50Speaker 01

  Does that mean that every problem is solved by a NoSQL database or by MongoDB in particular? No. But it also doesn't mean that every problem has to be solved with SQL. And that's how that sort of evolved out of, SQL was invented at a time where the problems were slightly different, and a lot of those problems still exist. SQL is a certainly valid solution to lots of different problems that we still have. But when you need speed, when you need scale, and increasingly, when you need better retrieval quality, that's really where MongoDB shines.

• 11:25Nathan Labenz?

  Reminded of the old saying that the problem has faded, but the solution persists.

• 11:31Speaker 01

  Yes.

• 11:33Nathan Labenz?

  There's, yeah, see that in a lot of parts of life these days. Tell me a little bit more, you were just kind of getting to it, but tell me a little bit more about, I mean, Mongo is a pretty big company, $30 billion market cap, obviously has made serious inroads into some of the biggest companies in the world as customers. Give me kind of the next level of detail in terms of before the AI moment, and then obviously we'll get into that in depth. What are the kind of big decision factors that would push an enterprise to go one way or the other with their database decision? And how are they, do they typically do it on like an application by application basis these days?

• 12:14Nathan Labenz?

  Or do big companies feel that they need to have a single kind of consistent database technology for everything they're doing?

• 12:23Speaker 01

  Sure. It's a mix. Like you just said, we're in like 75% of the Fortune 500. But if you look at the overall market, you mentioned our market cap, we're about 2.5 billion is what we ended our revenue, that was our revenue number for our fiscal 26. We're currently in the middle of our fiscal 27. And that's about 2%, 3% of the 100 to 110 billion dollar database market. Where most enterprises choose us is those first two things that I mentioned, when you need it fast or when you need it at a higher scale. Because you're not storing disk, you're not storing that data across multiple places of disk, that's how you get the speed, that's how you can get better scale if you're, if you're storing, if you're going to retrieve JSON, because every modern programming language

• 13:07Speaker 01

  can serialize JSON into some object format so that you could use it immediately in your business logic. And that's the other aspect of time. It's not just wall time for the transaction, but it's also the developer time of what do I have to do with this WOD that I get back from my API call and then make some kind of useful business decision out of it. But what we see is right solution for the right problem. And increasingly, what AI has done is it's opened up the number of problems that were a better foot fit for than a traditional SQL approach, especially those at higher scale. And that's why I mentioned the three things, the speed, the scale, the retrieval quality has really become an important factor.

• 13:47Speaker 01

  It continues to, especially later here in 2026, as you start to see people strategize around token maxing to evolve different kinds of agentic architectures in ways that we didn't see in the first half of 2026. Retrieval quality has become more important for AI, but it does end up being a mix. It depends on the corporation. Some of them will take it workload by workload. And for this workload, we'll use Mongo and for a different workload, we'll use SQL. Others are taking an all in strategy. It really varies based on, in some cases, how much greenfield versus brownfield do you have in your portfolio? In other cases, it's where's my talents already? I was just at, for a major automotive company here in North America, I was just at a hackathon

• 14:32Speaker 01

  where we trained up like 250 engineers and turned them loose for three days on a hackathon and they've got some new product ideas that fell out of that. So there's, there's no single right answer, right? There's lots of ways to sort of walk through that forest. But at the end of the day, it does come down to right problem for, a right solution for the right problem. And where we typically help people is with those modern applications that need that speed at scale and those gigantic use cases that increasingly need that better retrieval quality.

• 15:02Nathan Labenz?

  I'm really interested, though, in big three tech modernization, and I have a bit of a brush with it from time to time, just enough to know that there's a lot of appetite, but there's also a lot of friction. What was your experience of, and I'm sure a lot of these app concepts that people were working on are AI apps, right? Without naming names, what more can you tell us about the kind of gonzo on the ground experience in the big three?

• 15:30Speaker 01

  Some of it was customer experience kind of applications. Others was how can we make the dealers more efficient? How can we make the mechanics more efficient? Some of it was how can we make the in-car experience more efficient? It really spanned the gambit of different types of use cases that a big three automaker might have. But really, it's, I mean, I'll go as far as to say, like, our education system has a bias, a historical bias that thou shalt always normalize. And for a lot of use cases, that's valid, but there are an increasing number of use cases where these big three that I just mentioned matter. And increasingly, that retrieval quality, one, is becoming a much bigger deal for agentic

• 16:12Speaker 01

  architectures as you start to try to take better advantage of the context window than just stuffing it with a million tokens every time. And we can get into some of the details as to why we're a good fit for that and why that's emerged, like in the wake of what happened to Uber earlier in the year with them burning through their token budget for 2026 and 13 weeks. That grabbed a lot of headlines and there's been similar stories about where like maybe token maxing wasn't such a great idea to begin with.

• 16:38Nathan Labenz?

  Yeah, it was a phase. We've all, we've been, we've all been through our phases. I was really attracted to the schemaless database model way back when because I'm just kind of a free flowing person who loves to just make a mess now and tell myself I'll clean it up later. And so this is one of the things that I'm really interested in, in terms of how DB interacts with AI applications. Because it seems like on the one hand, it's more tempting than ever to do that, right? It's more valuable than ever to just be like, let me just dump all the data in here and like maybe an AI will make use of it sometime later. But then you do have interesting challenges downstream of that.

• 17:22Nathan Labenz?

  One with your example of like the two records with the same address, now you got to update it in two places. So that's something to keep track of, obviously. I'm also kind of struck that like the advantage of latency, you may have a different take on this, but it feels to me like the advantage of latency is somewhat reduced because now like everything gets put through a slow step anyway. And so the ability to like shave a little time off the DB call when I'm then going to have to wait for tokens maybe matters less. And then I'm also thinking like, if I do start just dropping all my data payloads into a sort of extra field, now I'm going to be doing like exactly what you're saying I shouldn't be doing in terms of just bringing a million tokens into context all the time.

• 18:07Nathan Labenz?

  So take that in whatever direction you will. What do you see as kind of the biggest trends in application development that have bearing on your database choices as we are AI-ifying everything?

• 18:25Speaker 01

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:29Nathan Labenz?

  Today's episode is brought to you by Granola, the AI-powered notepad built for the way real people actually meet. Here's how it works. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup and no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word and still walk away knowing exactly what was decided, who's doing what, and what comes next.

• 19:15Nathan Labenz?

  When I had Granola co-founder Sam Stevenson on the show earlier this year, he explained how Granola aims to provide a calming experience for people with crazy workdays. And as a user of the app myself, I have been struck by how streamlined, even minimalist, the Granola product experience is. That takes real discipline, but the result is a product that works not just for AI early adopters, but diverse teams of people who just want to get things done more efficiently and effectively. Listen to my full episode with Granola co-founder Sam Stevenson for a masterclass in designing AI products for mass market adoption. And try Granola for free at granola.ai/tcr. That's granola.ai/tcr.

• 19:59Nathan Labenz?

  The cognitive revolution is brought to you by Diffusion, the AI transformation specialists that help organizations from traditional SaaS businesses to defense companies to nonprofits. Build software factories that can scale not just outputs, but business outcomes. You probably know that the majority of enterprise AI projects fail. In general, that's because leadership fails to realize that AI isn't like traditional software that you can just buy and install. On the contrary, if you want AI to amplify your business's unique DNA, you'll need to make a sustained effort to record, understand, simulate, and optimize your business processes.

• 20:40Nathan Labenz?

  Building these skills by trial and error takes years, but your business problems can't afford to wait. So here's how Diffusion can help. You identify your most important business problem. Fly to Silicon Valley for an intense week of problem solving with the Diffusion team. And by the time you leave, you'll have not only cracked a critical challenge, but built the core skills needed to do it over and over again from home. Cognitive Revolution listeners receive a 25% service credit on their first engagement with Diffusion. So visit diffusion.io/tcr to learn more about how custom-built software factories can scale critical outcomes for your business.

• 21:22Nathan Labenz?

  That's diffusion.io/tcr.

• 21:27Speaker 01

  Sure. So let me, so first, let me push back on one thing you just said. Most people think that we're schemaless, and that's not actually true. It's not that we're schemaless, it's that we're schema flexible, that you can change the schema more easily over time because more of the data is denormalized and therefore centralized in a way.

• 21:50Speaker 01

  It's not a strict requirement unless you specifically say so that every document in a collection, so we don't have tables and rows, right? We have, think of a JSON WAD as a document and you have multiple JSON WADs put together into a collection. Not every document in a collection necessarily has to have the same shape. You can have different shaped documents in the same collection. So because the schemas are flexible and because you can have different document shapes in the same collection, that's why people assume that we're schemaless. But what it does is it gives us the ability to be malleable in a way that a traditional SQL schema is less malleable.

• 22:34Speaker 01

  I won't say not malleable at all because it depends on how you're laying out the data. But anybody who's ever had to change a SQL schema that's already in production and the cascading effect that has on multiple tables knows the pain of what I'm talking about, that if you had some of that data denormalized, it's far easier to add attributes to a document that's already there and do so selectively in a way that isn't possible nearly to the same extent in the SQL world. And this is where how we've implemented, like the history of how we implemented vector search and the impact that has on application architectures for agents ends up, like the details matter here. So if you'll allow me, I'd like to talk a little bit about that history. Does that sound okay?

• 23:19Nathan Labenz?

  Yeah, please.

• 23:20Speaker 01

  Okay, so for us, it really started with in an unusual place, and that is with lexical search. So in 2020, we noticed that a common use case for MongoDB was for people to stand up their own Apache Lucene servers co-located with wherever their MongoDB clusters might be. And the reason they were doing it is they wanted to be able to point that Lucene cluster to different text fields in a document and be able to do keyword retrieval off of them. Perfectly reasonable thing. So we thought, well, as you might know, there's, there's three different versions of MongoDB. There's community, which is you're responsible for the support, you're responsible for the operations.

• 24:02Speaker 01

  There's enterprise advanced, which most customers are using on-prem, where you're responsible for the operations, but we're responsible for the support. And then there's Atlas, which is our managed service version of it, which you could deploy your instances on any cloud hyperscaler data center you'd like amongst Amazon, Google, and Azure. And in that form factor, we will do the support and we'll do the operations. So those are the different choices you have when you deploy MongoDB. And regardless of which one you choose, like I said, we noticed people standing up Lucene servers so that they could get keyword retrieval and what they had.

• 24:42Speaker 01

  So in 2020, we introduced what's now known as Atlas search, which is on the managed version, the managed version of MongoDB. You automatically just, you just get it as part of the instance that there's a lower priced tier where you just point it to attributes, some text attributes that you already have in your data will automatically index them. You can do keyword retrieval search on those. Then the next logical thing was a couple of years later, well, if you're going to have lexical search, you might as well also have vector search. So think about this, what's a vector? At the end of the day, a vector is an array of floats, right? You take some piece of data and whether that piece of data is text or an image or audio

• 25:27Speaker 01

  or video, you pass it to an embedding model of your choosing and what you get back is an array of floats. So to us, an array of floats is just an additional attribute to the documents you already have. So for us to implement vector search was just add an additional attribute to the flexible schemas that we already have and now build a vector index on top of that array of floats. And that's your vector search. And what it created was this notion of being able to do these powerful hybrid searches as well. So if you think about the, sort of the trivial example that most people learn MongoDB off of is suppose you have a book and a book might have a text field for the title and

• 26:11Speaker 01

  it might have a text field that is a URL that points to the cover of that book. You might have an integer that's the number of pages. You might have an integer that's the year of publish and you might have a text field that's the synopsis. So if that's a standard record for a book and if that's a standard document for a book in MongoDB and that's put in a collection, suppose now I want to create a vector index off of the synopsis. I could take that text, synopsis text, I could pass it through an embedding model of my choice and I then store the array of floats that come back from that embedding model.

• 26:52Speaker 01

  And if I wanted to, suppose I had 10,000 records, I had 10,000 documents in my collection, maybe I only want to do a vector search on the top thousand. Maybe I don't want to do it on all of them. Because of the flexibility of the schemas, because you can have different shapes, MongoDB not only allows that, but thrives on it really well. So we had lexical search, we had vector search, and then that enabled us to do this notion of hybrid search. So suppose I wanted to do, combine a lexical search of the title of the book with this now vector search that I have on the synopsis. And what if I also wanted to pre-filter, because I've got some integer fields in here? Suppose I want to do a search there where I do a lexical search on the subject and

• 27:37Speaker 01

  a vector search on the synopsis, but I want to eliminate any documents, any books that weren't published after, that were public. Oh, I only want to see books in my dataset that were published after the year 2000. So now I have these three levers of query power that together are giving me lots of interesting ways to now query my data, that I combine this sort of pre-filtering based on the metadata with the lexical search with the, with the vector search. So once we had that in play by like 2023, we thought, well, this works with any embedding model you want, but could we make this easier to develop if we had an embedding model that

• 28:22Speaker 01

  was part of it? And that's why we purchased Voyage in 2025, which should create this better together story. We can get into all kinds of things with the better together, but the way that we go to market with Voyage is you can use Voyage on whatever vector database you want, you can use any embedding model and use MongoDB as your vector database, or you can use them together and there's a better together story that's a part of that. So that's kind of our history. Like I said, kind of unusual that it started with lexical search, but it kind of went lexical search, vector search, hybrid search with pre-filtering, and now this better together storage with the Voyage embedding models. That's how that built over time for us.

• 29:02Nathan Labenz?

  What's it like to query that today? Are you able to, I think this is like a, has been relatively recently advanced, right? That I can now issue a single query and put kind of, in SQL terms, I guess, a where clause and like all three of those different kinds of data and the engine under the hood, which filter to apply first, like all that stuff kind of, I don't have to worry about anymore. Is that right?

• 29:28Speaker 01

  That is. So one of the advantages that we have in the way that we implemented vector search is it gets to stand on the shoulders of the base product that we already had. So we've been running Atlas in production for a decade now. So we already knew, know how to replicate data across multiple data centers, even across multiple clouds. We already know how to do sharding, whether you want to keep data close to specific users for business reasons or for legal reasons. We already know how to do security, not only query at rest or not only encrypted at rest and encrypted in transit, but we can also do during the query. We get to stand on the shoulders of all of that for the vector search. And one of the things that we get to do on top of that is MongoDB has always had this

• 30:13Speaker 01

  notion of aggregation pipelines. And the point of aggregation pipelines is instead of forcing you to do a query, get some JSON WAD back of your result set and then have to massage that before you use it. Instead, with the aggregation pipelines, you can give extra, you can give extra instructions to the backend so that that data gets sorted, filtered, massaged in different ways so that what you get back from that API call, you can now use immediately and you don't have to do any other further massaging. So because we already had that mechanic in the core product, what you're referring to is we just announced a pair of features. One's called rank fusion and one's called score fusion.

• 30:56Speaker 01

  And what that allows you to do is in absence of an aggregation pipeline in these new aggregation pipeline stages, you would have to make the vector search and then independently you would have to make the lexical search and then you would have to combine them yourself. Now with those two aggregation pipeline stages, you can do it in one API call. So you only have one round trip call to our servers and then on the backend, we'll do both of those calls for you and then we'll combine the results either based on their rank with rank fusion or based on their score with score fusion. So that what you get back is an already sorted list based on which one of those functions in the aggregation pipeline you chose, so that you could then use that JSON WAD, that

• 31:40Speaker 01

  result set immediately.

• 31:43Nathan Labenz?

  I'd love to learn a little bit more about how that works because this is something that I have run into and I've come up with pretty simple solutions for myself, right? But like, you sort of have the filter and then I'll just be like, okay, I guess I'll sort by the vector match score subject to the filter, right? Or we'll kind of tier them based on different filter values. It's always struck me that there should be smarter ways to do that that I've come up with. And I'm guys have some. So what are the kind of galaxy brained approaches for getting the best final sort order from these kind of fundamentally different things?

• 32:28Speaker 01

  So for now, what most people are doing is, and this is the reason why we have the pipeline stages that we do, it's usually based on a rank or by score. And if you also throw in the pre-filtering that lets you, that lets you reduce the data set before you do the queries. And then you get those queries back and you can interleave them based on either score or rank. For now, that's what most people are trying to do. But my joke for this Nathan is like, there's two kinds of developers. There's those that love MongoDB and there's ones that haven't tried it. Like we're, we're, we're always in a pretty tight loop of feedback where we, the whole reason we came up with rank fusion and score fusion is because the community told us they wanted it in very much the same way that why did we start doing Atlas search? Why didn't we make lexical search part of the platforms? Because we saw

• 33:13Speaker 01

  people were using it. We found a way, we saw a way to make their lives easier. If it's so that you don't have to spend the operational cycles managing your own Lucene servers, what could you do with the cycles that we give back to you? And it's the same thing with rank fusion and score fusion, so that you don't have to manage that code base yourself. So that if you could just throw it in as a verb in the query that you already have. Okay, now what are you going to do with that, with that time that we're giving back to you? So where it goes, community is going to tell us what other kinds of things they want with that.

• 33:50Nathan Labenz?

  Read more novels is one candidate answer for what I'll do with my time savings. And going back to your book example, another thing I've struggled with at times is, let's say I have the book, but let's say I actually have all the contents of the book too. Right now I've got hundreds of pages of text associated with this kind of one entity. Now I can break that down into, I just read a novel that had acts. So it had like four different acts. And then of course there were chapters within the acts. And then there were sections perhaps and paragraphs. And obviously you're down to sentences at some point. How do you think about, and I think there's some new work on this too, although I confess I don't

• 34:32Nathan Labenz?

  grok it fully. How do you think about like, how that raw text should be chunked or otherwise divided up to be represented in vector space? Is it like, am I looking for a single answer to that? Or is it, do you think of it more like a trunk and branch and leaf kind of thing? I've experimented with that kind of stuff a little bit as well. But I don't, I guess I, I don't really know what the state of the art is in that realm today.

• 35:07Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 35:12Speaker 04

  You're listening to Deep Gram Flux TTS. Different voices, same model, all ready to speak.

• 35:18Speaker 05

  Flux TTS is a streaming text to speech model built for voice agents.

• 35:22Speaker 04

  Flux reads the room. It holds context across the conversation, turn after turn.

• 35:26Speaker 05

  With consistent tone, interruption handling. And plenty of personality.

• 35:31Speaker 04

  So your agents keep it flowing and customers can just keep talking.

• 35:34Speaker 05

  Try Deep Gram Flux TTS free now until September 12. Visit deepgram.com/keeptalking. Terms apply.

• 35:42Nathan Labenz?

  Today's episode is brought to you by Anthropic. By now you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses. I give it feedback on its

• 36:26Nathan Labenz?

  misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of

• 37:10Nathan Labenz?

  the features mentioned in today's episode. Once more, that's claude.ai/tcr.

• 37:17Speaker 01

  Fair. So there's a couple of, more broadly speaking, and I promise I'll answer your question. More broadly speaking, the use case for this level of search, for this kind of semantic or vector searching is typically rag pipelines for agents or for a more sophisticated agentic memory for agents. That's when you typically see vector search start to enter the conversation when it comes into an agentic architecture in some way. Specifically, what you're talking about here is chunking and chunk size tends to be more in that rag pipeline use case. I've got some external data. I've got some proprietary data. I want to inject that proprietary data

• 37:58Speaker 01

  into my LLM of choice without having to go through an expensive fine tuning or training process. Because the LLMs are trained on public data. They're not trained on my proprietary data. So how do I inject that in a safe way that still gives me some data fidelity so that I'm not, I'm not giving away my proprietary data, but I can still get it to focus on my business problems instead of every LLM knows who the last 30 Oscar winners were. And that's not super helpful when I'm trying to get an agent to help me figure out what my NOC should do next, or my, my IT ticketing should do next. So that's the use case. So if you've got a bunch of documentation that someone who is a human professional in that world, you now need to

• 38:41Speaker 01

  figure out how big, how big should the chunks be that I place in the individual documents? And what's the fidelity of that? And when you're the application developer trying to figure this out, there's this trade off that you're looking at. So if I make my chunks too small, let's say to the fidelity of a sentence, if I make my, I potentially lose the context in which that sentence appears. Just if you just read one sentence from a novel, doesn't tell you a whole lot. But if you then also read the paragraph or the page, it tells you a lot more about what the context of that sentence is. So that's an argument for, well, maybe I should make my chunks

• 39:22Speaker 01

  bigger. There comes a point though, where if my chunks are too big, my storage costs start to go up and my retrieval quality starts to go down. Because if I've got three pages, let's say, and I'm trying to get to the fidelity of a sentence, I now have too much information and I lose the fidelity of that sentence. So what's the right answer? I've been doing this a long time, Nathan. The answer is always it depends, right? So you have to go through as a developer, you have to go through this iterative cycle. Like I chunk to a certain size, I run some tests, what's my retrieval quality? Okay, now I try it again. I try a different chunk size. I do some tests. I see what my retrieval quality is. And I have

• 40:04Speaker 01

  to do that three or four times that for my use case to try to find out what the right balance is between chunk size, storage costs, and retrieval quality. So one of the reasons we bought Voyage is because like those guys, like their side hustle is teaching people at Stanford how to, how to build LLMs. Those guys are all geniuses. And they came up with this new concept last summer called contextualized chunking. And we've got a blog article on this that we can potentially link through here. But the idea here is that if left to your own devices and figuring out your own iterations of chunk sizes, there comes a point in a graph where if you've got retrieval quality on your X, on your Y axis, and you've got chunk size on your X axis, with a

• 40:46Speaker 01

  traditional embedding model, there comes a point where at lower chunk sizes, you have zero context. So the retrieval quality is low and it builds. But then at some point it flattens and then it starts to degrade if your chunk sizes become too big. And that's why you have to go through these iterations to find the right combination. But what if you did, what if instead of sending that all as one big text blob? What if instead you send it as two? What if you sent the fidelity of the sentence you actually want? And then as a second string, you send the size, you send the other contextual informations where the name contextualized chunking comes from. And we will figure out for you what the right chunk size is for that combination. And we will give you

• 41:30Speaker 01

  a vector array that comes back that balances those things for you. When you flip that script, it turns out you can get better retrieval quality with a smaller chunk size, which otherwise is not possible. If you're doing it the traditional way, you, the way to get better chunks, way to get better retrieval quality was with higher chunk size. But if you use this contextualized chunking, we released version three of that last summer. We just released four of that in the last six weeks. You can flip that script. Now there still are use cases where you need to control the chunk size for different things. But if that's something that you wanted to not worry about or have to learn about, and the way I think about this is more developers will build

• 42:14Speaker 01

  AI agents in the next three years than did in the last three years. And in order to make that possible, we have to lower the learning curve. We have to make it easier for the developer ecosystem to learn how to do this. So if you don't want to have to learn how to go through the iterations of figuring out what the right balance of chunk size and retrieval quality is, instead, if you could use contextualized chunky, we'll figure it out for you and you still get good retrieval quality out of it. So that's, that's one of the benefits that Voyage offers that no other embedding model out on the market offers, that ease of use feature that gets you better retrieval quality.

• 42:51Nathan Labenz?

  I definitely appreciate not having to worry about it, but I do want to learn a little more about it. It sounds like this happens in sort of a Test-free way at the level of the developer. So like, I don't have to bring a bunch of eval suite work to the table. So how, at the level of kind of principles, like, how is it working in the background? And you said also, I'm getting one vector back, right? So I pass in the sort of, the chunk that I think I would really want to be able to zero in on, and context, and that gets converted into a single vector back that represents both of those things in, I guess, some sort of superposition?

• 43:37Speaker 01

  Yes, that's exactly how it works. So you get back one array of floats, exactly how you did before. It's just now you don't have to go through the iterations of figuring out the size of the chunks you need. And similarly, there's another Voyage feature that all the Voyage, all the Voyage embedding models have called Matriceka reasoning. Because the other place you have to make a decision about storage costs versus retrieval quality is with number of dimensions. So what do I mean by that? Everybody knows what two dimensions is if you've taken high school level algebra, right? X, Y, right? But in an embedding space, you tend to have at least 256 dimensions and sometimes as high as 2048.

• 44:19Speaker 01

  And the more dimensions you have, so each dimension is represented by one of those floats in that array of floats, the more dimensions you have, the richer your embedding space is, and the better retrieval quality you get. But at higher dimensions, that doesn't come for free. You've got a storing 256 floats, takes up less space on disk and in the index and memory than it would for 2048. So you, again, have to go through this iteration of what's the right number of dimensions for my use case, given my, what my storage costs might be. All the Voyage models have a feature in them called Matriceka reasoning. It comes from the Russian nesting dolls. If you think about how Russian nesting dolls work, right?

• 45:04Speaker 01

  You've got, you've got one doll and you open it up and there's another one exactly the same, but smaller, a smaller fidelity inside, and then you keep doing that and over and over. So what, what the, how the, how the Voyage models work is when we generate, let's suppose you did it 1024, suppose you wanted 1024 dimensions, you run some tests, now you want to try 512. With a traditional model, you have to run your entire corpus of data through a second time at 512. But with Voyage models, you don't have to do that. When you've run it through at 1024, vectors, those floats, they're ordered. So if you want to try 512, you just lop off the last 512, and that's, now you immediately

• 45:48Speaker 01

  can begin testing on the remaining 512. So again, it doesn't completely solve the problem of figuring out what the right combination of retrieval quality and storage space is, but it helps you get to the answer faster. So these are, there, when you take some of these things in some, there would be three or four of these kind of features that make it easier to use and help you get to your final answer more quickly. And the idea there is to give you time back in your day that you can work on your business logic instead of figuring out the plumbing.

• 46:22Nathan Labenz?

  Yeah, that's cool. I'm a huge fan of Matriceka, anything. When does this stuff become necessary? So for me, I'm basically a business of one, and I try to be an early adopter of everything that I can. And I do have a pretty well-working, I call it deep context, but basically a retrieval system that allows my agent to go into kind of all of my history from the last five years, essentially, which is emails and Slack messages and everything I publish online and DMs across all kinds of channels, the podcast of the transcript of the podcast, I should say, diri, so it knows what I've said and what the guest has said.

• 47:07Nathan Labenz?

  It adds up to about a gigabyte in my case. And I haven't really optimized it much at all. I just kind of let the agent throw it into a database of its choosing and put whatever optimizations on it it felt like it needed to. And then we did get a, at one point, it was like, well, yeah, we could probably do better than keyword. So we've got an embedding layer on there as well. I think, full disclosure, I believe I used the Gemini embedding model for that. But it's like, not very well optimized. How would I know if I'm really missing out on something? I don't have like, a huge eval suite, right? It's just like, I'm kind of vibing it with, it doesn't seem to be working well. Is it a matter of like, data scale, scale of users?

• 47:53Nathan Labenz?

  Is it about like, cost, why I want to optimize my inference cost? And that's where I really need to get serious about? How much data is, is being returned? Like, what are the, what are the thresholds that people, or obviously larger organizations, cross where they're like, okay, can't really do it the, let the agent choose its own adventure way anymore. We really need to get serious about some of these optimizations.

• 48:18Speaker 01

  Well, I'd return to the three things I mentioned in our first 20 minutes or so. It's, it's speed, scale, and retrieval quality are the main three things. What most people do, and you're not going to offend me if this is what you did, I mean, most people start with Postgres and PG Vector, and then they choose, they choose their embedding model with whatever cloud they're using. Like Gemini is prominent if you're going to use, if you're going to be on Google, just like OpenAI's embedding model is, is pretty popular over on Azure because of the historic relationship that those two companies have. But there comes a point in, when you're, when you're doing a demo, when you're doing a POC, it doesn't always show itself. But there comes a point where, when do milliseconds matter to your use case?

• 49:01Speaker 01

  When does scale matter to your use case? And that typically comes, depending upon chunk size, that typically comes at about 100,000 vectors is when, is what I mean by scale. And when does retrieval quality matter? If you look at Voyage AI model, Hugging Face has a benchmark out there called RTAB that Voyage AI models are typically at the top of, and we can get as much as 14% improvement compared to some of those other embedding models that we just mentioned. So are there use cases for which a 14% difference in embedding model quality is the difference between a hallucination and a correct answer? And that's before you even start putting rerankers on it, which is another way that

• 49:43Speaker 01

  you can boost retrieval quality without having to do anything special to your data. So like I said, it's, it's speed, it's scale, and by scale, I typically mean in the neighborhood of 100,000 vectors, and retrieval quality out of your embedding model. Most people think embedding models are commoditized, and that is not true. There is a very big difference that you can get in retrieval quality based on what embedding model you choose. Anthropic does not have an embedding model in market. They recommend us.

• 50:14Nathan Labenz?

  It's a great recommendation. You mentioned rerankers, and then there was also this kind of earlier concept of sorting results out of the database. This brings to mind this concept of bitter lesson engineering that I think is kind of growing in prominence, which, from my simple point of view, is just like, every so often, you should probably kind of go through your stack and look at all the klugey extra things that you did to make things work and say like, which of these do I no longer need? Because the model got smarter, or the embedding model got better, and so things are just

• 50:54Nathan Labenz?

  kind of naturally working now, or could naturally work now, where in the past I had to kind of do all these artisanal craft sort of things to, to make sure that they worked. Are you seeing examples of that in the retrieval space broadly, where things are in some ways getting easier? Or have you not really seen the bitter lesson apply to these pipelines, these production environments?

• 51:23Speaker 01

  Yeah, I think there's two places that I've seen that recently. And one of them is with the reranking that we were just talking about. So in the spring, we released Dollar Rerank, which is a companion to the score fusion and rank fusion that we already talked about. So those two relate to doing a hybrid search. The Dollar Rerank, again, you would, in a historical way, if you would have to do your vector search once, send your results to a reranker to get them reordered in a way that is most optimal for a ragged use case, to then pop them into the context window, just like we did with the score fusion and the rank fusion. We've now got a, a stage where if you do Dollar Rerank, you call the API once, we'll

• 52:06Speaker 01

  do both of them for you on the backend so that you only have to make one round trip to the server for that. So that's one place that we've seen some additional ease of use. The other one is a feature that we released recently called auto embeddings, which is, you tell us, this is a better together story, you tell us what, which collection, you tell us which attribute on documents in that collection, and you tell us which Voyage model and how many dimensions you want, and we'll take care of the rest of it. Anytime an existing document with that attribute in it changes, we will automatically take that new, let's say it's text, put it through the embedding model, update the vector in

• 52:50Speaker 01

  the document, update the index and memory, and we'll do all that for you. If a new document shows up into that collection that has that attribute, we'll go through the same cycle for you. So again, trying to remove some of the plumbing so that you get some time cycles back as a developer, so you don't have to, you don't have to go craft your own and maintain over time your own embedding pipeline. You just pass us one JSON WAD and we'll do it for you.

• 53:17Nathan Labenz?

  Zooming out a little bit from the database to the, the broader systems that wrap around them, you, stuff about memory for AI systems. And this is another thing where I've seen kind of the great creation and then somewhat simplification, probably simplification has been enabled by things that enterprises are maybe a little reluctant to pay for at scale. Like, when I get a million token context window, I'm like, great, now I don't have to worry about all this stuff. I can let the model handle a lot more of it. But I do take your point that obviously you got thousands of users that can add up.

• 54:00Nathan Labenz?

  What would you describe as, and of course it depends, but trying to abstract away from some of the details and give me kind of a center of the distribution description. What would you say is like the state of the art in what a good memory system looks like today?

• 54:15Speaker 01

  Sure. So I, again, history matters here, right? When ChatGPT came out in 2022, in the end of 2022, it had a very simple application architecture. You took, you took the query, you put it in a context window, the LLM would process it and give you a result. End of story. That's how it worked. And by spring of that year, by the next spring, you would see headlines like ChatGPT passes the bar exam. And it would have these very cool things it could do. We already talked about why you need RAG, because those LLMs even today are only trained on public data. If you want to inject proprietary data into it, you need to, to specifically do that with a RAG pipeline.

• 54:59Speaker 01

  What we saw in 2025 though, was the second problem that we see with, with the core LLMs. And that gave rise to the need of memory. And that is like in spring of 23, if you had asked ChatGPT, what's MongoDB's stock price today? It couldn't tell you. And the reason it couldn't tell you was the LLMs have a knowledge cutoff day. There's a date after which they don't know anything about the world because the model vendors at some point have to stop training and put them into inferencing mode. And the way that we overcame that in 2025 or so was with tools and with MCP. And being able to have to grant to the LLM, hey, here's this API or here's this tool we could call to go get additional information.

• 55:39Speaker 01

  That's why whether you're a Claude person or a Gemini person or a ChatGPT person, that's why today if you ask, hey, what's MongoDB's stock price? It could tell you because it can find a tool in its arsenal, go make a web search, get that information and report it back to you. The other thing we saw in 2025 though, is this notion of looping where the result of one loop of that agent then became the input of the last one. The context window is still very dumb, right? The context window is still, you have a fresh context window with every time you pass it to the LLM. And that's when we needed agentic memory. And the early passes we saw of agentic memory was short-term memory was typically, okay, take every response that we got within this session and cram it into the

• 56:25Speaker 01

  context window so that we can see what happened earlier in this conversation. And then long-term memory was, okay, now do the same thing for every session we've had for the last three days and cram that in there as well. And then had two negative side effects that I now see people starting to push against. Negative side effect, one, we already talked about is token maxing, right? If you, if every agentic loop puts a million tokens in the context window, even, even context you don't necessarily need, your cost starts to add up. But there's also a functional cost to that. There's a couple of academic studies that show that like the first 7K and like the last 70K tokens are the most important. And the stuff that's in between can end up just muddying and confusing the LLM as it's trying to come up with an answer.

• 57:09Speaker 01

  So instead of thinking of how could I maximize, how could I cram a hundred, a million tokens into the context window every time? What if I could choose just the right 200,000 for this agentic loop? So we're starting to see more sophisticated short-term memory that doesn't take everything in the session, but just the things in the session that are relevant for this pass. And the same thing with long-term. And we're starting to see some new memory types, especially in Fortune 500s. There's a new memory type called taxonomic memory. So think about either in a vertical, so like automotive, we were talking about before, or in a specific company.

• 57:49Speaker 01

  There's some terminology that is specific to verticals or specific to a particular company that a layperson who is not in that vertical, those words may mean different things. So you might have a list of, let's say a hundred of those terms. But for this agentic loop, what are the five that are relevant for this loop of the agent? And then do that again for the next loop. What are the five that are relevant for that loop of the agent? So again, instead of taking all hundred terms and putting them in the context window every time, when not all of them are going to be relevant. What if you could instead have a more sophisticated memory type, memory system?

• 58:31Speaker 01

  That instead of just taking everything, use vector search or semantic search or some combination of vector and lexical with some pre-filtering. And just got the, just the right, just there's some perfect, you'll never get there, but there's some perfect context window for this pass. And that perfect window doesn't use all of it. It uses just the stuff that it needs to. Those are the kinds of things that we're starting to see in enterprises, both to combat the cost of the tokens and to combat the relevance of the tokens. So you can get the best answer out of this call to the LLM.

• 59:10Nathan Labenz?

  And do you have any tips for maintaining these memory systems? I mean, this kind of goes back to some of the challenges with, and maybe there's good solves within MongoDB for some of this. If so, definitely highlight them for me. But I sort of struggle with the lack of normalization or my failure to observe the dry rule in my own memory system. Sometimes I'll have like, you know, I have these like raw exports that combine my email and DMs and everything else right into these kind of monthly log files. And these are typically like a couple hundred thousand tokens for a month. Then I'll like summarize those into a monthly summary.

• 59:54Nathan Labenz?

  And then I'll take the monthly summaries and I'll put those into a yearly summary. And then I've got like a wiki type Karpati inspired structure that sits on top of all that and is like more entity based and they kind of link to each other. And some of this lives in a database, some of it lives in a file system. But now I've got a challenge where I'm like, some new stuff has happened. Okay. What do I do? What do I change? How do I make sure that it's changed everywhere it needs to be changed? I sometimes also have instances where

• 1:00:26Nathan Labenz?

  I've had really funny, I mean, the models are getting better with this, it's less of a problem these days. But in setting this up in the first place, I had some problems where the models would see a project that I like experimented with a little bit or like was maybe going to pursue and then didn't. And it will like keep it as like an open thread for months and months. And I'm like, dude, I never actually did that. You can just forget about it. So forgetting is also sometimes a real virtue of a memory system. How, what have you learned? What tips would you offer in terms of how to maintain these things? So they're accurate, not just when you first create it, but on an ongoing basis?

• 1:01:04Speaker 01

  This is the hardest problem with agents right now, I think. And the way that I put this to people is we, as an industry, we've been building databases for 60 years. We've been building agents for like 18 months, man. Like nobody knows all the answers yet. We're all sort of collectively figuring it out. But what I do see with these more sophisticated memory types is that as the application author, that you have two responsibilities with these memory systems. Where with the prior one, you only had one really. So not only do you call this memory system, hey, here's my query, give me the best, maybe you give it a token budget is the way some of these work. So that you don't, you don't have to figure out how many memory types there are and how to query them. You're just like, give me the best 50K tokens or give me the best 100K tokens.

• 1:01:49Speaker 01

  You give it a token budget and then it gets it back from the mix of different memory types it might have. And then you put that in the context window, you get your answer. And then your second responsibility is to send that answer back to the memory system. So the memory system can then curate it and then write it into the memory system. And when it writes it into the memory system, the more sophisticated ones also have this notion of RBAC. So that if you and I are in the same job type, that we get to share the memories. So that if you come up with a really good memory and then I get to reuse it later, then that ends up being beneficial for both of us. So that's the second responsibility you have.

• 1:02:31Speaker 01

  But the hardest part is what you're talking about here, the way that one coworker just wrote this to me yesterday. I want to quote him, write, change, recall, forget. Because these things have a, a half-life to them, right? Like things that are more recent are more important than ones that, that took place weeks, if not months ago.

• 1:02:50Nathan Labenz?

  Maybe you can tell me about any like Mongo specific tricks or advantages for these sort of graph structures that I seem to keep making. I have people and the organizations they work for and the ideas that I sort of associate them with. And these things all kind of point to each other, but they do it in a pretty loose way where I'm right now kind of trusting the agent to hopefully notice those pointers and to the degree I'm like running maintenance, hopefully follow those pointers and do the necessary maintenance. I don't have a lot of guarantees and I suspect that there are better technologies that I

• 1:03:33Nathan Labenz?

  could be building on that would give me a lot more robustness.

• 1:03:37Speaker 01

  Well, there's a couple of different ways to tackle this. Like I said, the forget part is the hard part of it. But what we see our customers do is this is where like the retrieval quality of those memory types, that's where that non-commoditized embedding model can make a difference and where a rerank can make a difference. You can get, depending upon the use case, you can get like a 5 to 10% boost at your retrieval quality just by using a rerank on top of whatever your embedding model of choice is. And that's true of architecturally, that's why we made it so easy to implement the rerank on top of the voyage embedding models when you're doing that vector search. But there are use cases where maybe you have so much data that you might have to take a hybrid approach

• 1:04:21Speaker 01

  where maybe you use, maybe use a graph structure for, I don't know, two to six levels of data and then you get down to a leaf and you do a vector retrieval inside that leaf. We see people doing that as well. There's all kinds of information you can find on our website about how you can, because we're JSON based, you can use MongoDB to build graph structures into your data so that you only have to go one place for a graph database, a core database, a vector database, the embedding, reranking all in one place. So you don't have to try to stitch together multiple tools yourself and have to maintain that over time. But we do see people doing that with graph structures. It's typically not as deep as like you would

• 1:05:04Speaker 01

  traditionally think of as a, as a graph database need. But like I said, it's pretty common to go, I don't know, half a dozen or maybe, maybe a dozen layers that maybe for bigger corpuses of data that are more heavily categorized, like some of our retail customers do that with product databases. If you think about how a hierarchy of products might appear on a website or on a mobile app, they might segment that first by product category using more of a graph style. And then, once you get to a particular product category, then do some vector searching within that individual category node.

• 1:05:42Nathan Labenz?

  Are there any applications that you would point to as just being great examples of memory well implemented?

• 1:05:51Speaker 01

  Well, our biggest customer right now, they haven't been very public about how they did it. There's a company called 11 Labs out there. They have multiple agents per customer. This sort of micro agent approach where you get multiple smaller agents at the disposal for individual customers and given the number of customers they have. So they, 11 Labs started life as a model provider that was doing way sophisticated speech to text and text to speech. And they built a platform on top of that that is more of like an audio editing suite that they do for that. And as part of that, they have a bunch of agents doing all kinds of editing and transformation kinds of things for their customers. If you think about the kind of context memory that they have there, an important way to do that

• 1:06:34Speaker 01

  as well. I would worry about any memory architecture that instead of relying on lower costs, embedders and rerankers is relying on multiple passes of the LLM to help you categorize and shrink the corpus of data that you might want to then put into the context window for the larger, sort of more functional LLM call. Cause then that's just adding up tokens as well. I mean, that why things like embedders and rerankers have a lower token cost is again, right tool for the right problem.

• 1:07:10Nathan Labenz?

  Yeah, that adds a lot of latency too at this point. Even the small models can reason for quite some time before you actually get your answer from them.

• 1:07:18Speaker 01

  They can. And so all those things add up over time. And it's not just tokens, but you're right, the overall latency for the decision loop that you're into makes a big difference.

• 1:07:28Nathan Labenz?

  I think you also have some interesting takes on build versus buy analysis. I won't even try to summarize it. Just give me your, your hot takes on how people should be thinking about building versus buying. What is mature enough in the AI realm to buy? And what do you, even if you kind of buy a part of it, what do you still have to expect that you're going to end up building or customizing enough that it kind of feels like building?

• 1:07:53Speaker 01

  Yeah. So I have a good fortune in my job. I've been to seven countries this year to talk to probably a hundred different customers about where they are in their AI journey. And they tend to fall into sort of three camps. Camp one is I bought a license for this one tool and I'm done, right? Like my AI strategy is done. I bought one thing. And while that can be a good starting point, it typically is not specific enough to help solve the problems of your overall enterprise. Then you have a group of people who have done some POCs and maybe a couple of production deployments with varying degrees of success on ROI that have struggled with ROI. I always argue probably picked the wrong problem.

• 1:08:36Speaker 01

  Picking the right problem is really important in this space. And the way that I encourage customers to think about this is like, what are the top 10 to 15 problems that you have going on in your business right now? And of those, what do you have good data for? I mean, you made a comment earlier in this conversation, like data quality is a big deal. Things like bad data quality and bad security posture don't get solved by AI, they get amplified by AI. So what's your, what's your biggest problems? What do you have good data for? And then what do you already have metrics for around the problem? And that's the biggest difference that people that tend to skip over when it comes to problem selection. Because if you don't already have metrics for how something is performing, you won't know if

• 1:09:19Speaker 01

  it got better. So call center use cases are very popular, low hanging fruit in enterprises because I already know what cost per call is. I already know what call volume is based on how I'm already bonusing people in those jobs. So if I introduce AI into their workflow and I see those numbers jump, I can attribute that change to the AI and I can do a back of the envelope ROI. We see the same thing with software delivery lifecycle. Early in the year, we saw all kinds of bragging about, well, now because I've got Claude Code or I've got T-Gravity or if I've got Codex, now I can produce five times as much code as I did before. Anybody who's been doing it any length of time knows that lines

• 1:10:02Speaker 01

  of code is a terrible metric to judge the productivity of a set of developers off of. Instead, how, how fast are you getting from idea to correction deployment? That's the metric that matters there, not, not necessarily lines of code. So the metrics there matter. And that's the difference between the folks that are in that middle state, either stuck in sort of POC purgatory and haven't quite got to production deployments. The biggest difference isn't in how they're applying the tech, it's what problems they chose to try to solve. And then you've got people that are sort of more advanced that are looking at some of these more sophisticated memory types and things where they're trying to optimize the systems that they have. And in some cases, they're, they're making purchases

• 1:10:44Speaker 01

  of larger platforms to help them do that. Other times they're doing homegrown. And like I said, we're, we're like 18, 24 months into this. There's, there's no, there's no one way to do this. There's, there's no Lampstack for agents yet in the way that we have with web development. We will get there, having lived through that, having lived through that lifecycle, we'll eventually get there. But like, there's, there's no React and Angular, there's no Lampstack for agents right now.

• 1:11:12Nathan Labenz?

  You mentioned call centers and this is the second time today that this has come to mind. I just ordered a pizza last Friday night from the local pizza place in my neighborhood. Shout out once again to Greg's Pizza in Detroit. I think it's just a one location business. And who answers the phone?

• 1:11:31Speaker 01

  Detroit style pizza is underrated, by the way.

• 1:11:33Nathan Labenz?

  It is. This actually is a traditional circular round pizza, but I agree. After auto, it's Detroit's biggest export to the world is pizza. But even just this one location business, right? I call and who answers but an AI agent? And it was a pretty impressive experience, I have to say, like very natural conversation, not flawless. There were a couple of little things where, especially as I started to get a little adversarial in my testing of it, like I asked it, what company is powering this AI agent experience that I'm having? And it took that as it somehow thought I was ordering pizza for my company. So, you know, there's some kink still, but I overall was like very impressed, like the

• 1:12:15Nathan Labenz?

  Happy Path, I thought worked really well. And this has me thinking like, boy, there must be just a ton of transformation happening at call center businesses. How, how much are you seeing like?

• 1:12:29Nathan Labenz?

  There's always this kind of waves of change, right? Where like the first wave is like, we give our agents a tool where they can like get help faster and be more accurate. But then there's this like next wave, or there might be several waves, but certainly one that seems like it's coming is you just call and talk to AIs, right? How far along that journey are the customers that you are talking to? How much like actual disruption have you seen in that part of people's businesses so far?

• 1:12:56Speaker 01

  Most of the Fortune 500s that I talk to are doing employee facing use cases with human in the loop for exactly the reasons that you just laid out and for some of the ROI reasons that I laid out before. So if you think about the employee facing use case, you already know what your bonusing, your different employees are and what KPIs, what key performance indicators you're using to judge whether or not they're doing a good job. So like I said, I use the example of call centers before, but you can think of any job, whether you're an insurance adjuster or a line worker in a manufacturing plant or whatever your job might be. Everybody's got some key performance indicators. And if you introduce AI into whatever that workflow is and you see those jump, that means

• 1:13:41Speaker 01

  that you're, you've figured out some ROI. But there's also security, a data security issue for that as well. If you and I are employees for the same company, is it great if my salary accidentally leaks to you? No, but it's way better to have that than if we are two customers of the same company and I get to see your data. That is, a VP has to get on a plane for that kind of scenario. So like the risk reward is higher for customer facing use cases than it is for employee facing use cases. So at least where we are now in the ecosystem, where I mostly see the Fortune 500, there's exceptions, of course, but I see a lot more work going into those employee facing use

• 1:14:25Speaker 01

  cases with human in the loop than I see going full autonomous AI with customer facing use cases.

• 1:14:32Nathan Labenz?

  For you and what you're doing at and for MongoDB, are you, of course, you've got traditional documentation, right? That human developers can go and read, but I'm sure you're also thinking a lot about how do you make your documentation AI friendly? And then how do you have an AI that can proxy into that documentation? And then potentially you're like packaging up and distributing skills and trying to think about like how to become the database that is like recommended by the agents when somebody's doing something for the first time? How was, how was your role as like bringing this technology to the world evolved in light of all the different ways that people are discovering and starting their implementation process?

• 1:15:18Speaker 01

  So all the things you just mentioned are certainly things that not just us, but every software company is involved in one way or another. The two that I would shine spotlight on are the MCP tools to make it easier for agents to talk to your data in your MongoDB instances. We had an MCP server for a while and we were now kind of self-hosting some of those as part of Atlas, as part of the ecosystem. And then the other one is the agent skills. So think of that as kind of curated system prompts for doing things like data modeling, doing things like optimizing operations, the things that we have well organized playbooks

• 1:15:58Speaker 01

  for that previously we would have as documentation, but now we have them as a series of markdown files that you can feed to an agent of your choice to give it sort of the lowdown on different ways that it should approach different tasks. And there's maybe the, I think it's agent skills we just dropped a couple of months ago, has like the top eight. It's like, it's six to eight different agent skills that we have for things like operations and data modeling to help you do those sorts of things. So yeah, that's become pretty commonplace these days and we're certainly part of that ecosystem.

• 1:16:32Nathan Labenz?

  You've mentioned this acquisition that MongoDB did of Voyage a couple of times. And I had a kind of just a couple like theoretical questions about that.

• 1:16:44Speaker 01

  Okay.

• 1:16:45Nathan Labenz?

  One is it's striking that MongoDB built its own vector DB, right? And there's like a bunch of vector DB startups out there. So like one could have imagined an alternative scenario where there was an acquisition of a vector DB startup, but there wasn't. There was a acquisition of the model startup. And then also just look at the relative value and you said these guys are geniuses and I don't doubt that, but the price at even at $220 million, which is, I'm old enough to remember when that was serious money. You know, it's, it's less than a 10th of a percent of MongoDB's overall market cap. So I'm like, or less than 1%, I should say.

• 1:17:29Nathan Labenz?

  So I'm just kind of wondering like, what should we infer from these observations about value and defensibility in the software business over whatever passes for the long term in your mind? You know, you could tell a story about like, infrastructure is the big winner, models get commoditized, incumbents can defend themselves against startups. Like, what do you think are the right macro lessons to draw from this experience?

• 1:18:01Speaker 01

  I mean, fundamentally, what we've always been about has been, how can we make the day in and life of a developer easier? So we can make it easier for them to build their business logic and spend less time worrying about the plumbing lower in the stack. I went through a description earlier in this conversation about how we implemented vector search. And for us, because we already had the JSON base, it was, it was relatively straightforward for us to just, okay, you add an additional attribute, that attribute is your array of floats, you generate an index based on that array of flows. So that was pretty straightforward for us to add to the existing product and take advantage of the things like the charting, the security and the data replication that we already

• 1:18:43Speaker 01

  had in the base product that some of the net new vector database companies out there have to catch up and struggle with a little bit. So that's why we chose to sort of do vector search the way that we did is because of the flexibility based into, built into the document model from the very beginning, it was, it was pretty straightforward for us to do that. But where it was not as straightforward and why we made the Voyage acquisition was today you can still use whatever embedding model you want, as long as it generates that array of floats, you put that array of floats in your document, you go create your index and you're off and running. But we saw an opportunity to, A, that the market was seeing embeddings and re-rankers as a commodity, whereas we, we saw Voyage really standing out and like I said before,

• 1:19:28Speaker 01

  Anthropic agrees with us. And we could create these better together stories like the auto embeddings that I mentioned before. Another one I didn't mention is being able in Atlas to manage all your API keys in one place so that you don't have to go to one console for your core data, another console for your vector data, a third console for your embedding models, and a fourth console for your re-rankers. You can get all that in one place. The management of it is easier over time. So if you look at the history of MongoDB features and really what our focus has been, we're all developers at heart and it's, it's about making it easier for the developer ecosystem to learn and operate this, these new techniques that we see in application

• 1:20:13Speaker 01

  architectures around agents, whether it's a RAG pipeline or whether it's the agentic memory. So how do you make, how do you make vector search more approachable? How do you reduce that, lower, lower that learning curve so that more people can learn it more quickly and start to participate in this ecosystem?

• 1:20:30Nathan Labenz?

  Another story that I've heard that I want to just get your reaction to and see if you think it's true in your experience is I was speaking to a founder of a vector DB startup one time and I kind of made the case or put it to him that, geez, it seems to me like the incumbents are going to be able to add a vector aspect to what they're doing before you're going to be able to kind of replace everything that they're doing. And so it seems like they're going to have a hard, you're going to have a hard time like really displacing them. And his answer was, well, that may be true, but most of the data that is coming into our vector database has never been in a database before at all, right?

• 1:21:12Nathan Labenz?

  It's just been sitting out in some data lake or data warehouse or just kind of unstructured piles of documents. And now it's coming into a higher level of infrastructure and it's being made more valuable in a way that just wasn't happening at all before.

• 1:21:28Speaker 01

  Sure.

• 1:21:29Nathan Labenz?

  Are you seeing something like that? And, and what does that look like? I also noticed that there's the multimodal embedding model that supports video. So video would be potentially a great candidate for the kind of thing that has never been in a database before. Are you seeing this like phase shift of just like a much greater universe of data coming into MongoDB than in previous eras?

• 1:21:51Speaker 01

  We are see, we are seeing this broader ecosystem of data that, that wasn't indexed before because it didn't lend itself to a traditional lexical search. The cool thing about the vector search is, I mean, vector search at its core is taking some piece of data and mapping it into N space, into some geographical geometric space. And then really all a vector search is similarity, like where, what are the closest vectors to this new thing that I'm searching on? And video is a great example, audio is a great example, unstructured data, you just take a bunch of PDFs that you have sitting around in a SharePoint. I mean, those are good examples as well.

• 1:22:32Speaker 01

  So yeah, I think that there's, there's some truth to that, that we're, it's not just, how do I take the data and put it into a database, but now how do I find it? And how do I find in a way that is fast, is scalable, that's a good quality? And that's why the way that we implemented vector search and being able to, within the same platform, have the combination of the pre-filtering, the vector search and the lexical search, you feel like that gives us an advantage and gives our developer community more levers to pull from than what some of the alternatives are. And like I said before, because it's, it stands on the shoulders of the core product with the, with the Atlas version, you can already deploy that, any hyperscale data

• 1:23:14Speaker 01

  center you want with good data resiliency, with good security. And if you need to, to shard that data so that it doesn't leak particular geographies, we got all that for, essentially for free because of how we implemented vector search on top of the core product. And yeah, we're, we're seeing all kinds of different kinds of data get put into those documents in a way that we didn't before.

• 1:23:38Nathan Labenz?

  You mentioned travels, I think, so far to seven countries this year. Where I'm so kind of myopically focused on what's going on in San Francisco and Silicon Valley that I'm mindful that I may be missing important stories or differences in perspective that are going on around the world. I try to fill that gap, at least when it comes to China, but it's a big world out there. What has stood out to you in your travels this year in terms of differences of perspective on AI, usage patterns, values, you name it, could be anything, but just kind of what do you think the US audience in our inward looking way is missing that the rest of the world is doing?

• 1:24:21Speaker 01

  I think the biggest thing, if I turn question on its head just a little bit, there's a presumption in other countries that the US is ahead in doing things that other people are not. And I found the opposite to be true. So, I mean, I live and work in out of Cincinnati. So, US is one of those countries. I've spent some time in both Amsterdam and London earlier in the year, but I just did a tour that included stops in Toronto, Binglearoo, Mexico City, and Sao Paulo. And the two most sophisticated customers I talked to this year were in Mexico City and Sao Paulo. And they assumed when we started their conversation

• 1:25:04Speaker 01

  that they had US competitors that were doing things that they weren't. And like I said, the opposite was true. I think we've reached a point with some of these technologies that like geographic barriers don't matter nearly as much as they did during like the web app era, or even during the cloud era. Because for the cloud era, like if one of the hyper stealers didn't have a data center in your country yet, you were kind of out of luck. That's not true anymore. Like, pretty much every country has at least one hyper stealer data center in it, and by extension, access to models and vector databases and embeddings and bring rankers, like access for these things is far better than what I've seen

• 1:25:47Speaker 01

  with previous technology revolutions that we've seen. So, I think, understand why there would be an assumption that US companies would be ahead since so many of the bigger AI companies are US-based. But like I said, the two coolest things I've seen this year were in Mexico City and in Sao Paulo. So, I think that those geographic barriers to being ahead in the market are starting to disappear.

• 1:26:15Nathan Labenz?

  What do you think are the barriers? Why is that, I guess? Is it that the American companies are culturally too conservative to run as fast as some of these international companies are? Is it that, is it like a leapfrog story where they sort of had, the international companies, I mean, had kind of less recent technology investment that they would have to get comfortable replacing? Or what's, what's driving that surprising observation?

• 1:26:45Speaker 01

  Well, I think it has more to do with the lack of barriers outside the US than it does with any of the behaviors of the companies within the US. It's just access is democratized now in a way that it wasn't, that way that it wasn't for the cloud and for the mobile waves, right? Like, if you didn't have the radio towers yet, an iPhone's not going to do you much good. But now, we have all this infrastructure that's laid worldwide in a way that we didn't 15 years ago, that makes access to these things far easier than they were before. So, there's just, there's just a wider set of people in a wider set of countries that have access now that they didn't before. So, I don't think it's any, it's not an indictment of the way that I think US companies are thinking

• 1:27:26Speaker 01

  about it. I just think that there, that greater access erodes those geographic barriers that we've seen with previous technology waves.

• 1:27:35Nathan Labenz?

  Any closing thoughts? Any, anything you'd want to leave people with, or anything I didn't get to that you would want to make sure we touch on?

• 1:27:42Speaker 01

  Well, there's one technology thing that we have, I didn't get a chance to talk about, and that's shared embedding spaces. And then there's, there's also this notion of just like kind of where we are on this journey together. The shared embedding spaces, real quick, is one of the other ease of use things that the voyage models provide that nobody else does, is in January, we released four versions of the, of the version four of the text model. Everybody typically releases small, medium, and large, and at those different price, that each one has a different price point and a different retrieval quality that comes along with that. But we also introduced something called the Nano, which is open weight and free for anybody to download in, off of HuggingSpace. And what shared embedding

• 1:28:26Speaker 01

  spaces does is each one of those four models shares an embedding space, meaning that embeddings that are generated with one of those models, they're compatible with the other three. And what that lets you set up is you could take your corpus of data and you could embed it using, say, the large and developments. Every other embedding model is going to force you to pay tokens during your development cycle to hit against their embedding model. So, as you're querying, as you're taking your queries and you're, you're passing them through the embedding model to do your searches. But with us, you can use the Nano there. You can run that on your laptop if you wanted to. So, you

• 1:29:06Speaker 01

  could, if you wanted to, have zero token costs aside from the embedding of the corpus of data. But for your queries, you could eliminate your, your token costs during development in a way that wasn't possible before January. So, that's not for everybody. You do take a little bit of a retrieval hit when you use a different, the retrieval quality hit when you use a different model. But it, for some customers that don't want to have to, that want to lower their token costs during development, this is one way that they can potentially do that. So, that's, that's shared embedding spaces, which we introduced in January. And more broadly, I want to come back to the idea that we've been building databases for a really long time. We have not been building agents for a very long

• 1:29:47Speaker 01

  time. We're, we're going to continue to see iterative improvement, and it's going to happen fast. Like, MC, published MCP on like Monday before Thanksgiving two years ago, and by March, all of their competitors had embraced it as a protocol. Token Maxing first got mentioned in like early March, and by April, it was over as like a conversation piece. Like, the quickness, the speed at which we have these conversations and we go through these cycles of these different features, it just, it's faster than it ever has been before. So, the fun part is there's still a lot we all have collectively to learn. We're going to all collectively figure out how to make this easier for everybody, and

• 1:30:32Speaker 01

  you'll see more and more agents in everyday life as a result.

• 1:30:37Nathan Labenz?

  Well, as field CTO of AI at MongoDB, I know you're going to be racing as hard as you can to keep up with all these cycles, and we'll continue to look for new updates and more great progress from you. Pete Jazin, thank you for being part of the Cognitive Revolution.

• 1:34:40Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:35:21Nathan Labenz?

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.