Chroma and Agentic Retrieval

Data retrieval is a fundamental challenge in AI systems, and the approaches for solving it are still evolving. Vector search was an early answer to the retrieval problem, but the rise of agentic systems has raised the stakes considerably. Agents issue queries at machine speed, decompose complex questions into parallel searches, and require retrieval infrastructure that can keep pace without becoming prohibitively expensive. Chroma https://www.trychroma.com/ is a company building open source infrastructure for AI applications, best known for its widely used database of the same name. The company also published the influential Context Rot paper, which documented how model performance degrades as context window utilization increases, and recently released Context One, a 20 billion parameter retrieval sub-agent trained to do agentic search at frontier model quality but at an order of magnitude lower cost and higher speed. Hammad Bashir https://x.com/HammadTime is the CTO of Chroma, with a background spanning machine learning, computer vision, and data systems. In this episode, Hammad joins Gregor Vand to discuss the origins of ChromaDB, our current understanding of context rot, why a purpose-built small model can match frontier models on search tasks, the philosophy behind Chroma’s open source approach, and where the company sees AI data infrastructure heading. Sponsorship inquiries: [email protected] mailto:[email protected] The post Chroma and Agentic Retrieval https://softwareengineeringdaily.com/podcasts/chroma-and-agentic-retrieval/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Are you passionate about software development and the tech industry? Software Engineering Daily is looking for a new podcast host to grow its hosting team. In this role, you'll help shape the show's editorial direction and interview engineers, founders, hackers, and tech leaders. Podcasting experience is a plus, but not required. Curiosity, great communication skills, and a genuine interest in the craft of building software are what matter most. If this sounds like you, reach out at [email protected]. Data retrieval is a fundamental challenge in AI systems, and the approaches for solving it are still evolving. Vector search was an early answer to the retrieval problem, but the rise of agentic

• 0:45Speaker 00

  systems has raised the stakes considerably. Agents issue queries at machine speed, decompose complex questions into parallel searches, and require retrieval infrastructure that can keep pace without becoming prohibitively expensive. Chroma is a company building open source infrastructure for AI applications, best known for its widely-used database of the same name. The company also published the influential Context Rot paper, which documented how model performance degrades as context window utilization increases, and recently released Context One, a 20 billion parameter retrieval subagent trained to do agentic search at frontier

• 1:26Speaker 00

  model quality but at an order of magnitude lower cost and higher speed. Hammad Bashir is the CTO of Chroma with a background spanning machine learning, computer vision, and data systems. In this episode, Hammad joins Gregor Van to discuss the origins of Chroma DB, our current understanding of context rot, why a purpose-built small model can match frontier models on search tasks, the philosophy behind Chroma's open source approach, and where the company sees AI data infrastructure heading. Gregor Van is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies.

• 2:09Speaker 00

  He is based in Singapore and can be found via his profile at van.hk or on LinkedIn.

• 2:28Speaker 01

  Hello and welcome to Software Engineering Daily. My guest today is Hammad Bashir from Chroma, so nice to have you.

• 2:35Speaker 02

  Yeah, great to be here.

• 2:36Speaker 01

  Yeah, awesome. So, we're going to be hearing all about Chroma. I think it's a product that probably a lot of our audience are familiar with and probably familiar with these types of products as well. But before we get into that, I'd just love to hear, as we like to, Hamad, you've worked at quite a few interesting companies, I think in the Bay Area and beyond. What's your quick path to getting to Chroma and all that?

• 2:58Speaker 02

  Yeah, I spent better part of the decade as a software engineer and broad expertise in machine learning, computer vision, and data systems was most of what I worked on for the past decade. And before this, I was spending a lot of time specifically on computer vision. And initially, when we started on Chroma, the problem that we were solving is one of active learning for computer vision systems, which is the problem of every day you have a bunch of live-deployed computer vision systems. How do you actively label the data that you're looking at in the wild and select which data is worth labeling? Because if you label everything, it's cost prohibitive. So that process of choosing what data to label is a train-time data curation problem. And as language models started to emerge, we started to look at that problem in the

• 3:43Speaker 02

  context of inference, which is how do you actually curate data for models to do inference on? And specifically, language models, this is the problem of search. How do you actually find the most relevant data to enhance the quality of their answer? Which is the stack of tools that we now build at Chroma, is a stack of tools to help people develop search for their downstream AI applications so that they can get the right information or the right context into their language models.

• 4:07Speaker 01

  Yeah, nice. So, yeah, interesting background leading into like how, I guess, everything that you guys are now doing at Chroma. Maybe just a sort of easy introduction, what is Chroma? There's a few bits to the product, but like roughly speaking, what is Chroma?

• 4:21Speaker 02

  So Chroma, at its core, is trying to solve the problem of getting models the context they need. And to that end, we build three main pieces of technology in our platform that people can put together to solve that problem. The core of it is the product that most people use, which is our core database, so Chroma DB, which is now one of the most widely-used databases in conjunction with language models. What it does is you can very easily ingest data into it and use a variety of different search techniques, vector search, full-text search, metadata search, in order to get information that might be relevant to a language model. In conjunction with that, we also have a part of our product we call SINK, which is a managed ETL pipeline for ingesting data from various sources into that database.

• 5:05Speaker 02

  We found that most developers don't want to be building that sort of a pipeline themselves because putting data into a search database requires a lot of processing. You have to clean the data, you have to transform the data in a way that makes it vectorizable so you can search it, you ideally want to enrich that data with metadata, and that process is quite cumbersome to do in a high-throughput way, so our SINK product solves that. And then, quite recently, earlier this year, we launched a product we call Context One. It's mostly a research endeavor, and we're in the process of productionizing it. What it is, is a model that does agentic search. So agentic search is the process of, instead of a language model application just doing a single search and then putting that search into the context window directly, a process commonly known as RAG, or Retrieval Augmented Generation, what you can now do is tell

• 5:50Speaker 02

  a language model, please take this query, perform a multiple set of queries, take some high-level English query, turn it and decompose it into a set of low-level queries against some search system, and then repeat that process until you feel comfortable with your end conclusion, kind of the way you and I would sit down and answer a query. We wouldn't type something into Google, look at the first 10 links, and then say, my job is done. We would read some documents, author more queries, read some more documents, author some more queries, follow the links on those pages, and do exhaustive research. The models now do that process themselves. However, it's quite cost prohibitive to do with large frontier models. So Context One is a model that was trained using reinforcement learning to be just as good at frontier models at this process, but at an order of magnitude less cost and

• 6:35Speaker 02

  an order of magnitude more speed. So compared to OPUS 4.6, we run at around 400 tokens per second, 500 tokens per second, whereas something like an OPUS would run at around 40 to 80 tokens per second. So the end result is that you get your answers at the same quality, much, much faster, and for much, much cheaper.

• 6:51Speaker 01

  Yeah, and we're going to go into Context One a bit later on, so we'll get to hear all the interesting detail of, I guess, how that actually works in practice. So, I think just looking at Chroma generally, when I first came across this space, vector search was such a big thing for a while, and then, at least in my framing, it kind of disappeared for a while, but now it's coming back in what I feel is quite a big way. So if we kind of start there, like, I guess, the journey of Chroma from its founding, I guess, or when you came into Chroma, and what were you working on at the start, and like, how has that evolved?

• 7:23Speaker 02

  Yeah, so as I mentioned, what we were working on at the very beginning was how do you choose what data is worth retraining models on? So the problem of, hey, every day, my models do some task out into the world. At the time, it was actually computer vision models, and some subset of that data is worth relabeling for the process of what's called active learning. And it turns out that that same process can be used for language models at inference time. So how do you actually choose what data to put into the context window? Well, you do search, and you find related information, the same way that you might find related information when choosing what data to label at train time. That was the initial problem that we went out to solve. And actually, the way we went about solving it was using existing vector search solutions.

• 8:07Speaker 02

  We didn't set out wanting to build a database. But what we found is that every other solution on the market was quite poor in terms of serving the workload that AI applications had. Specifically, systems that were built for legacy search, not for AI applications, were really designed around the idea that you have one very large index. So you might have a billion pieces of information to search over, a million documents to search over, but they're all in one index. And if you want to add another half a million, what do you have to do? You have to recharge your cluster, you have to add a node, subtract a node, rebalance across those nodes, which is a very manual and operationally-intensive process. AI applications have an almost opposite workload, where instead of one or a handful of very large indices,

• 8:51Speaker 02

  you have extremely partitioned indices. So you might have an index per customer, an index per team, an index per chat history, an index per agent. And that means that your system has to be able to store a very, very large volume of unique indices and be able to search over them with a high degree of accuracy. And if you're to store all of those in one index, one, you have to pay the cost of keeping that live at all times. And then, two, you have to deal with the accuracy degradation in vector search when everything is in one index. Vector search doesn't do well with being partitioned. And so the best thing to do is hard partition the data, keep it actually in logically separate indexes. You can't filter the search at query time. You have to do it at index time. And so if your data has this partitionable structure, you can build many, many indexes.

• 9:36Speaker 02

  And instead of needing to build one giant index, you get a much higher quality and cost trade off. And so that's actually why we started to build Chroma the database is because nothing else solved that problem for us in a high quality way. And the core of it was, at the time, object storage. So Amazon has S3 and GCP has GCS. Azure has their own thing. All started to release support for conditional rights, which is a new thing, especially on AWS. It only came out as recently as 2024. And conditional rights allow you to actually build a database on top of object storage. And so what we did is leverage that functionality in order to be able to build our database, ChromaDB, on top of object storage, so that the economics would be something

• 10:20Speaker 02

  that make applications for AI actually viable. Now you can actually afford to have indices for every customer or for every team or for every agent. Whereas before, you would simply run out of money trying to even build some products that people were trying to build on triply replicated pairs of disks or NVMe SSDs. So by building a database solely on top of object storage and then providing a really easy to use user experience on top of that, we just built that initially for ourselves, for us to be able to serve the workload that we wanted to serve, which was, hey, how do we choose what data to put into language models at runtime? Because nothing else quite was doing the job.

• 10:56Speaker 01

  And we said this is in forest to today, and we'll get into it in more detail. But the change to agents doing the searching here, I think, has had quite a big impact then on how that retrieval effectively has become even harder, if I'm not mistaken.

• 11:11Speaker 02

  Yeah, absolutely. And I think part of it is now the agents don't just issue one query. They'll issue five queries in parallel. And the agents don't query at a human rate. They query at an agentic rate. And so whereas you and I might issue a query once every 30 seconds at most, and that would be considered quite fast for us to be issuing queries, the agents will gladly issue a query every one to two seconds. And each of those queries will be five queries in parallel or six queries in parallel. So this agentic process also puts a much more heavy burden on the read side of the database. And if you build a system that is fundamentally stateless in its core architecture, in that you can just add additional nodes to supply queries because you have a shared storage layer. So I can just add an additional node to serve reads.

• 11:56Speaker 02

  You can absorb that elasticity in a way that I think systems that are built otherwise can't. So I think the ability to also leverage shared storage, not just to deliver economics that work, but also to deliver a performance envelope that works is something that actually we didn't anticipate when we started, to be completely honest, but we benefited greatly from after having built it.

• 12:19Speaker 01

  Yeah, amazing. I'd love to go into the journey, especially towards Context One. And there was a paper that Chroma put out last year called Context Rot, which I think has become, especially in certain circles, very respected and understood for what it uncovered. This is back in, I think, June or July 2025. Could you just walk through the core of what the Context Rot finding was, like especially for anyone that hasn't read the paper?

• 12:46Speaker 02

  Yeah, I think Context One was our attempt to prove something to ourselves empirically and scientifically, what most people already kind of knew on a vibes basis. Most people already knew of this degradation. But Context Rot was formally an investigation into how does model performance degrade as context window utilization increases? So as we add more and more tokens to the context window, does the model degrade in its ability to perform certain tasks? And we performed a series of very simple and targeted experiments where we do things like inject a repeated word a bunch of times and then ask the model, where are you seeing the word stop appearing? Or we do simple reorderings of needle in a haystack problems where we give it a bunch of documents and we say,

• 13:30Speaker 02

  can you find this specific piece of information in this haystack, which is a needle in a haystack problem? And we perturb it in interesting ways. And by running these experiments, what you quickly see is a super sharp drop off in the tens of thousands of tokens. And this holds even in today's frontier models. It's a problem that has persisted even, we released the report last year, and we've had like three model generations or minor iterations of the models since then. And we still see that degradation in the models that as you increase information to their context, they start to degrade in quality. And what this points to is the need for a disciplined approach to deciding what you're going to put into the context window.

• 14:13Speaker 02

  And whenever there's some new paradigms in computer science, people are very quick to ascribe terms like so and so engineering or so and so science. But we have, I think, been okay with adopting the term context engineering, which is the problem of how do you actually choose what information to put into the context window to maximize the chance of success in the next turn or the next set of turns of the language model? And I think that process is one that requires a lot of care in choosing not just like what queries you make, but also how you structure the data, how you prompt the model, what data you give it access to in order to not suffer from this problem of context rot. Because even if you solve the problem of context rot, because of the way that models work, it's quite cost prohibitive to just assume

• 14:58Speaker 02

  that you can grow the model's context window ad infinitum. At some point, the matter of cost also becomes a concern, which is, well, if you can more carefully manage the context window, then you can also deliver much better economics. And as we all have learned the hard way, it's very easy to go crazy spending a lot of money in these systems in a very short amount of time.

• 15:18Speaker 01

  If I think back to, I think it was November 24, something like that, when RAG was suddenly like a buzzword and people barely even understood this idea of, okay, if I add a document basically to my prompt and then tell it to do things. And nowadays, this context engineering is simply like the way it has to be done. Like, there's no way you can kind of make progress on something without having multiple sources feeding into what you're trying to produce, whether that's code related or especially in my case, using a ton of work documents effectively to all kind of come together. Just going back to Context Rot for a second. I mean, this was tested, I believe, like across, again, we're talking a year ago, so I mean, things moving incredibly fast. But 18 models, I think, this was tested across.

• 16:03Speaker 01

  So like, I guess, all the big models from the people we know, like Anthropic, but a lot of open weight models as well. Is that right?

• 16:10Speaker 02

  Yeah, that's correct. And you can also see like very sharp differences. At the time, I think the open weight models have caught up tremendously in the past year. But at the time, you would even see a much sharper discrepancy between the open weight models and then the kind of frontier closed labs. And I think the thing that's been most interesting to me in terms of the open weight models is how quickly the gap has closed. And we run the benchmark internally still, just for curiosity's sake, every time a new model comes out. And the gap continues to close. Where actually, in some cases, on some tasks, the open weight models actually can outperform some of the closed weight models. So the gap between the open weight models and the closed weight models, at least in this very narrow and contrived benchmark, Context Rot is obviously a very contrived set of experiments.

• 16:54Speaker 02

  But with the way we view it as a test bed or a gym to evaluate a very specific behavior, the open weight models have made a lot of progress in the last year, too.

• 17:02Speaker 01

  Yeah, and you touched on it there, the needle in a haystack problem. The findings were basically that a sort of shuffled in coherent haystack can actually help models find needles compared to the structured ones. Like, can you talk to us a bit about that?

• 17:17Speaker 02

  Yeah, I think the general observation is that as we add distractors to the context window, the ability of the model to ignore distracting information or information that looks like it could be correct decreases. So models are quite bad at disambiguating between two facts that aren't in contradiction, but that are about the same thing. Meaning, I have two facts that are about the same thing and seem related, and they point to two different aspects of that same thing, and they don't necessarily agree or disagree. And they maybe distract from the core thing that you're trying to talk about. So as a concrete example, if I'm asking a language model, can you find for me the population of Georgia? And I give it a document that has both populations of the country,

• 18:00Speaker 02

  Georgia, and information about Georgia. And then I also give it things about the state of Georgia. And I've specified, I want to know about the state of Georgia. It may confuse those two simple examples between each other. Now, that's a very simple example. But the idea is that like, if you have things that seem on the surface quite similar and you mix them in, the ability of the model to discern between those degrades as you increase more distractors. And some models are quite good at ignoring information that seems relevant but isn't really. And then some models are quite poor at it. But universally, all of the models do suffer from this, where as information looks distracting, it starts to degrade its ability to reason if it just seems like it could be relevant. And you can really quickly start to imagine how that's relevant in functions of the real world. These are very contrived examples that we test on.

• 18:45Speaker 02

  Most of them are also synthetically generated. But when we look at the real world, you can imagine simple cases where a model retrieves documents about two people with the same name, and you're talking about work that was done by one person, and it starts to confuse one person's work for another and points you to ask a question to the wrong person inside of your company. Or you can imagine when a coding agent looks at files, it starts to quickly discern the wrong area of the code base to narrow in on because it seems relevant, because maybe you're asking about the authentication system. And there are two authentication systems that live in different places. And you're referring to one and it confuses it for another. So it happens all over the place that the model can get confused between two things that seem similar but aren't actually related in the way that you're looking for. And so that's kind of what we were trying to study is,

• 19:30Speaker 02

  how much are the models able to parse through and disambiguate between pieces of information that seem similar but aren't actually similar?

• 19:38Speaker 01

  Yeah, super interesting.

• 19:40Speaker 00

  Every visitor on your platform looks the same at first. Can you trust them? Or are they a threat? Fingerprint lets you answer this question right away. With one API call, no model to maintain, and implementation in minutes. Fingerprint recognizes bots, AI agents, VPNs, tampering, and over 100 more unique signals to give you durable risk scoring and threat detection in real time, even as human and agentic activity blends together. You can identify suspicious traffic faster and build safer and more secure sites and apps. Over 6,000 companies rely on it every day. Join them now at fingerprint.com. Think about your mobile app's source code.

• 20:24Speaker 00

  Once it hits the App Store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CI/CD pipeline. We're talking polymorphic, multi-layered code-hardening techniques and automated runtime application self-protection, paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed. Secure your mobile applications today.

• 21:10Speaker 00

  Go to guardsquare.com to learn more.

• 21:14Speaker 03

  You're listening to Deepgram Flux TTS. Different voices, same model, all ready to speak.

• 21:19Speaker 04

  Flux TTS is a streaming text-to-speech model built for voice agents.

• 21:24Speaker 03

  Flux reads the room. It holds context across the conversation, turn after turn.

• 21:28Speaker 04

  With consistent tone, interruption handling,

• 21:30Speaker 05

  and plenty of personality.

• 21:32Speaker 03

  So your agents keep it flowing and customers can just keep talking.

• 21:36Speaker 04

  Try Deepgram Flux TTS free now until September 12. Visit deepgram.com/keeptalking. Terms apply.

• 21:44Speaker 01

  I guess let's go into Context One because this is, with that in mind, everything you just said, like, Context One is how to get over this, get over the context problem. So curious of where we'd like to start with actually with Context One. I mean, we could start, say, with the architecture. Like, I think it's explicitly called a retrieval subagent. Could we just start there? Like, what does that even mean? And, you know?

• 22:07Speaker 02

  Yeah, so the way that we designed Context One to be used is as a subagent with a larger model. And generally, I think this is a pattern that you'll already see a lot of the coding agents do now, which is larger coding agents, Codex, Claude Code, Cursor, and then a lot of the open harnesses as well, will launch a subagent, which is just another language model that then goes off and does some task. In this case, the task that we're interested in is, can you help me find all the information that may be relevant to doing the function that I'm trying to perform? Can you get me the context that I need? And so Context One is designed to be a generic search subagent for any large reasoning model. And the idea behind it is that, as in traditional computing, can we just separate concerns and give tasks

• 22:52Speaker 02

  to the right system at the right time? And in this case, our belief is that the narrow problem of search is one that can be solved pretty easily by a smaller model because it's mostly the problem of understanding user intent, decomposing that into queries, analyzing relevance of documents, and then performing that in an iterative fashion. And in most cases, you don't need that much raw intelligence as much as you need the ability to do a lot of that very quickly. And so for us, tokens per second is really the way that we think about it. And, you know, as we start to think about generations of Context One beyond the current, we're even starting to think about, how can you start to leverage parallelism and sharding to actually attend to very large corpora? And so its use as a background subagent

• 23:34Speaker 02

  is mostly driven by this idea that we should separate concerns where it makes sense for performance reasons. And we dispatch this background subagent. It just finds the set of documents that it thinks are relevant to a task and then returns them back to a larger reasoning agent. And that larger reasoning agent can then go about doing its function.

• 23:50Speaker 01

  Yeah, I think exactly that sort of separation of concerns is super important because at the end of the day, this is not something that will actually answer questions. This is a, as we were saying, it's a retrieval subagent. Its sole purpose is to go and understand what to go and find and return that, but then hand that off.

• 24:08Speaker 02

  Exactly. And I think that, like, you can even start to look at approaches that ask the large reasoning model to be more clever in its use of subagents and actually start to internally decompose tasks into multiple subagent calls and then get all of those things back in parallel. And so leveraging the large model as kind of an orchestrator and a planner over a fleet of smaller models is, I think, a paradigm that we believe in quite firmly. And it's already proving out to be how coding systems will work. And I think it'll find its way into almost every agent over the coming year. And so Context One is just trying to fit into that paradigm.

• 24:44Speaker 01

  So I guess looking again, what actually makes up this subagent? I think there are like four tools. They've got Vector Plus, BM25, Hybrid Search, RegExGrep, Redocument, and Prune Chunks. And there could be a lot to cover there, but I think more to the point of like, why those four specifically?

• 25:03Speaker 02

  Yeah, I think, I mean, the standard paradigm for any agentic search harness is you just want to give the model two tools, which is one, allow it to search. And the second is allow it to fetch results that it sees. So give it away to get its 10 blue links from Google and then give it away to click the link. Those are the two things that you're trying to give it. And then the one thing that we were interested in is this idea of self-editing context or active context management. So the predominant paradigm to managing long context windows is compaction. So if you've used any coding agent at the end of your context window, it'll summarize or compact its context. Our belief was that for leveraging a small model, this would be a very difficult endeavor. It would be quite lossy.

• 25:48Speaker 02

  You would potentially lose valuable information. You might lose context if you're being dispatched by a larger agent in a narrow fashion. And so how could we avoid any lossiness in the small model due to its inefficiencies and its inaccuracies? The idea that we came to is can we just have it edit its context as it goes? So we give it this prune chunks tool. And doing agentic search is a very token-heavy process because every document that you touch, you end up loading into the context window and you very quickly process a very large volume of tokens very quickly. What we do is we give it this tool that allows it to go back and remove documents from its own context window. And this is a trade-off between KV cache coherency and the overall token usage. And what we found is the model is quite good

• 26:33Speaker 02

  at making judgments about relevancy and removing prior information from its own context window, which is not all that surprising in hindsight because the model already has to make that judgment in order to be successful at its task. The fundamental problem that you run into, though, is sometimes a document early on in the search rollout appears to not be relevant, but then later becomes relevant. And so a lot of the subtlety of training the model becomes, how do you actually train it on a wide enough variety of cases that have a wide enough variety of problems that have different amounts of relevancy in the documents that it returns? How can you perturb the documents that it returns in a way that you can create sufficiently challenging examples for the model such that it's trained to be very accurate in when it prunes?

• 27:18Speaker 02

  So when we trained Context One, the base model is actually GPT-OSS-20B. So it's a 20 billion parameter model. And when we trained it, initially, the accuracy of pruning, I think, was around 80%. So 80% of the time, it selected the right documents to remove. So that basically makes the prune chunks tool for the base model kind of a non-starter. You're going to have an immediate 20% accuracy loss. By the end, I think we were around 95% plus accurate on a wide variety of tasks after training. And I think on our most recent checkpoints internally, it's like closer to 97, 98% accuracy on the prune tool. And most of the loss in accuracy comes from the search itself. And the model itself is quite good at discerning what's accurate and relevant versus not. So that ability to self-edit the context

• 28:03Speaker 02

  ends up leading to a very large reduction in the overall token usage. And what that does is it also makes it so that you can do inference a lot faster. Because we're keeping the context window relatively bounded at any given period of time, around 32,000 tokens, the pre-fill phase, which is the first phase of language model inference, ends up being relatively fast. And then the decode phase is relatively fast as well because we're keeping it fairly bounded at all times. So we can afford to pay this repeated pre-fill cost because we're pruning the context and the model is small enough that we don't kind of feel that penalty from that trade-off. And there's obviously some subtlety there in like, how much should you prune versus not? When should you prune versus not? And we're also starting to work on, can you actually like use latency as a training objective

• 28:48Speaker 02

  and let the model actually self-optimize around latency at train time? And so that it kind of learns the most optimal pruning strategy, not from an accuracy perspective, but also from a latency perspective. So there's all these subtle tricks you have to do in order to make it work. But fundamentally, I think the core contributing idea on the harness side of the model, the thing, the set of tools it has and the way that the model behaves, is this idea of self-editing its own context and self-editing its own history.

• 29:14Speaker 01

  I think if we just like look at how, like, what the result of this is, I think it was stated that your Context One at 20B matches, this isn't that long ago, so OPUS 4.5, Sonnet 4.5, Gemini 3.1 Pro, and so most of your benchmarks. That seems pretty impressive. It has an understatement. So I mean, like, what's kind of your read on why a purpose-trained small model can actually get there, I guess?

• 29:41Speaker 02

  I think the core belief is if you start to decompose what a model does when it searches. The set of behaviors it has are more valuable than the reasoning that it's doing. So there's a program that you can follow or an algorithm that you can follow to do search. And a lot of the time that algorithm is a set of very narrow tasks chained together. And if you can just figure out the right time to deploy those tasks. And if you can figure out how to be good at those subtasks, then you can be good at agentic search. Fundamentally, I think the set of tasks is how do I look at a high level user query? And then decompose it into a plan of lower level queries to issue against the search system. How do I look at a page and determine its relevance?

• 30:25Speaker 02

  How do I look at a page and determine what follow up queries to make? And then how do I analyze the set of results I have at the end and determine what could be relevant? Right. Which is a different relevance determination. So I think that set of tasks, which are fairly narrow, you can optimize for in isolation through a set of training examples. And so actually, in in training context one, we have a staged curriculum. Where early on in the lifecycle of training, we only give it problems that are easier and need one to two hops to satisfy. And then later on, we start to like expand the, you know, difficulty of the tasks that it has. And then we also do two phases of training. It's not only trained through reinforcement learning. We also have an additional supervised fine tuning stage, which uses synthetic data and data distilled from open weights models.

• 31:09Speaker 02

  So we take open weights models and we distill their behaviors into the smaller model. And that initial set of behaviors alone actually shows you a very large performance increase. But the thing you're after is grounding the model in the right set of priors around how should I behave when I'm in certain cases? And once you get that into the model, it's pretty easy to optimize. Once you have those basic set of behaviors, even just getting the model to tool call in parallel gives it a huge advantage in its ability to very quickly reduce the number of turns that it needs in order to discover information. And GPT OSS 20B doesn't natively do parallel tool calls. And so if you just, you know, do some amount of SFT or some fine tuning on the model in order to teach it that behavior from the data, it very quickly gets better at that and leverages it

• 31:54Speaker 02

  even before you do any sort of reinforcement learning. So I think the reason that you can get these small models to be good at these tasks is because they don't require a lot of reasoning capacity and they don't require a lot of subtle decision making. And we have done some work to show that for ourselves, we've done some work to show that, you know, oftentimes having world knowledge does help you. Like if you, if you know, you know, information about a given topic and you've memorized information about a given topic, you're going to be able to issue better queries. And so a larger language model will probably always outperform a smaller one at study state. But the smaller ones are cheap enough to fine tune using Laura that on narrow enough corpuses, you can very quickly also fine tune a small model on a narrow corpus and make it better at individualized corpuses

• 32:39Speaker 02

  and personalize the model for specific tasks. And I think that's where the advantage really starts to shine is can you actually just give it a map of the corpus or teach it the vocabulary of a corpus so it's able to search it better? The same way like if you and I go and search our Google drives, we already know what's in there. So we're going to be able to search for it more effectively. You can see that sort of behavior in smaller models as well.

• 32:58Speaker 01

  Yeah, that's really interesting. I think just to wrap up context one, the bit that might not be clear or surprising even to some people is that you actually released the weights, open source, like under Apache license. So what was the thinking there? I mean, on SC daily, we're always very pro open source, that that's fantastic. But every company has probably gone through a thought process of why to do that.

• 33:21Speaker 02

  Yeah. So Chroma is an open source company, like even our core database is Apache 2.0. I think there's many reasons to do open source. Before Chroma, I wouldn't say I was the largest, you know, open source zealot. But I've over the years, you know, become a card carrying open source member or whatever you want to call it. Where I'm quite zealous about things being open source. So I think there is like some amount of just personal belief and philosophy inside the company that things should be open source where they make sense. And I think it's fundamentally like, one, we wanted to learn very quickly. And more people are going to use the model if it's open source and tell us what's working and what's not working. More people are going to be interested in building with it. More people are going to be interested in integrating it into other places where we may not be able to integrate it.

• 34:05Speaker 02

  And also the model is very small, so you can run it locally. And the way that Chroma initially grew as a product in the market was we offered a local version of the database. And so we're quite fond of solutions that meet developers where they are, which is on their laptops or on their machines at home and let them, you know, tinker around and experiment with technologies that are close to the frontier in a way that doesn't cost them an arm and a leg in order to play around with it. And maybe it won't be the case that every subsequent model we release is open source. But at least for the time being, I don't see a reason for us not to continue to do things in an open source fashion. I think it's positive some. I think that we learn when other people release their work, we benefit greatly from it. And we obviously benefit from open weights models, both quite literally in the training of context one,

• 34:50Speaker 02

  but then also just in our usage of them. And so I think mostly for us, it's just a matter of getting people to use it, learning very quickly and then contributing back to an ecosystem. And our belief is that our ability to grow that ecosystem at this moment in time. AI is still very early, even though it feels like the world is almost over or whatever people on the news want to tell you. AI is still very early in its lifecycle. And I think that right now is not the time to be zero sum in our thinking with respect to it.

• 35:18Speaker 01

  Yeah, that's great. I mean, we've talked to quite a few companies who have decided to even move from closed source to open source over the last year, for example. And I think we're seeing more and more of that. Either the company just starts day one that way. I work for a company that started that way as well. Or actually, it doesn't look great, actually, if you're working on anything sort of on the frontier that is all closed source. So and in Anthropic's case, you can be closed source and then somebody leaks the whole code anyway. So so you might as well be working in the open. So definitely support that. If we just look at the product stack of Chroma, this database, there's sync and the agent product. Like, could you just kind of walk us through where does a developer start maybe? Or like, it does not matter, it's just the right tool for the job?

• 36:03Speaker 01

  Or is there some classic arc you see of like how a developer starts and then adopts more of the product?

• 36:09Speaker 02

  Yeah, I think most people, their first interaction with Chroma to date is the local embedded version of the database. So you pip install a package and all of a sudden you're in a Jupiter notebook or you're just coding in TypeScript and something is running locally on your machine. And then we have a CLI that lets you, once you're done prototyping, if you need to upload your data. Because unlike SQL databases or like traditional, you know, the use cases that relational databases serve, it actually does make sense to have this notion of pushing data up from staging to production. With these kinds of applications, you might curate some sort of knowledge base or curate some data set and then push it up. So that's the place that most people start. And then they use their CLI to upload the data to our cloud product. Once they're ready to start going in production.

• 36:54Speaker 02

  There's two ways they can do that. Either they can use the database directly if they want to control their chunking strategy and their embedding model choice. And, you know, choose all of these nuanced parameters. But over the years, I think things have kind of coalesced onto a set of best practices. You should use sparse vectors with Splayed. You should use a dense vector embedding model that's of a sufficient quality, but not too large. Turns out you don't need 4096 dimensions. You're very much okay with a smaller dimensionality embedding model. I mean, you'll get just as much performance in terms of accuracy at a much lower cost footprint. If you want to just have those choices all made for you, you use our sync product. And you just, you know, you can either dump your data in an S3 bucket or upload a file or connect it to one of our supported sources. So like GitHub, a website.

• 37:38Speaker 02

  And then, you know, we're soon releasing support for a bunch of other upstream connectors. And then we'll just automatically keep those sources in sync with your database, with your Chroma. And then that's kind of the normal arc. Context one then becomes something that people, you know, once they start to think about, how do I actually really get high quality search in a low latency manner? They usually start adopting that part of the stack. I think agentic search is still new enough that people tend to be adopting it fairly late into their product development.

• 38:10Speaker 01

  But I guess, do you see that maybe accelerating into the future?

• 38:14Speaker 02

  Yeah, now people are coming to us, especially after we released it. Now people are coming to us and it's kind of the starting point for their product development. And that's been really interesting too, which is, you know, how can we? Okay, you actually start to build your product differently if you start with that frame of mind. If you start with like, no, an agent is going to do the querying, you start to actually build your product differently. So I've had people come to me and be like, you know, well, the agent's doing the querying. Why should humans do any of the editing? And so actually they have products now where like, you know, users actually don't edit information directly. They're only allowed to edit through the model because they think that the model will be able to do a superior job of traversing the information space and making all the right edits in the right place and keeping all the information in line. I think it's also changing how people think about their product development.

• 38:57Speaker 02

  This notion of like, you know, not everything needs to be a set of checkboxes and forms and sliders. Actually, you should just drive everything through natural language and push everything into the agent.

• 39:06Speaker 01

  Yeah, I definitely see the same thing where I sit. And it's definitely just an interesting time to be building products where every conversation is. Is this a human or an agent? When you're talking about how it's being used, we're always saying, is it the user or is it an agent? And how does that differ? And I just find that super fun and interesting that that's kind of where we are now when it comes to building product. If we sort of look at like how a larger enterprise would want to use something like this, you guys have quite a bit of material on what's called BYOC, so Bring Your Own Cloud. Could you like, talk to me a bit about when in Chroma's journey did this come about? And why? And small anecdote from talking to someone who is in the current YC batch.

• 39:47Speaker 01

  And they were actually asking me and saying, oh, we're being pushed to, you know, by a certain big customer to do BYOC. And I said, really think about that strongly because that's a big undertaking. And of course, they're going to say they want that, but don't take that on lightly. I think you could probably help expand on why not to take that on lightly.

• 40:09Speaker 02

  Yeah, I think Databricks was kind of famous for doing this like hybrid deployment model early on in their lifecycle. And I think when they asked the CEO of Databricks, what are two things you would do differently? One of the two things was, he was like, I wouldn't do the hybrid deployment model. I would have just skipped straight to the, the other way of doing it, which is maybe kind of privileged hindsight, right? Maybe you wouldn't have succeeded if you, if you hadn't done the hybrid deployment model. So BYOC for us means we deploy our code into a customer's VPC. And then we get back some amount of observability into that system in order to operate it in management. And we keep our control plane on our side. And then we're able to kind of monitor the health of the system, perform operational procedures, updates, upgrades, scaling, some basic logging and telemetry comes to us.

• 40:52Speaker 02

  And then you get the experience of a managed service, but you don't have to deal with actually running the infrastructure yourself. It's a very easy setup. You just kind of, you know, apply some Terraform and run a Helm chart and you're up and running. And then our team kind of handles the day to day for you, all the day to operations. And we actually had the idea to do BYOC from day one. I think if you don't choose to do it from day one, it's really, really, really difficult to bolt on because your entire architecture hinges on the fact that you're doing BYOC. And it actually was an immense pain to convince everyone on the team that this was worth doing. Because it's really painful to build. It's really, really hard to build a system in this way. It makes you have to do things in a much more difficult way upfront, greatly sacrifices your velocity.

• 41:36Speaker 02

  You have to draw boundaries in your system architecture that feel very unnatural to draw at times. Or you could argue, I'd actually say there are boundaries you should have, but ones that you don't, you know, necessarily need right then and there. And I think that if you don't really need to do it, I wouldn't advise that you do it unless it's like quite core for your ability to grow your business. For us, it's really important because the customers that we want to serve that are at a large scale, which is where our product, the core database really becomes useful, is at a large scale. If you have a small volume of data, other solutions are fine, right? If you only have like 10,000 records, you probably don't need scale out horizontal serverless search engine. You're probably fine just throwing all your data in a Postgres database and it will work.

• 42:18Speaker 02

  There's no reason to use our cloud product at that point, like other than just ease of experience. And, you know, just wanting something that really fits AI applications natively. But I think unless you really need to, I would not recommend it. And I think that my advice to this person in their YC batch would be like, make sure you're going to make a lot of money if you do that. Make sure that it's like something that you think you can repeat and deploy because it's, it's a very large distraction. But I do think that if it makes sense to you, it is kind of an increasing trend. And I've noticed that buyers especially are pushing for it, where buyers have started to kind of learn that this is a deployment model. It gets through security, it gets through compliance, ticks all the boxes. And that is probably reason enough to invest in doing it if it can result in a meaningful outcome for your company.

• 43:03Speaker 02

  That said, when BYOC started, I think it was pretty debated whether it was a deployment model that made sense at all. There was a lot of well-known people online who would tell you that, you know, this is the silliest thing ever. No one should do it. It actually makes absolutely no sense. There's blog posts you can go read about this. I think it's one of those things that's a little bit of an organizational tax, right? You have to make the compliance people happy. You have to make the security people happy. You want something that ultimately makes you able to sleep at night and give you control. And I think BYOC is great for that. Whether it's actually secure, more secure or less secure, I think that really ultimately depends. I think that at the end of the day, you still have to do your due diligence and you still have to check on the buyer's side, what's going on under the hood. I don't think it reduces that work or makes it go away.

• 43:47Speaker 01

  Yeah, I definitely agree with everything you've just said. On the control plane side, like, do you get any pushback that the control plane does still live with you? And how do you, I guess, talk to companies about that?

• 43:58Speaker 02

  Yeah, I think there's various degrees of pushback. When we initially started talking to customers about doing BYOC, there were some customers who were quite negative about having any data on our side, even telemetry. And that's just a matter of education. The product can't work if we don't have basic telemetry on the system. So at this point, everything that we store on our side of the fence is just metadata. It's, you know, what data exists in the system? What is the utilization of the system? What is the like version and epoch counters of various pieces of information in the system? Nothing is directly revealing private information. And because it's open source, we literally just send people the code. We're like, here, here's the control plane code.

• 44:43Speaker 02

  You can read the schemas, you can look at it yourself, and you can make your own determinations as to what we're saying. You don't have to trust us entirely. So I think that that ability to just literally in conversation, you know, just send someone the code and just be like, here, this is exactly what I'm talking about. You can poke through it and read it. And now even with agents, you know, people will just quite literally, we have this like Slack bot agent and a customer will be asking us in Slack, hey, can you tell me like what data is stored? And we can just paste the code and then ask the agent, can you traverse all the files related to this? And then compile a list of everything that is related. And you get like an extremely fresh and accurate grounded answer about what data we capture. And it's just points to the code. I think that for us has been how we've alleviated people's concerns about what we're capturing and storing.

• 45:27Speaker 02

  And then the core of it is that we have the idea that no ports are open on the data plane. So everything is done through kind of a reverse tunnel approach where the data plane reaches out to us. We don't ever go in and reach into the data plane. That's also part of it is if you can just come and say like, hey, it's your VPC. By default, no ports are open. Everything comes on a push based model out. So even telemetry, we kind of buffer and then allow it to push out the telemetry. But then also like operations, we don't reach into the data plane in order to do operations. There's kind of a service that runs inside of the data plane that we post a message and then it polls, checks, is there any operation that needs to be done? That gets pulled down into the data plane and then that operation gets performed. And that's kind of like tax, right? Ideally, you wouldn't build it that way.

• 46:10Speaker 02

  That's a very strange way to build some sort of operations thing where you're like pulling for operations and then, you know, applying the operation. So that's a lot of moving parts in order to do a very simple thing inside your cluster, which ideally you would just be able to issue a post request and boom, the operation is completed. Now you have to do it in a slightly different way. Although because we're built on Kubernetes, you can kind of leverage a lot of the hooks that Kubernetes gives you to do declarative progression in the same way and benefit from that. But it's still, you know, it requires a lot more work than you might otherwise want to take on.

• 46:39Speaker 01

  Yeah, I think that's a really good summary. And as you say, I think starting on Kubernetes is a huge help. As an infrastructure product as yourselves, it kind of seems like the only way. But there will be other products out there that don't start that way and does make something like a BYOC deployment even more complex.

• 46:55Speaker 02

  Yeah, if you came to me six months in and you were like, we're doing BYOC, I would have been like, we can't, we absolutely can't. It's too hard to bolt on.

• 47:02Speaker 01

  Yeah, no, that's very interesting. So I'll probably go back to this YC founder and give him some tips on this basis as well. So just looking, I guess, ahead, the future, obviously Postgres is having another moment these days. And I think people will just say, oh, well, PG Vector is like good enough for things like this. Like, where do you see things going as like, are we, I guess, asking the devil advocate here of like, could things converge back to, you know, a frontier large model is actually going to get good enough at this stuff? And context rot isn't such a thing anymore? Or like, how are you guys thinking about it? And like, how are you thinking of building it into the future?

• 47:36Speaker 02

  I think there will always be a need for managing the context of language models. I think the problem of what should I know right now for a language model is a problem that will always exist. In language models today, if you just went to your chat GPT and you were like, tell me my schedule for the day, even if you hook in the MCPs and stuff, a lot of the time, they won't call the right tool. They won't say the right thing at the right time. And so I think there will always be a need for, can you get the right information into the model? And that's fundamentally the problem that we're interested in solving. I think about it a lot like the CPU versus the file system. The language model is the CPU. And then there's a disk off to the side.

• 48:17Speaker 02

  And there's a ton of complexity inside of your hard disk and the interface to that hard disk and the controller and the operating system that talks to it. And then the firmware on the hard disk. I think that that interface is the one that our company believes in, which is there is a language model. It acts as this new frontier natural language information processing system. And then there is a subsystem off to the side that decides what information that model should see and decides what information should be stored and decides what information needs to be potentially updated and consolidated with existing information. That's the system that we in the long term want to build. Today, it looks a lot like a database. But then where we're going, if you can kind of tell with how we're building components on top of the database, right?

• 49:01Speaker 02

  Why is a database company training its own models? Is because our belief is that soon it won't be, I have a database and I have intelligence. It'll be intelligence actually runs inside of the database. And a lot of the way that the database actually performs its function is by leveraging intelligence subsystems. And you need to think about those two things at the same time for performance reasons and also just for domain awareness. It is very cost prohibitive to move large volumes of data around and put them into representations for various downstream processing systems and do all of this conversion and ETL. And then, you know, do all of this reformatting and re-indexing and reparsing. And so if you can build a system that manages that all in one place and makes it very cheap

• 49:43Speaker 02

  to move compute closer to the data by leveraging object storage, by leveraging a shared storage layer, such that if I want to, for example, just run asynchronous language model jobs on my data, I can. If I want to co-locate the node that literally has the query indices with the GPU, such that now my GPU can talk not to a remote service, but talk to a local language model, I can. Because this year, I think it'll be very common for us to, as consumers, use language models that run at 1,000 tokens per second. Last year, we were all running with language models that ran at 100 tokens per second. And so it's not inconceivable that in a couple of years, we're going to have language models that run at 10,000 tokens per second. And so today, all of these search systems make the assumption that a network call

• 50:27Speaker 02

  and the latency to go to your search system is fine, because, hey, the language model is slow. But I don't think the language model will always be slow. And putting these two systems next to each other, such that you can put intelligence and data in a low latency path next to each other, and maybe even in the future, bypass the CPU entirely. Why does the language model need to, you know, traverse indices via the CPU? Can it just directly have a subkernel launch on the GPU that does the search of the index in line in a very low latency fashion, and then feed that data directly back into the language model? I think those sorts of systems will exist. And our goal long term is to be in a place in the market that we can build towards that. I think that that's kind of what we've been building towards all these years, is you should

• 51:12Speaker 02

  be able to put, there's some system that will need to exist, that you can put any data into, any unstructured data, and then it can get any data, and then it can satisfy any query on the way out. And I don't think that system will look like a Postgres. I think that system will look like something much different. And I think it'll look vastly different than what we have today. But if we can get the right abstractions in place and get the right building blocks in place, maybe we'll get an opportunity to earn the right to build that system in the future.

• 51:38Speaker 01

  Yeah, amazing. Well, on a sort of total sense, I think that's a great place to leave it. Two more questions. The first is a usual of just like up and running for a developer. But I feel like, in this case, is it a website? Or should we be saying, what CLI command did you start with? Or, you know, what would be best?

• 51:55Speaker 02

  Yeah, pip install chroma DB, chroma DB run. That's all you need to do to get up and running. But then if you're interested in learning more, try chroma.com.

• 52:02Speaker 01

  Great. And yeah, just a curious one for me. What's in the name Chroma? Where did that come from?

• 52:07Speaker 02

  That is a great question. I think it's just one of those things where it's short, it's easy to say, it's memorable, and then it has a nice ring to it.

• 52:16Speaker 01

  Yeah. You guys have a nice aesthetic as well around the whole product as well. So definitely do encourage people to go to chroma.com and check that out, because I think you guys have done a great job with that as well. So amazing to have you on. So much detail, really learned a lot myself. Today, I'm sure the audience did as well. So really appreciate it, and we'll be following along with Chroma.

• 52:35Speaker 02

  Thank you so much for having me.