RAG & Beyond: Semantic Storage and Retrieval

Anton Troynikov, cofounder of Chroma, joins Nathan Labenz to discuss the importance of keeping the retrieval-augmented generation (RAG) loop in house, what it means for Chroma to be in “wartime” mode right now, and how much of the data going into Chroma has never been in a database before. If you need an ERP platform, check out our sponsor NetSuite: http://netsuite.com/cognitive. SPONSORS: NetSuite | Omneky NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. LINKS: Part 1 with Anton: https://youtu.be/ogy37CdIljg X/SOCIAL: @labenz (Nathan) @atroyn (Anton) @eriktorenberg (Erik) @CogRev_Podcast TIMESTAMPS: (00:00:00) - Introduction by Nathan, setting up the conversation with Anton (00:02:16) - Anton articulates Chroma's mission to build a horizontally scalable system (00:03:06) - Rise in popularity of retrieval-augmented generation (RAG) (00:06:03) - Chroma's focus on delivering a horizontally scalable cloud service for vector search and storage (00:08:07) - Nathan describes his experience building a RAG application for a client profiling use case (00:10:27) - Anton advises measuring retrieval quality and maximizing relevant information returned (00:15:05) - Sponsors: Netsuite | Omneky (00:17:02) - Popular use of open source vs. proprietary embedding models like Anthropic's Ada (00:19:30) - The importance of keeping the RAG loop in house and not relying solely on external APIs (00:27:41) - The huge amount of unstructured data that can now be processed by AI (00:30:40) - Providing a unified interface to structured and unstructured data (00:33:15) - Much of the data going into Chroma has never been in a database before (00:38:47) - Categories of organizations adapting to AI: legacy, AI-native, and AI-first (00:40:55) - Where Chroma is seeing most of its growth right now (00:46:20) - Interpretability work like Anthropic's circuit evaluation (00:52:23) - Anton believes new tooling can make latent spaces accessible without AI expertise (01:06:08) - Scaling constraints between search indexes vs. application databases (01:09:10) - Potential for time as a dimension in embedding spaces (01:13:46) - Likelihood of missing results due to representational issues vs. approximate nearest neighbor (01:15:22) - Automatically handling small data sets without needing elaborate indexing (01:17:20) - Anton's perspective on whether OpenAI will build its own database (01:19:43) - Partnering with OpenAI and other labs to increase use of their models (01:21:19) - Anton's experiments probing GPT's reasoning abilities with Game of Life (01:25:41) - Closing thoughts on the conversation This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Anton Troynikov?

  We are conditioned to think about data as this static thing, right? It's like it's sitting somewhere and it has a particular instance in time. And then we access that instance in time. And then, you know, the next time it might be different, but it's like, it's still essentially, mentally, we think of it as static. I really think of these things more like a loop, right? It's like a, it's like a control system loop, almost, right? Where data is actually an engine. It's something that is interacting with the outside world mediated through computation. And then it's constantly adapting and it's improving itself.

• 0:32Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Anton Trunikoff is the founding CTO of Chroma. We first had Anton on the show back on episode five, which was released in early March, before GPT-4's release. And he impressed me then and since as a real intellectual force. Anton is someone for whom the fundamentals seem to come easily,

• 1:15Nathan Labenz?

  who codes all the time and for whom it's natural to think in higher dimensions and abstractions. He's also a super quick wit, a prolific and at times provocative commenter, and amazingly, at the end of the day, still a bit of a doubter, certainly more of a doubter than I am when it comes to LLM reasoning abilities. In March, I treated my conversation with Anton as an opportunity to speak to an expert tutor on core concepts of embeddings and retrieval. And I tried to explore how he thinks about navigating the latent space as a way to develop my own intuitions and to help you develop yours. If you haven't heard that episode and want something more foundational from Anton, definitely check that one out. This time around, I've wanted to talk about everything that's happened since, including the fact that RAG, or Retrieval Augmented Generation,

• 2:00Nathan Labenz?

  has been LLM's first big commercial application hit, with tons of companies spinning up all manner of document-backed bots for all sorts of labor-saving purposes. And in the process, nearly as many companies spinning up and deploying a vector database for the very first time. We talked about what's working today in retrieval and also what's next, including for Chroma as a business. Now coming in, I was honestly wondering if Chroma and other vector database startups might struggle to grow businesses, given how incumbent database providers are now racing to implement their own vector stores. But Anton did a good job of reminding me of one of my own maxims, that there's a good chance that we're all still thinking too small.

• 2:43Nathan Labenz?

  His observation that most of the data stored in Chroma has never been in a database before suggests that there will be plenty of growth to go around, at least at the infrastructure layer for a while yet. And his plan to bring more and more value into the database so that retrieval becomes as simple as dumping text into a text box, not unlike how people use chatbots today, seems like one that can drive Chroma to a great outcome, even as lots of incumbents get into the game. I just wanted to take one moment to say that I'm really quite proud of this content. This is a focused but fast-moving conversation conducted at an expert level with a genuine intellectual leader currently working as a wartime CTO. I've learned an incredible amount from the process of making this show, and I'm grateful

• 3:27Nathan Labenz?

  that thousands of you from an incredibly diverse range of backgrounds have become regular listeners. I also want to thank the team at Turpentine, including our producers, Vivian Meng and Natalie Torin, our editor, Graham Besalou, and of course, Eric, for putting the whole thing together. Working with Turpentine does make for an unbelievably convenient operation, in which I am 100% focused on understanding, communicating, and interviewing as well as I possibly can. With that, as always, we'd appreciate it if you'd share the show with a friend. I'll suggest that you send this one to a software or an AI engineer in your life. And we always welcome your feedback or other outreach at [email protected] or by DMing me on the social media platform of your choice. Now, here's my conversation

• 4:12Nathan Labenz?

  with Anton Troynikov, wartime CTO of embedding database company Chroma. Anton Troynikov, welcome back to the Cognitive Revolution.

• 4:21Anton Troynikov?

  Thanks for having me.

• 4:22Nathan Labenz?

  Lots to cover. Last time you were here, eight months ago, it was before GPT-4, it was before RAG, broadly speaking, overtook agents as the sort of hottest trend. And...

• 4:37Nathan Labenz?

  You know, it was before, I believe you had wartime and a pirate flag in your Twitter handle. So

• 4:44Anton Troynikov?

  I believe I had the pirate flag. I'm not sure that I had the wartime yet. War time is a more recent one.

• 4:50Nathan Labenz?

  So tell me, you know, what it means to be a wartime startup founder right now in the incredibly fast-moving AI space, and then I'll dig into lots more questions beyond.

• 5:00Anton Troynikov?

  Yeah, I think the thing that caused the switch for me, at least mentally, and of course, you know, part of it is, of course, just branding exercise. But you know, what caused the shift for me mentally was I think that Chroma as a company has a very clear objective and a very clear mandate right now, which we need to achieve. And we are taking on what I think of as the right level of risk for a startup right now, which is a lot of risk in order to achieve that. And so having that concrete objective, knowing what we're risking is a very different place to be in than when you're sort of still in this exploratory mode and you're figuring out what people want. I think we actually have a very clear idea of where we sit in the market, what we're able to achieve. And so for that reason, I think I made the change.

• 5:44Nathan Labenz?

  Gotcha. Interesting. So can you articulate that straight away? Is that something you can lay out for us at this point?

• 5:51Anton Troynikov?

  Yeah, absolutely. I mean, our roadmap is fairly public, so I think it's pretty clear. But the main thing right now is to build a horizontally scalable system on the basis of Chroma's vector search and storage engine, and then deliver that as a cloud service to people. It's the number one thing that people are asking us for. We have a very clear shot on getting that done, especially as the Chroma team has been growing over those months that we, since we last spoke. We know what the shape of it has to be. We know the quality that is expected of us and that we want to deliver to the market. We know our key differentiation. We understand what we're building for. And so all those things are just in place. And it's now, it's like purely execution risk. It's how well can we achieve this mission that we've set out for?

• 6:31Nathan Labenz?

  Okay, cool. I have a lot of questions. Just for context, and this is, you know, this is not meant to be a customer role play, but I think it's kind of naturally, you know, could go a little bit in that direction because I reached out to you with a general kind of Feeling like I need to do some sort of rundown of all things RAG, because it has been the most talked about application development trend of the summer. And it seems to be the thing that is like really working and driving a lot of value for kind of every business that tries it, at least to some degree, right? I mean, you've got, and I think our audience is pretty deep in the weeds of this. I usually don't, you know, sugarcoat it too much on the terminology. But just for anyone, you know, very briefly,

• 7:15Nathan Labenz?

  RAG, retrieval, augmented generation. This is the loop where a user asks a question or, you know, has some sort of input at runtime. A database is searched using that query to bring additional context into the context window, and then the language model can use both the query and the retrieved information to generate stuff. Obviously, a lot of variations on that, but you know, that's kind of been the big trend. And so, you know, Swix had a, an episode on this recently that was quite good. And there was a lot of content about it at the AI Engineer Summit, which I understand you made it to in person. I unfortunately was traveling elsewhere for a wedding at the time. But I'm also doing this, right? So I'm advising this company, Athena, I've mentioned this many times.

• 7:56Nathan Labenz?

  And we have recently gotten to kind of a good, you know, V1 that we're really comfortable putting in the company's hands. And the company's in the executive assistant space. They have a thousand executive assistants with a thousand executive clients, highly idiosyncratic. What we, you know, developed in a RAG lane is an individual client profile backed application that could hopefully help the assistant, you know, kind of, you know, get access to kind of useful content about the client.

• 8:24Anton Troynikov?

  Yeah, perform the work more, more efficiently and faster with all the available information.

• 8:30Nathan Labenz?

  So I don't know if I'm a typical customer or not a typical customer, but we're definitely, you know, looking at this. Boy, wouldn't it be great if you could have this, you know, kind of semantically sophisticated access to all the things, you know, that a client has sort of accumulated or declared or the preferences, the email history, man, could, you know, could go on and on and on. We're the early days of this, but you know, a big part of that obviously is the, is the backing, right? How does, where does this, how does this information get chunked? How does it get stored? How do we access it? How do we assess whether we're accessing it effectively or not? So I've got, you know, a product, quote unquote, in market with an internal group there. And I think what kind of stood out to me most, you know, as a developer. Based on what you're talking about is how do people choose in the first place?

• 9:14Nathan Labenz?

  You know, it feels to me like a very black box. Like people, you said, people are coming to you and saying, hey, I want a hosted service. I honestly can't necessarily yet differentiate between the vector database providers. I'm not even sure how we should think about differentiating.

• 9:30Anton Troynikov?

  Yeah, I think that's a very reasonable question. And I think that's an artifact of the position that we find ourselves in on the adoption curve of these technologies overall. Right? Like so many individuals and enterprise teams and just actually just people building with AI in general are very, very early to this technology. The way that we thinking about it is 99% of people who are going to one day build an AI in the loop application haven't touched that at all yet. And so because we're very early in the adoption curve, differentiation isn't as clear. Right? The differentiation comes into play more as you get further along the adoption curve. And we have users all the way from people building their very first AI project at a hackathon where they're finding out about RAG for the first time, where I have to really educate them about, hey, this is actually possible.

• 10:15Anton Troynikov?

  You can inject information into the model. Here's how you do it. All the way down to pretty sophisticated production deployments where those are enterprises who have the core RAG loop up and running. And they're asking, well, how do I improve my retrieval quality? How do I actually, you know, how do I actually incorporate all the human feedback and evaluations that we've done back into the data layer? Like, what can I do to make sure that the output is improving over time? And there are far fewer teams towards that later end of the curve than there are in the earlier one. And there's an intermediate step here as well where it's okay, my RAG application experiment has succeeded. Now we're scaling, right? You have to scale in the right way as well. So Chroma has differentiation at each point in this curve, actually.

• 10:58Anton Troynikov?

  The first part where it's, we really think of ourselves as enabling experimentation in this space. If we make the retrieval piece of retrieval augmented generation loops really easy to use, and Chroma is definitely the easiest thing out there to get up and running with. You just pip install chroma DB. And actually, these days with our CLI, it can literally just do chroma run and have a chroma server up and running. It's the easiest thing to start experimenting with, right? It's why we're the default in many frameworks. It's why it's so easy to get demos up and running with us is it requires no setup at all and it gives you sensible defaults and it just works out of the box. I mean, with Chroma, you never even need to think about an embedding function. You literally just throw text at it and it's going to work. So we're differentiated at that starting point. And then the thing that we're working on right now, of course, is horizontally scaling this in the right way.

• 11:41Anton Troynikov?

  Today, you can easily scale a Chroma instance, and many people do, actually. They just put it on a machine with a lot of RAM, and then they run tens to hundreds of thousands of collections with millions of vectors each, out of the box, with a single node Chroma. But obviously, for really enterprise-grade data, when you need your entire company to run on one thing, even though your engineers are building different applications with it, you need something that scales horizontally across multiple nodes. And it's important to actually get that right. A lot of other products in the market were built as search indexes, right, in the two or three years. We've talked about this before, but most products that are doing vector-based retrieval were built for use cases that existed before AI became practical. Web-scale recommender systems, web-scale semantic search, the sort of thing that runs

• 12:25Anton Troynikov?

  like Pinterest similar image results, right? That was the application of embedding space retrieval. However, it doesn't scale the same way if you're using this as a component in an application database for a variety of reasons. The first of these, of course, is a search index is mostly data that doesn't change very much over time. You have your billion existing entries, maybe each day, if you're very rich, you add a couple million entries, so it's a tiny percentage of the overall data. Most of the changes that you're making are additions, deletions, and mutations are actually very rare, right? And then the other piece to this, of course, is that in a search index, the entire point is that every user has access to the entire index, right? You want everyone to see all the available images in Pinterest, for example, which is

• 13:08Anton Troynikov?

  fine, but that imposes very different architectural constraints, both for the core vector retrieval algorithms, but also for the way that such a system scales. Scaling and sharding and distributing a single monolithic index, which can then be replicated and written in a particular way, is very, very different to an application stack where you have an index maybe per user space, and all of these are sort of scaling independently and being updated independently, right? Scaling that horizontally, and then providing that as a service, again, is another more difficult challenge, even beyond that. The way that this will express itself to our users, especially when Chroma Cloud is up and running, is we're going to provide completely elastic scaling. Instead of thinking about pods, or deployment, or managing a server, which, by the way, makes total sense if you were building a search index. The way these things are architected is fine if you wanted a search index.

• 13:53Anton Troynikov?

  We will provide this in a completely elastic way. We'll charge you per query essentially and some rate for storage. And then you just never have to think about the scaling part of it at all. But to deliver that requires us to build a quite differentiated architecture from the one that's traditionally been made for these kind of search indexes. Now, that puts us on the scaling point in the adoption curve. So as I mentioned, there's like different points. The final point, and this is something that we're working on in parallel, is basically taking all of those problems around retrieval quality, around how do I chunk my data, around how do I select my embedding model, and bringing those down into the Chrome product. Because the way that I think about this often is like, AI application developers right now are putting a lot of effort into making sure that the retrieval component is working very well.

• 14:37Anton Troynikov?

  And that's effort that they're not putting into like experimenting and actually building products, right? So you often see at the enterprise level, who are further along this adoption curve, you often see data science teams tasked with figuring out how to make their retrieval system better, which to my mind is like, that's really the responsibility of the product itself. It's not the responsibility of the application developer to do that. And so Chrome intends to solve all those problems at the top end, too. That's basically our, that's our starting point. That's our roadmap over the next few months.

• 15:02Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:06Nathan Labenz?

  Got it. I think the last part there is certainly extremely compelling immediately because I agree, there is a lot of effort that is still going into the, you know, am I getting the right things back? And it's not always even, you know, super obvious whether you are or are not. Maybe we could just spend a little bit of time there. I've kind of, you know, developed a couple of, or, you know, gleaned from the literature and validated for myself is probably a more accurate way to say a couple of things that I think are pretty good best practices. But I want to hear, you know, kind of even just with current Chrome, like what you would advise people to do or what you'd advise me to do, you know, to improve on what I've got. And then also want to hear how you're kind of planning to bring that.

• 15:48Nathan Labenz?

  In, I would assume today, like, OpenAI Ada embeddings, I assume have like a huge percentage of the market. Is that true of the usage that you're seeing?

• 15:57Anton Troynikov?

  It's interesting. So actually, what we're seeing is most people are happy with open source embedding models as well. You know, a lot of people are deploying their own local embedding models, especially fairly straightforward to find, you know, like a sentence transformer to get more performance out of than you want. I think Ada are fine, but I think people are finding that there's not like a that significant a trade off between these two. There's other reasons, though, to use Ada embeddings. One of them is that there's an Azure endpoint for them. So if your data is already sitting around in Azure, then it's easy, it's easy for you to do that, right? You don't have to run your own model because you might be running this data at a large scale. You don't have to pay egress costs or network, sort of the network cost penalties of using something like Ada.

• 16:37Nathan Labenz?

  Yeah, I was gonna ask how that, how those preferences kind of relate to scale, because I think I'm maybe atypical in some sense, or I'm in some maybe middle ground, because you're kind of saying, yeah, you know, Chrome is the default because you can just hit run and it goes. That is certainly, you know, great for the very starting point. I often jump like one layer ahead and I'll, I'm definitely, you know, count me among the customers that are like, please give me the hosted version. Because what I typically, you know, the cost is often so elastic on these kinds of products these days, that the entry point is super low, even free. So I can kind of, you know, hack as freely with the hosted version as I can with the, you know, pip install version. And then if

• 17:22Nathan Labenz?

  I do actually start to take it to kind of any sort of mid scale, where I'm gonna have users or whatever, then I, then I'm happy I did that in many cases. Cause I'm like, sweet, now I can pay eight bucks a month and, you know, for as long as this, you know, thing that I've built lasts, like, cool. And then it then kind of, you know, real production scale, you know, kind of comes later. And then I honestly haven't taken Any embedding, any RAG project to that level of scale yet to where I'm like, you know, weighing, you know, the cost of there, I guess the main trade offs. I guess there could be performance trade offs like, but at the high scale, I'm curious to know if you see people like actually outperforming Ada with their own stuff or trying to save money on it because damn, it's cheap, right? So like.

• 18:02Anton Troynikov?

  I think it's not just a question of saving money. I think it's a question of keeping as much of the data and RAG loop in house as you can as well. You know, going out and sending over your data over a network is not necessarily cheap or desirable in many cases. Right? We're seeing actually a very much an increased interest in bringing the entire RAG loop in house, not just calling out to say GPT four, but looking at, you know, Mistral and Lama two and fine tuning those for particular tasks. And I have these conversations pretty often with sort of analysts and other founders and NVCs in the space where there's an open question right now, which is like, okay, where OpenAI is pushing for AGI, they want to develop the one model to rule them all.

• 18:45Anton Troynikov?

  But what utility, if any, do the smaller fine tuned open source models have? And I think that there's actually an interesting inflection point in the market right now where people are saying, okay, well, actually we want to run this entire RAG loop in house and optimize it for our particular use case, but also not give other people our data and not have to worry about that at all. And I think besides the scaling question, because like actually using like GPT at scale is expensive. The embeddings are cheap, but using the LLM itself is very expensive. People are asking themselves, can they keep that entire loop in house? And and sort of keeping the embeddings in house might be a early sort of step in that direction.

• 19:19Nathan Labenz?

  So we'll be back to the just the kind of strategies I've been using. So main one, typical, you know, straightforward, just throw some tokens at it. I do think if you're doing something like this and you're early and you're like trying to get to proof of concept, don't skimp on, you know, the number of records that you retrieve is kind of one obvious, but definitely useful tip, like use the context window.

• 19:42Anton Troynikov?

  Yes and no. So here's the thing, right? This is one of the things where we need to make retrieval work better for it to actually work well in production. It is a fact that has been folklore for some time in this space. However, the recent research has sort of demonstrated this more empirically. Distracting information in the model's context window does tend to measurably destroy the performance of the overall application. And it destroys that performance in actually a very difficult to measure way. Right? So the question is, you know, try to return all the relevant results, but make sure you inside that those relevant results, you haven't accidentally returned irrelevant information.

• 20:23Anton Troynikov?

  And this is how this, this is actually like a fairly complicated system to isolate where the performance gains are coming from. Because obviously the results that you return depend on your chunking strategy. You need to make sure that the chunks that you're creating are semantically meaningful, not just in some general sense, but in the task specific sense that you care about. Right? And your application needs to reason about. So I don't think it's always the case that you always want more tokens, that you always want more retrieved results. You have to find a way to retrieve only relevant sections. But of course, that that depends then, of course, on the embedding model that you're using, the chunking strategy that you're using. All of these things are interconnected.

• 20:58Nathan Labenz?

  Yeah, certainly. You know, at the very high end, I've experienced like Claude 2. If you really push it to the full 100K, it seems to kind of still go off the rails for me. Whereas at like 50K, it performs kind of as expected and like seems to have reasonable command of the full 50K. So certainly at the very high end, I've experienced some of the kind of, you know, too much information makes things go crazy. Do you have a rule of thumb? I've, you know, we started with for kind of token conservation reasons, like two to three results in our first implementation. And then we've now boosted that to like 10. And I think at least on that margin, you know, strictly for us, like more is better. Cause like, you know, the best models can handle the 10 chunks.

• 21:41Nathan Labenz?

  And our retrieval, I trust less than I trust, you know, GPT-4 or Claude 2 to like, find the best, you know, find the right thing in the 10 chunks if it's there.

• 21:51Anton Troynikov?

  So what I would actually do is take a step further back here and ensure that you have a good way of measuring performance. And that can come from two places. That can come from sort of offline evaluation of results, and it can come from human feedback directly in the application itself. I would make sure that I was measuring effectiveness first, like I had a really good measure of that. And then, you know, of course, these are all knobs to tune, right? But the first thing is to have a reliable measure of how well things are actually working in the first place. And again, my advice is straightforwardly, as much relevant information as you can fit, as little, and try to get as much irrelevant information out of there as you can.

• 22:25Nathan Labenz?

  Another thing that I've experimented with, and I think you have really good ideas here as well, is trying to sort of create some sort of adapter layer between the user's original query and the, you know, kind of inter-product type math that the vector database is doing.

• 22:45Anton Troynikov?

  Yeah. So this is, this is an idea that, you know, again, has been around for a while. It's something that I've pointed out many times that it's actually to fine-tune your embedding space. First of all, it's been demonstrated that it's sufficient to fit an affine transform to actually transform one embedding space to another embedding space, right? An affine transform, if you don't know, is just squeezing, stretching, and rotating space. That's all it does, right? And it tends to be enough. Intuitively, what it's really doing is like expanding the importance of certain dimensions of the embedding vector and constricting the importance of other dimensions of the embedding vector under the certain distance space. Now, one way to do that, of course, is to recompute all

• 23:25Anton Troynikov?

  your embeddings and apply that transform. But if you, if you compute the forward transform instead, you can just apply that to the query instead. And what's interesting is that means that you have great flexibility to even apply this transform per user. And you can learn the individual user's preferences, because the same user may be using different, the same data for different reasons, right? And you can imagine Different applications operating on top of the same vector store, which are using the data in different ways. And without modifying the data in the vector store itself, you can just apply these F and transforms to fine tune it per application or per user, which is a really exciting thing. And it's great that the math works out. Of course, the other approach is to to fine tune the embedding model overall. And I think that you can actually probably back propagate information, feedback information from all of the applications or all the users sitting on a

• 24:10Anton Troynikov?

  sitting on your database to just get an overall improvement by performing that fine tuning. And these are features that Chrome intends to provide as well, out of the box.

• 24:18Nathan Labenz?

  Okay, cool. Just in case that sounds, you know, daunting to folks, to be honest, where we are in actual practice is not yet to any fine tuning of anything, but just using the language model to transform text to text. And our kind of instruction for that, which is actually quite effective, is to essentially hallucinate, or we do it a little bit more of sort of a generic, like abstract way. We tell the language model, first, generate what you think this, you know, what the answer might look like. And then it will do that, you know, we put like variables, just kind of placeholders in. And then that, you know, has a much higher chance of hitting the right thing in the database, if it exists, of course.

• 25:00Anton Troynikov?

  Yeah, and that, so that that technique's been around for a little while. It's called hypothetical document embeddings. It's interesting, the the open question there, of course, is that typically general purpose embedding models are already trained in that forward way. In other words, like, they are trained to land queries about information near where that information may be located. I would say that that should be a property of the embedding model itself. But if you're finding that Hyde works for you and it's working in a cost effective way, in other words, you know, you have that extra model call there per query, then if it's useful, then then use it otherwise. Otherwise, it may actually be in the long run cheaper to fine tune the embedding space itself.

• 25:37Nathan Labenz?

  I think, I mean, definitely at this, at scale, I think it certainly would be. That is, yeah, interesting, though you're, it's a very interesting, you know, good note on sort of the forward. Nature of the training and the, you know, the objective function, the whole thing is kind of working in that direction. You know, very concretely, things that we've seen are just like, you know, what are the clients' kids' birthdays? It doesn't, you know, it should work, but definitely getting it way more likely if we're like, you know, kid A is born whatever date, you know, it just seems to pop the race up to the top a little more reliably.

• 26:15Anton Troynikov?

  That speaks to a different possibility here, right? Which is some data is better stored structured in a structured way and some data is better stored in an unstructured way. Now numerical information about individual people is probably better stored in just a SQL table, right? Like, it's easy, if you have everyone's names, birthdays, everything in a SQL table, you can just go look it up. And the interesting part is, is like, if the model is equipped with knowledge of where to look for that information and has information about the structure of the database that it's in, can of course generate just straight against that table without having to go through a, what is potentially this kind of lossy embedding search. Embeddings are great for unstructured data, right? Embeddings are great for finding information about unstructured data, but if the question is about things like birthdays or numerical data or events, typically you can also get

• 26:58Anton Troynikov?

  the model to directly query a structured data store as well. And that's another approach to take here.

• 27:04Nathan Labenz?

  We now have kind of a, I mean, certainly we've had database, you know, splits, conceptual splits, split personalities in the database realm before with things like SQL and NoSQL. Now we kind of have, you know, structured and unstructured, you know, traditional tabular versus vector style database. And I definitely do find myself wanting both at the same time, right? So this seems like you kind of have both developers of like applications on both sides of this divide running toward one another. Do you think that these things end up being kind of served by the same solution or do you envision?

• 27:41Nathan Labenz?

  People having distinct solutions for different kinds of databases and somehow making them talk to each other.

• 27:45Anton Troynikov?

  I think it's worth zooming out a little bit here and talking about what problem we're actually trying to solve. And the problem we're trying to solve is not to store our data in one particular representation or another. The problem we're trying to solve is to make sure that the model has all the information that it needs to complete a task or answer a query that the user has asked it to do, right? So whether or not the actual relational database is a part of your your storage product, the interface that you provide to the application developer has to take that into account. That's what matters, right? And so the way that we'll be solving that, of course, is to just have the appropriate adapters. If it turns out that there's advantage in expanding into, say, like having a relational store, which we already do, by the way, have inside Chroma.

• 28:28Anton Troynikov?

  The document store and the metadata store is a relational database. We use that as an augment to the vector DB and that's how we support, for example, keyword search out of the box, right? You can filter by keywords in Chroma out of the box because we have that relational backing. It's an open question about like how much of that, how much of the actual database component should live inside something like Chroma, but what's very clear is it needs to be behind the same interface, right? So that the application developer, again, isn't thinking about all these plumbing components, isn't thinking about, oh, how do I wire together this framework? How do I get these techniques like hypothetical document embeddings? And when should I know how to use them and writing all of this stuff that belongs to the database? I mean, the metaphor that I like to use is we're in an era today where instead of Postgres, the database management system, which gives you everything out of the box,

• 29:12Anton Troynikov?

  you have the Postgres storage engine. But it's up to you as the application developer to write the query planner, to write the SQL interpreter, to write the block storage algorithms, right? If you're spending all your time doing all of that, you're not spending your time actually building the thing you want to do, which is the AI application. So I guess my answer, my answer what this comes down to is like Chroma will provide a unified interface for data for your AI application. That's the way that we think about it. And what we're doing behind that depends on how we interface with either other data sources or how we're going to plug those data sources directly into Chroma.

• 29:42Nathan Labenz?

  Yeah, that certainly makes a lot of sense. I think from a developer customer profile standpoint, if you can unify an interface and take away a lot of those problems.

• 29:53Anton Troynikov?

  And the thing is, the models can help us a lot with that. We have, as a general principle, like something we're definitely know we're going to do in the future is to bring more of the intelligence down into the data layer itself, right? So we imagine, for example, having a local model whose job it is to essentially figure out how to connect to the data that is responsible for a particular query. When you model this, for example, as a decision problem, right? As you would model a decision problem in, say, reinforcement learning or other AI paradigms, what you're looking for is conditional on the user's query, find the right set of data. That's what you're actually doing. And that's readily modeled already in this kind of like language model or RL model way.

• 30:38Anton Troynikov?

  We have ways to do this. It's just we need to execute on them. In the time being, I think that like, there's very obvious things that we can do, like this kind of SQL query generation stuff is an obvious step that we can take. And then all the user has to do is like, yeah, here's some SQL tables that I have. And the model and like Chroma will just talk to all of that.

• 30:56Nathan Labenz?

  Well, I'm just thinking about the business side of this a little bit more as well. You have, I think the unified interface makes a ton of sense if you are hitting somebody at the beginning of their lifecycle, you know, like, and that is where kind of most of these AI apps are today is like somebody is starting one. I do kind of wonder, how do you think this takes shape? Because I mean, obviously, this is a debate that is raging across every sub-part of the AI space. Like, how much value can this startups get? How much accrue to the incumbents? You know, nobody's going to rebuild Salesforce before Salesforce implements their AI layer. Nobody's going to switch from Salesforce in the window, you know, when that happens, even if somebody, you know, kind of does. I tend to believe those sorts of things. I wonder how you see that dynamic playing out in the just like enterprise software space, right?

• 31:41Nathan Labenz?

  Because there is so much deployed database. And, you know, you're going to pry that out of a lot of mid-career developers' hands, you know, with great difficulty, I would think, right?

• 31:53Anton Troynikov?

  So here's something very interesting that we've noticed. Data that goes into Chroma has never been in a database before, in many cases. That's actually, that's like a big wave coming, right? It's just, we are giving access to data that has never been accessed before in a computer sort of oriented, like in a computable way, essentially, right? It's been sitting there, sure, on, you know, in like document stores or whatever for humans to read, but it's never been, and the reason that this is happening now is because that data has become processable by computers for the first time, right? The AIs can reason about large quantities of documents, it can reason about natural language, whereas before they could only reason about structured data. So all the data that was available to computers were sitting in these structured data stores.

• 32:36Anton Troynikov?

  So it's not that I need to sort of convince people who already have a database deployment that they ought to be using Chroma instead, it's more like, no, listen, now you can actually work with all this other data that you have available, that you weren't using before. It's great that you bring up Salesforce, uh, cause I've been reading Softwar. Do you know this book? It's the Larry Ellison slash Oracle biography, and of course, Marc Benioff was an Oracle executive, uh, and Ellison was an early investor in, in Salesforce. And what Salesforce did, of course, is unify an organization's data around the customer funnel, right? The earliest you can get information about a customer to your organization is when they're in your sales pipeline, right? So it makes complete sense. So this, where the data's coming in, that's where you unify it, and then you build this

• 33:20Anton Troynikov?

  platform of applications around it. I see Chroma actually following kind of a similar model. This is, this is, this is all speculative. Honestly, right now, if I'm completely honest with what we're doing, Chroma's current stage is about getting us a seat at that table, right? This is, this is what the mission of the company is currently. It's demonstrating that we can deliver a product that the market wants in a way that the market wants it, and then building out from that. So I can speculate about the future in many different ways. I actually don't necessarily think that we are a, we are in competition with a lot of these incumbent data stores. I think that we're actually a great complement for them, so long as we can interface with them in a way that it's, it's possible to build applications on top.

• 33:57Anton Troynikov?

  Right. In the same way that Salesforce and Oracle work together.

• 33:59Nathan Labenz?

  So when you say the data that's coming in, I think that's a fascinating analogy and definitely, I would think compelling to your investors and potential future investors as well. You know, that's like the kind of thing that the VCs are looking for, right? Is the sort of, we're not competing with the existing database. We're 10 X-ing how much data goes into databases. When you say that, I'm gonna have an episode coming up with a historian. So I'm like, first thing that's coming to my mind is like handwritten letters, but I know it's not that that you're dealing with, right? So it's like stored digitally already, right? But

• 34:30Anton Troynikov?

  It's stored digitally. It's a lot of it is documents. Today it's documents. And the reason that it's documents today is because we have text processing models, general purpose text processing models is what GPT does today. Tomorrow, and I mean this almost literally, it's more like a week from now, we will have models that we can, that can interpret images and sound. Right? And so that's data that's just lying there. And we've, we've heard really interesting ideas that people have about this stuff. They have data lying around as PowerPoint presentations, and they want the model to be able to understand the slide and distill it into actionable information. All of this stuff is lying around in a way that is not machine interpretable until we have these general purpose models that work in the space of human inputs, which we have now. There's tons and tons of that data.

• 35:14Anton Troynikov?

  And in terms of, in terms of volume, like in terms of sheer megabytes, I think it rivals what lives in the relational tables too. So that sort of stuff, right? Just as, just as an example, I mean, frankly, the ability to interact with general purpose textual data in a conversational way, completely non-linearly, that technology is already enough to create enormous business value.

• 35:33Anton Troynikov?

  But the fact is like, we're far from done with, with what you can actually do.

• 35:36Nathan Labenz?

  Yeah, no doubt. Honestly, the first use case, you know, we've got this, again, this Athena chat, and that was predated by Athena GPT, which is just an internal tool that simply has like 400 documents and allows, you know, the thousand executive assistants to like query policy, best practice, whatever, over these 400 documents. And that thing gets non-trivial use, you know, it is not a

• 36:02Nathan Labenz?

  It's no, in fact, I think that was created with chatbase.co very simply, you know, just kind of drop in the things, boom, you know, they handle literally everything else. And yes, I agree. I mean, that's already like quite valuable. Apparently, they used to get hundreds of questions a day. And, you know, some significant share of those can now just be answered, you know, immediately by the AI, which is.

• 36:24Anton Troynikov?

  If that's all we were getting, this is already like a significant business, right? But it's not. There's so much more to do and it's a function of what information can the models process and how well can they process it? That's what matters, right? That's what we're, I wouldn't say we're downstream of it. We're actually complementary to it because having retrieval makes the models more useful. And then make the models being more useful gives you more reason to do retrieval. It's this very virtuous cycle between us and the model developers. We're very, very complementary to each other, which is another reason why I'm actually like, I'm glad that we're focused on this component instead of the sort of the model improvement component. You, you touched on a really interesting point here. You touched on an internal AI tool, right?

• 37:03Anton Troynikov?

  And again, if you look at the history of computing, most of what is built on unified enterprise data today is internal tooling to make your business processes more efficient. And AI has like this latent potential to make so many of those processes more efficient. And again, reading this, reading this book, Softwar, which I highly recommend for anyone interested in the history of this space. Like the whole point of Oracle, one of the things that they really pointed out to their customers, especially during like the dawn of the web, was you can't just stay on your old processes and expect to get the most mileage out of this.

• 37:42Anton Troynikov?

  You have to adapt your processes to get the real returns out of a lot of this technology. But if you do adapt them, the returns are huge. And I, I think that we haven't seen this yet in AI, but I think that's the potential. If this stuff works, that's the potential. And I think of this, there's actually three kinds of organizations around this. The first kind is an organization with existing business processes that is now adapting to AI, right? So for want of a better word, there's, that's a legacy business. That's a business that exists today, or every business that exists today is a legacy business, which is adapting its internal processes to use AI in one way or another, right? And, and some of those will succeed, some of those will fail. The ones that succeed are going to outcompete the ones that fail.

• 38:23Anton Troynikov?

  The second category is, of course, companies that are building AI tooling, right? Tooling that is enabled only by AI couldn't exist before. Brand-new stuff, right? Like totally new automation, stuff you couldn't ever do without the model being there. New businesses entirely. There's a very interesting third category, though, which I think is yet to emerge, but I think will, which is businesses serving the same customers as legacy businesses, but which are built from the ground up around AI-first processes. And those, I think, are actually going to be incredibly wildly successful, the first few that manage to pull it off. And I'm very excited for those. And of course, Chrome is going to power all of that, so.

• 39:04Nathan Labenz?

  What do you think is a good category to try that in if I was going to go out and steal somebody's business?

• 39:11Anton Troynikov?

  I've spoken to a lot of people in this sector. There's so much complexity in the business processes in real estate, and so much of it is just to do with processing documents because they come in in heterogeneous formats, they're like filled out differently, they all have different requirements, all these things. It's a vast, human-powered document processing operation. And if you can verticalize around AI processes in that domain, I think you have a real chance of doing things well. But of course, there are tech-forward incumbents in that space, too, which I think have a crack at it if they modernize fast enough. That's the one that comes to mind immediately. But the reality is, is like, you have to think about this as, where can I get distribution fast enough

• 39:55Anton Troynikov?

  such that my AI efficiencies actually matter?

• 39:59Anton Troynikov?

  That's where to look for those businesses. I wouldn't try to compete with Coca-Cola today, for example, because it's on the basis that you're using AI and they're not.

• 40:08Nathan Labenz?

  Do you have a sense for like where most of your growth is coming from today? And how you think that will evolve across those categories?

• 40:13Anton Troynikov?

  Absolutely. I mean, look, we have, as I mentioned, users up and down the adoption curve. Most users are down the earlier part of the adoption curve, so that's where most of our growth is coming from. But of course, users graduate up that adoption curve. So the longer that Chrome is around, the more of our users get further up the adoption curve. We have enterprises with Kroma deployed in production today. We have fast growing startups which use Kroma. We have people at, like I mentioned, literally at hackathons trying RAG for the first time using Kroma. And I'm actually very proud that we've built a product that serves all those points along the curve. I think we've succeeded. I'm actually even proud of the fact that we're even being used in ML research. Right? The Voyager paper that came out, the Minecraft playing bot, Dr. Jim Phan's group in Nvidia put that out a little while ago.

• 40:57Anton Troynikov?

  I actually didn't know this when that paper came out at first, but I went and looked in their code base and said, oh, Chrome is the memory engine for this agent system. So it's coming from everywhere and you have to look at this like, I hate to use these like 90s enterprise sales metaphors. I catch myself doing it more and more often, but it really is a wave that's coming, right? And it is about like riding that wave to the right point. And I feel like that's what we're doing. One thing I wanted to mention, actually, you talked about how like for a second there, it seemed like agents were the main thing. And then RAG became the main thing. I don't think that these are orthogonal things. I think that agents use retrieval and I think making retrieval work better will make agents work better out of the box.

• 41:38Anton Troynikov?

  And we're talking to like a lot of the agent developer companies about that right now. I actually think that good retrieval is like a significant fraction of what we need to make good agents work.

• 41:47Nathan Labenz?

  Yeah, certainly that Voyager paper got to people talking and was a very impressive accomplishment. And that one was no training, right? That was just kind of gripping around developing policies, saving it and retrieving it.

• 41:59Anton Troynikov?

  It does in fact have some human examples. And this is actually part of the beauty of retrieval as a complete AI system in that you can implant things in it. And you can delete things from it when you don't want it to behave a certain way, or when you want it to learn a skill it hasn't learnt yet. Right? This is actually very different to, you know, sort of in the notes that you sent me ahead of time. You talked about like how fine tuning doesn't seem to work to implant facts. And this is a very rough mental model. I wouldn't rely on this 100%, but I think of fine tuning as the style or manner in which you want a task accomplished. So in other words, fine tuning is about what information you should pay attention to and how you should then synthesize a conclusion. It changes that, not the facts. And then retrieval is the actual

• 42:42Anton Troynikov?

  data that you operate on. So these things are actually complementary again. There are so many things here that are not at odds with each other at all. If you look at their latest crop of retrieval augmented papers where they do retrieval augmented pre-training, or they do like retrieval augmented fine tuning, you'll find that like, fine tuning allows the model to better use data that it's getting from retrieval as well. So these things are just very complementary.

• 43:04Nathan Labenz?

  Yeah, certainly, the bitter lesson of just any end-to-end training that you can manage to do seems to be the top performer continues to hold true. And that's definitely been a theme of my study of the RAG literature.

• 43:18Anton Troynikov?

  I would also note, though, of course, that the scaling curves are real. If you don't have sufficient tokens to like overfit your model onto the facts that are available to you, it's unlikely that you're going to get much out of tuning at all. You're likely to get more out of a general purpose system with retrieval, right? Which comes again back to this debate about like, is it one giant Death Star model to rule everybody? Or are we going to have these little test-specific guys running around? Because that also really defines the capital requirements of the market too, because it tells you where the compute's going to live. Is it going to live in an Azure data center? Or is it going to live on, you know, your network? Nobody knows. It's tremendously exciting to be a part of this.

• 43:57Nathan Labenz?

  Yeah. Do you have a sense? I mean, this is obviously a huge question, but do you have a sense for how much knowledge can sort of be squeezed out of the model while still retaining the kind of raw G, if you will?

• 44:14Anton Troynikov?

  Yeah, isn't that the question? Right? The way that I've started thinking about this is like, you arrive at a machine that at least emulates reasoning to some degree by showing it examples of like reasoning, concrete examples of reasoning, right? There's no abstract corpus of reasoning that we can show it. So it's like, okay, you need all these facts, and you need to see in the different ways that they can be combined so that you can sort of infer something that can emulate a reasoning system. But then it's like, okay, how do you preserve just the abstract reasoning part of this without it remembering the facts part? I don't think anybody knows the answer to that yet, but I think we are on the way to discovering and working with it. There's been incredible work in interpretability lately.

• 44:56Anton Troynikov?

  I'm not sure if you saw the big anthropic circuit evaluation. I think that that is a tremendous breakthrough. It's on a very small scale. I'm not entirely convinced as they are that it's just an engineering problem from here on out, but I think it is a very important breakthrough. There's other work in terms of finding latent knowledge vectors in the latent space of the model, which is, by the way, exactly the type of interpretability work I've been advocating for for quite a long time now. It's like, actually, let's just look into the space of what the model is reasoning about. I think that as we understand more of that and as we look at different training regimes, I mean, if you look at the caveats in a lot of the retrieval augmented generation papers today, it's like, yeah, the next step is obviously to like, train this with more retrieval in the loop. If we had better examples during the instruction fine tuning step for retrieval, we would have done that too.

• 45:42Anton Troynikov?

  Stuff like that, I think, is yet to be tried.

• 45:46Anton Troynikov?

  And I think that, you know, in the ideal case, you have this 140 IQ machine, which knows literally nothing about the world until you tell it something and then it's able to reason about it. That would be ideal because that would make the system also like completely controllable and interpretable too. You know exactly what information it's using and exactly how it's combining it down to like the individual neuron level.

• 46:05Nathan Labenz?

  Yeah, I found both that anthropic paper and also the representation engineering paper from Dan Hendricks and Zika Coulter and Andy. So, yeah, those are both very compelling and I do think they sort of lead in another question I wanted to ask you, which is, to what degree do you expect that the that which is stored and that which is retrieved may start to slip from the space of that which is human interpretable or, you know, even represented in any sort of familiar format?

• 46:41Anton Troynikov?

  But that's not interesting. That's the thing. So this is, this is a nice little control, right? Because actually, this is where sort of AI safety and commercial interest completely align. A commercial, a commercial deployment has no interest in the function of a system it cannot predict. Right? It's just not valuable. So the data that the model is working with is necessarily human interpretable. Now, of course, of course, there's other potentials here, right? There's the potential that the model learns steganography. There's the potential that multiple models learn to collaborate invisibly because they're reading and writing from the same data store. But at least that is a much tougher ask than the model stores knowledge in its weights in a way that we can't interpret.

• 47:25Anton Troynikov?

  It is a much tougher ask than, okay, this has to be like human interpretable text. And if I don't understand it, I'm like, I'm going to delete it.

• 47:33Nathan Labenz?

  Yeah, yeah. Let me, so let me take one step back on that because I wasn't meaning to suggest, although I do think it's an interesting question distinctly, but not quite the one I meant to pose. If I understand your response, you're kind of referring to the fear that the models will come to deceive us and slip our detection abilities. But take one step in the more kind of modest direction. What I'm kind of wondering about is instead of storing text in some embedded form and then using the embeddings to retrieve the text and put the text in the context, I think we might cut text out of that loop before long and just kind of end up being like, I'm going to store embeddings and I'm just going to inject numbers right into the model wherever.

• 48:15Anton Troynikov?

  Yeah, yeah, 100%. I actually think that this is a natural progression of where we are now. I think having this extra lossy pathway of like returning documents and then putting those documents in a context window, it's silly because like, you know, we have an embedding, we essentially decode that embedding and then re-encode it into another embedding space, or just pass it into that embedding space. That's, that's the way to really do things in the future. But I don't think that that's actually a problem for interpretability. We know where all of those vectors came from. We know what, what outputs they, inputs and outputs they, they correspond to. I don't think that's a problem.

• 48:43Nathan Labenz?

  Yeah, maybe. It seems like it could easily start to detach from the text, though. I mean, when I read these like representation engineering papers or even or the anthropic paper, and I'm like, okay, in these middle layers, we have this kind of high concept representation that we're starting to be able to tease apart. And it's not exactly, you know, once we tease it apart, certainly it's not like one to one with text, but it does seem to be the kind of thing that we can like store and then, you know, kind of load in, inject into the, you know, the process in the future forward pass, you know, as we may wish to. I sort of see that.

• 49:24Nathan Labenz?

  Being really effective, probably, especially if you kind of do some end to end fine tuning and you're like, if you were to relax your, you know, your design constraints and say, I don't really care if I can see what's going into or out of the database. I just want to save stuff and get it and use it and get the right answer. It seems like that will really work.

• 49:40Anton Troynikov?

  I think it will, but I think, again, like I said, there is, you want controllability. You want interpretability. It's completely aligned with commercial interest to have an interpretable internal representation. And there's a few ways to impose that, like making, making sure that the embeddings that you're working with are decodable. And if they pass outside the realm of what's decodable, then you're like, okay, this is garbage. I want to delete this from the database or like the model is misbehaving. Let's find out why it's writing this weird representation and fix that. Or you can even say, okay, well, I'm not going to allow you to go into this part of latent space because I know that like, there's not any stuff there that I want you to process. I think it's actually not that clear cut. I think, I think we need to get away from this idea of just because a representation isn't directly human interpretable because it's not, not in language, right?

• 50:25Anton Troynikov?

  It's a vector of numbers, doesn't mean we don't understand it. I think we have to get away from, from that idea a little bit. Like, sure, you and I, you and I can't sight read it, although I think in the future, especially people working with this type of data will start to develop intuitions about how things are laid out, even if they can't explain them. But I think that doesn't mean that we can't develop tools that allow us to work with this data. I think that that's actually completely normal. Most, most physical systems are encoded as sets of numbers that we operate on. And like when we develop an airliner, we're not, we're not writing the instructions for the airliner in English, but we know how it's going to, because we have tools to interpret.

• 50:59Nathan Labenz?

  Yeah, that's, I mean, does that, does that relegate this stuff to the, to the realm of like real specialists? Because certainly the layperson can't fly the jet. And I sort of imagine like a lot of the AI engineers that are in the, you know, kind of new to the space. Could very easily end up with like developing systems they don't really have visibility into.

• 51:19Anton Troynikov?

  It's just a tooling question, right? It's a question of developing the right tools to work with this type of modality. And that's something we're also thinking about as well, starting with the right visualization tooling, which in the history of Chrome's life, the first thing we built was actually a latent space visualizer. And now coming full circle, we need to do it. We need to build it again into the product because in a traditional, say, SQL database, you can just, you know, get your top, the top N rows of your database and be like, oh, that's what my data looks like. Here you need like different tooling, specialized tooling so that a developer can be like, oh, that's what's in my database. And I think it's just a question of tooling. I certainly don't think that, I don't think that only experts will understand this stuff in the future. I think actually what we're seeing in general in AI is like the barrier falling and falling

• 52:03Anton Troynikov?

  and falling to the point where because you can interact with them in natural language, you can, like an average person, just by talking to it can really perform what would have been concerted advanced AI research in the past by just understanding what the models are doing.

• 52:15Nathan Labenz?

  Yeah, it is crazy that just literally chatting with a bot is like legitimate research into this world. I didn't have that on my 2020s bingo card by any means.

• 52:25Anton Troynikov?

  It reminds me so much of the early web, right? Because in the early web, everybody was experimenting in the same way because it was so accessible. Anybody could put a website up, right? Anyone with an ISP could put up a website. Nobody knew what was going on. Nobody knew how to build these things. Certainly there was no like UI frameworks, let alone anything else. And people just kind of talked to each other and experimented. And the fact that it's powerful but accessible is like, that produces Cambrian explosions of innovation and that makes me very like bullish on the potential here.

• 52:58Nathan Labenz?

  Yeah, it is certainly going exponential on multiple dimensions at the same time. So let's go back to, you know, kind of how you're going to take more of the stack and take away my practical rag problems. How are you going to do that? And how should I be thinking about doing that, you know, right now while you're still building the hosted version?

• 53:18Anton Troynikov?

  Yeah, I mean, the first step to this, of course, is have good evaluations. Like I said, and you know, our, our, our friends and partners at other companies are building great evaluation tooling and our first step to there is to plug Chroma into those evaluation tools, right? Evaluations are great, but you need to be able to do something with them. So what we're doing there is we're developing a feedback endpoint, uh, which is basically you will plug into your favorite evaluation tool of choice, Langsmith or, or any of the others that are coming out right now. Press a button, we'll automatically adapt your embedding space to, based on the evaluation that you're getting, right? So that's one of our very first steps. The next step that we're taking, of course, is we are exploring ways to sub-sample our

• 53:59Anton Troynikov?

  users' data and select the right embedding model for them automatically, right? Based on a sub-sample of your data so that before we have to embed your entire data set, we already know, like, you don't have to think about this or decide. We say, this, this is what we think works the best for you. And, and like, you know, here's how we think fine tuning might help. All this stuff will output automatically and that'll come with a Chroma cloud platform. Uh, for the most part, it'll be a button you can press and be like, give me the best embedding function. What sort of performance improvement can I get?

• 54:27Nathan Labenz?

  How do you measure that sort of thing? Do you sort of look for like variation in the embedded space and kind of try to see that you can like see the difference between the data that's being input or what sort of heuristics?

• 54:39Anton Troynikov?

  So the, the, the answer here is complex, right? There are several different attacks on this. Well, some of them are from the sort of, let's say, pure reasoning perspective where we're dealing with densities and distributions of data in latent spaces, which is, you know, kind of Chroma's bread and butter. Uh, and the other side is of course signals from evals that we have, um, around, for example, similar tasks and data like, okay, well, you know, your data resembles this distribution that we've seen before. It's probably like this particular task. We know it performs well from evaluation. Uh, this is the embedding model we think you should use. Approaches like these, right? And then we can do a much more holistic and in-depth evaluation as your data is actually being used and sort of be able to do that for you in the background, switch that on.

• 55:19Nathan Labenz?

  How much data do I need? How many like, this would be like specifically, was this a good retrieval? So kind of envisioning, you know, I'm in my Athena chat. I ask for something, you know, I've got my function call. It shows me what's been back and I just have a thumbs up, thumbs down to say irrelevant, relevant.

• 55:36Anton Troynikov?

  That's exactly what we're working with. That's exactly right. And so the number of data points that you need is at minimum enough to construct that affine transform densely, which is a function of the dimensionality of your embedding space, which makes this a really interesting trade off, right? Because it means a higher dimensional space may be sort of natively more representative, but a lower dimensional space is more amenable to this kind of fine tuning because you need less data to do it. Now, of course, there's a third point here, which is like, even though you've used, say, eta 2 embeddings, you aren't using the entire dimensionality of those eta 2 embeddings. Your data is probably on some manifold inside that, like embedded in that vector space. And we can project down to that manifold and then do our fine tuning on that. So again, unfortunately, the only answer I have for you is these are open questions,

• 56:21Anton Troynikov?

  but we have a lot of lines of attack. We are hiring a head of research for anybody who's deeply interested in sort of the applications of embeddings and in this retrieval context, especially if they're interested in a lot of those advanced retrieval techniques around, for example, just pushing embeddings directly into the inference layer. So that's a little shout out to our hiring process.

• 56:39Nathan Labenz?

  So you're actually, I think I understood this correctly, but you're actually kind of doing dimension reduction on a commercial embedding like an eta to then be able to fine tune it more conveniently?

• 56:51Anton Troynikov?

  Correct. Yes, we could do that because again, if we realize that your embedding set occupies like a flatter manifold, then we can throw away tons of dimensions, just project down and see. And of course, like we, that's not fixed. We don't necessarily lose any information there either because we can still maintain the full dimensionality of the dataset. And as your data adapts, we can also adapt that like lower dimensional manifold too. Because that projection itself is another transform in a betting space. It's a learned transform.

• 57:21Anton Troynikov?

  All of this stuff is very malleable.

• 57:22Nathan Labenz?

  Everything is one linear projection into, you know, something else, maybe a couple of hops at most, right? That's been a, one of the most profound things I think I've learned over the last two years is just how bridgeable all these different spaces ultimately seem to be.

• 57:38Anton Troynikov?

  Yeah, and it's about just finding the right space a lot of the time, right? To maximize the performance of your retriever. And of course, there's plenty of other little things we can do, right? There's a million little convenience things that we can build, like automatic deduplication and summarization for your data. If you find, if like, there's a bunch of documents and inside like your conditional space, they all lie near to each other and it's like your retrieval results will return it. So this is, this is actually the other part, right? We're going all the way back to one of the first questions that you asked me in this, in this recording was like, should I try to put as much information in as possible? Well, if like 12 of your documents are redundant, you've actually managed to capture none, nothing new. And there's this maximal, there's this idea of

• 58:21Anton Troynikov?

  maximal marginal relevancy as well, which is like a heuristic for, for not returning the same information more than once. But we have intelligence. We can summarize that set of documents into a single document, which the summaries of which highlight the information you're actually interested in. Because remember summary, like everything else is conditional on what the user is actually trying to accomplish. But the models can reason about that. So we can get these conditional summaries, which can collapse all these redundant documents into a single data point. And stuff like that is like very, it's just convenience. It just makes your stuff work better. It makes your costs cheaper because you have fewer data points. All this stuff is stuff we like will build or are currently building, I should say.

• 59:01Nathan Labenz?

  Yeah, I was just thinking about doing something like that for this Athena chat project as well, where, you know, for probably both cost and latency and maybe also overall quality, a sort of quad two instant filter that sits between my retrieve 10 and my, you know, insertion into context. Might actually help because then I'm using a bigger model. Right now it's 254 and it's like, you know, that's a bunch of tokens that I'm feeding in there. If I could feed in fewer and feed in the right ones, I would, you know, save money and latency. Although that obviously comes with complexity, but yeah, I mean, those are the kinds of things I can certainly see. If the database just handles it, you know, that could be very.

• 59:43Anton Troynikov?

  You never need to see it. You shouldn't ever be thinking about these things, right? You never think of what the, what query plan did Postgres generate for this like thing into my, into my web app? You've never thought about that in your life unless you're doing infra work. Similar kind of set of problems here. And this is actually one of those spaces where I think the application of those lighter weight, open source models that we talked about earlier, actually, I think are really applicable. Right? If you are running an embedding function locally, then that can be made more efficient by actually running inference hardware at the point where you're running some inference hardware. Well, you might as well run at least one of these tiny models on it. Right? And you can definitely like get to a higher utilization on that inference hardware that way. And everything just like connects together in a way that that creates a lot of value.

• 1:00:26Nathan Labenz?

  Yeah, it's funny. I've had that, that's happened, I think, maybe a couple of times for me now where I've started from some, you know, commercial, or in some cases, just nothing available, but some, uh, you know, need to actually scale something up into the AWS, uh, you know, kind of Lambda function type situation. And that is a pain in the butt, uh, but it definitely, once you kind of get it working for a particular task, we do enjoy the cost savings and the super easy, you know, horizontal scalability, so.

• 1:00:57Anton Troynikov?

  But also you have the data flywheel. You own the data flywheel at that point, right? In the same way that you're in the customer, you own the data flywheel. Um, and that means that you have a model that adapts continuously to your specific business use case, not some general idea of, of like a general purpose model. Right? To be fair, I don't think general purpose models are going anywhere. I just, I just think it's an open question right now, whether they're these like specialized, fine tuned off the shelf, open source models, which are smaller and in many ways more efficient in terms of compute and cost, uh, could actually be very useful.

• 1:01:31Nathan Labenz?

  There are kind of two related ideas there are owning the data flywheel and the kind of ongoing improvement of the model. And if I was advising, I think, like, most people, you know, obviously, context is everything. But for most people, I would say, it's really important that you start capturing that information and gathering that data. It's probably less important in the short term that you be like, doing ongoing fine tuning.

• 1:01:57Anton Troynikov?

  Correct. But you should be capturing that data, right? And this is, again, like, I may be revealing too much of the master plan here, but consider how similar that is to just capturing information about retrieval quality. We are conditioned to think about data as this static thing, right? It's like, it's sitting somewhere, and it has a particular instance in time, and then we access that instance in time, and then, you know, the next time, it might be different, but it's like, it's still essentially, mentally, we think of it as static. I really think of these things more like a loop, right? It's like, it's like a control system loop, almost, right? Where data, data is actually an engine. It's something that is interacting with the outside world mediated through computation,

• 1:02:41Anton Troynikov?

  and then it's constantly adapting, and it's improving itself. And if you have the right affordances to be able to do that. And so, that comes into this, like, system of continuous improvement in many, many different ways. I agree with you. I think that most people shouldn't be fine tuning online. I think that most people don't have enough data to be ever able to do that. But I think that there's a lot of possible approaches for this in future. One of the things, here's another example, right? One of the things that we're going to be deploying sometime around when we launch Cloud is, you know, Chroma data sets. And this is the fact that a lot of people want to be able to do, say, retrieval against English language Wikipedia, right? Because it's useful to have facts about the world that are not in the training set of the model yet. Or like, if not Wikipedia, then find something from Bloomberg or whatever, real time use sources. There is no reason

• 1:03:26Anton Troynikov?

  for everybody in the world to go around embedding that on their own. They could just hit a Chroma endpoint and get it, right? And then, you know, we charge them a few cents, and just get the document of the embedding. We just do the retrieval for you. And obviously, once we have the cloud service up and running, that's as simple as running a public collection, almost. The other question about that, although, of course, is now, you have all these applications using the same data, right? So Chroma is now a data platform powering a bunch of applications, as well as a database that people use for their applications. Now you have a lot of information about how all those different applications are using the same data. And so now you're like, okay, you know, we know which model this is being fed to. We know what the task is. This is how it's being used. We can fine tune it to that particular task. We can give you that data in exactly the way that your particular application and use case needs it,

• 1:04:11Anton Troynikov?

  instead of this generic embedded way. Right? So that's a path. That's like a way for us to provide almost this public service of these continuously improving representations of these data sets. Right? Because for example, it's very unlikely that that people would use the information from Bloomberg Terminal for a dating app. Unless it's a very high-end dating app.

• 1:04:29Nathan Labenz?

  Depends on your definition of high-end as well. We may have to put that out to the the audience.

• 1:04:34Anton Troynikov?

  If the marital status of your CEO can meaningfully move markets, that's a different dating app. But basically, like the embedding space that you would use for that kind of information is very different to the embedding space that you would use for, say, demographic information or information about like, I don't know, events, basically, right news. They might be related. They might actually be even used the same way in the same tasks. But the information that people care about is actually specific to that particular type of information and the task that people are doing with it. And so over time, by having these kind of publicly available data sets, we can converge on those things. You can have these living, living sets of data that are constantly improving to what people actually want to be doing with them. So.

• 1:05:10Nathan Labenz?

  So I think that's also a super big topic, the sort of evolution of data sets. And a lot of angles, that's such a big topic, a lot of angles on it. So you're kind of evolving there specifically with like, how do I serve this particular customer use case or even individual better over time, based on kind of what they're doing or how they relate to data and so on. I'm also really just interested in maintaining the actual data in the damn database. And having some interesting challenges with that where, you know, one big one is just like. Do you think, you said earlier that like, updates and, you know, inserts are relatively rare?

• 1:05:52Anton Troynikov?

  Not in AI land. What I was saying is they're rare in search indexes. They're not rare in applications at all. This is happening all the time, right? The data is being updated interactively. And so, you know, you have to build, you have to build the data layer in a way that's robust to that. Now, in terms of like actually keeping the data up to date, we've been speaking to quite a few folks around this. I think a lot of people are developing tooling to essentially let you stream data into your database and keep it up to date and keep it synchronized with events in the outside world. And we're happy to support those projects too. You know, we've spoken with the Unstructured folks, we've spoken to a few other folks that are doing stuff like this. And I think that those are very valuable projects that, that like need to get built.

• 1:06:29Nathan Labenz?

  That kind of speaks again, and it makes sense from a database provider standpoint that you're just like, stream it on in, capture everything, you know, it'll be, it'll naturally find its first and, uh, you know, hopefully eternal home here in this database. I guess I'm a little slow, you know, and I'm surprised that you, I don't know, maybe, I'm not sure how many like people you actually seen doing this kind of stuff, but what I'm still kind of trying to figure out is just like, to what degree do I want to store everything that happens in my little Athena chat app as history? If I do start to do that, it probably needs to be some sort of different set than like my kind of canonical reference material or my, you know, initial customer profile. Then I also do want to update my customer profile, but then do I want to have a record of the fact that I updated my customer profile?

• 1:07:10Nathan Labenz?

  Like, I'm kind of, I feel like I'm, everywhere I look, I'm like, boy, uh, my own memory, first of all, I'm appreciating my own memory and it's, you know, while it's certainly a shortcomings, it's relative elegance, you know, is apparent in contrast to what I'm kind of hacking together. So what advice do you, so what practical solutions, what advice do you have for people that are struggling with that right now, namely me?

• 1:07:30Anton Troynikov?

  Yeah, that's very interesting. I mean, this sort of like, um, journaling almost is how to think about it. Uh, one of our hackers in residence, Savan, actually like ran into this as an interesting, you know, set of problems to tackle about like, how do you represent the fact that more recent information is what we want to reference unless we explicitly want to reference information about the past? Right, and there's a few approaches. There's, of course, as with everything in the space, there's kind of two ways to go about this. There's like a heuristic approach. And something like Chroma allows you to do like, you know, date-based filtering and store data in your metadata and be like, "Okay, I only want information that came from this date and onwards." Right, I don't want to know about the history. You can store the history in a collection, but you just filter by dates that are further in the future

• 1:08:15Anton Troynikov?

  or like further in the past, depending on what you want to see. It's kind of a heuristic way of doing it. Frankly, I think that like, there's no reason why we can't have a time dimension in the embedding space itself and be like, "Oh, these are statements about the past." And then you like know to decay that value over time. And then the model itself, the retriever piece of the model can be like, "Okay, well, actually, I need to weight the time dimension more heavily because I need to look at like the history of one person over time, or I need to look at the history of like the space over time and land in the right place." And launch that query. And one way to think about it is if you attach time as an additional dimension on these embedding vectors, then like, if you want all the information, you can project out the time dimension. Or if you want it from a particular time, you can just slice it there.

• 1:09:00Anton Troynikov?

  I think that that's the long-term solution. I think that getting the models to actually understand, getting the embedding model itself to actually understand time is the right way to go about it. Again, though, however, don't forget that not every piece of data probably belongs in the vector store. Some just is better off in a, you know, algebraic data structure like a SQL database.

• 1:09:19Nathan Labenz?

  Yeah, you're certainly making me recall also the recent TechMark group paper, I think it's Language Models Represent Space and Time. And I don't know if you've seen that one in depth yet, but...

• 1:09:30Anton Troynikov?

  The ones where they found out it has like an implicit representation of latitude and longitude.

• 1:09:34Nathan Labenz?

  Yeah, and there's a timeline component to that as well.

• 1:09:36Anton Troynikov?

  Yeah, I'm yet to read that paper. It's unsurprising to me that they'll develop these sorts of representations. Right, like, information about place and time is encoded in the training set. I don't see why they wouldn't represent that. Again, everything is trained on the same contrastive triplet loss. If you have signal that places are different in different places and times, and they're different in a specific way, then it will capture that signal. That result was actually not that surprising to me. I thought that, okay, that's a very natural consequence of this kind of training.

• 1:10:04Nathan Labenz?

  Yeah, I would agree. I wasn't, what I was blown away by was the visualization, which I think is becoming the signature of the group. Because they've they've done a couple of really nice versions of this where they'll have, you know, here's 12 seconds of whatever and you immediately get it. In this case, I really felt like it solidified this model that I did kind of already have, but you know, wasn't so sure how confident I was in it of the middle layers being where sort of the highest, you know, order concepts live. And they seem like quite decoupled there in in many ways, like the amount of associative stuff that's loaded in, you know, even just given a place name.

• 1:10:46Anton Troynikov?

  If you think about that, it makes sense because of the way the contrast of pre-training works. Contrastive pre-training is only about similarities and differences. When you are at the highest level of difference, it doesn't matter in which direction the difference in, because you don't have like a third thing with which to make reference to, right? All difference is relative. So you can be like completely floaty at the top level. At the top level of the hierarchy, you can be very like ambiguous, without any meaning in the space itself, because like, okay, is this left, right, up, down, doesn't really matter, as long as they're separated in some way. And then as you get inside these clusters, which is like the lower levels, the structure of those differences matters more, because more of those points are like nearer to each other, and you still need to separate them. So structure gets imposed in that way. For me, that makes complete sense, actually.

• 1:11:31Nathan Labenz?

  You should apologize for your privilege for saying that. But increasingly, I feel like I'm right there with you, although, you know, I think still a step behind. But I wanted to also ask, just this, this is a very tactical question, but, you know, if I'm missing stuff, and, you know, I heard you loud and clear on kind of the importance of evals as like a, you know, way to have some sanity around whether I am or I'm not missing stuff. If I am missing stuff, am I more likely to, like, how much, I assume it's more, but like, how much more likely is that to be happening due to representational issues versus the nearest neighbor being sort of approximate and maybe missing something at that, like?

• 1:12:16Anton Troynikov?

  Almost all of it is due to the representation.

• 1:12:18Nathan Labenz?

  Is there a scaling law there? Or can I can I quote something, you know, if anybody ever tells me like, oh, your nearest, it could be you missed it, nearest neighbor step, like, how do I say, no, that's a one in a billion? Or like, how do I?

• 1:12:28Anton Troynikov?

  So here's, here's the easy way to figure this out. Like, the nearest neighbor benchmarks, we understand the trade off very well between recall and speed. Right, that's a completely tunable parameter. You can just go look at the graphs, right, for various algorithmic implementations of approximate nearest neighbor. When your results are differing from the graphs, that's the representation's fault.

• 1:12:46Nathan Labenz?

  Gotcha. Okay. Is there one number I should keep in mind for kind of, you know, if I just use the default setting, you know, what do I miss? One in a million, one in a billion?

• 1:12:55Anton Troynikov?

  Things are quite tunable. I think like one in a million is probably about right on. I would have to check up our internal benchmarks on this, but I reckon I think on our default setting, something. And again, the thing is, is this is never a concrete answer because it strongly depends on your data distribution. If you have some pathological data distribution, it's going to be much worse than that. It's kind of easy to cause this. If you build a HNSW graph where you just repeat the same entry over and over and over again as unique entries, and then you add some other entries around that, then it'll be completely corrupted. You'll never get good results. So it does depend on you. My point in saying that is it depends on your data distribution, but I would say like one in a million is pretty reasonable. But you can tune that, like you can completely tune that depending on the scale of your data set as well. If you're willing to trade some latency.

• 1:13:39Anton Troynikov?

  And I think, of course, right now, you should probably be willing to trade a bit of latency, given that the latency is completely dominated by the by the model inference itself.

• 1:13:46Nathan Labenz?

  Yeah, definitely. So I'm actually kind of thinking for an application like what I'm building, I could eliminate that fear entirely and just search the whole database, I think. I don't, I don't need the approximation.

• 1:13:57Anton Troynikov?

  Yeah, you could. I mean, you could get away with a matrix multiplication in NumPy, depending on the scale of your data.

• 1:14:01Nathan Labenz?

  Well, that sounds like more work than using Chroma.

• 1:14:04Anton Troynikov?

  It is certainly more work than using Chroma, but you could do it. The other thing, of course, is like, we're not stupid. We know that that is faster. And so we will actually, like, if, if, if the data is small, we'll put you in a small thing that just does that without constructing all the index and everything. We automatically like switch over to the right data structure when we need to, when the scale reaches it.

• 1:14:23Nathan Labenz?

  How much of the future do you think is partnership with other technology companies for you? Like, it seems like Chroma can be the sort of You know, piece that, for example, any number of, you know, other dev platform type things add, or you can, I mean, this is kind of another classic. You can go back to software on this one because how much do you partner into existing platforms versus try to, you know, get them to partner into yours?

• 1:14:48Anton Troynikov?

  Yeah, I think so. Look, we like to work with the AI labs with the LLM API providers because again, when their API gets better, we get better. And when we get better, more people have reason to use their API. Right? So it's this very virtuous cycle of working with all of them together to make sure we're shipping the right thing to our users and customers. And again, we've got great relationships with the application development frameworks for LLM applications too, because it just makes sense. If retrieval works well, then the rest of your application is going to work better for the same reason. We are building, as I mentioned, our sort of evaluation endpoint where we can consume evaluations, which are produced by some of these eval platforms which are coming online. We have spoken with kind of the people doing ETL for this kind of unstructured data and feeding it into,

• 1:15:32Anton Troynikov?

  feeding it into Chroma from other places. I think it makes sense for us to work with a wide fraction of the ecosystem. I think that Chroma, Chroma's remit is like well scoped. We understand what we're building and what we're building for.

• 1:15:44Anton Troynikov?

  Certainly, we're out of the first part of the exploration phase of our products. We know what we have to do. And so like, we feel pretty comfortable working with a lot of people to do that.

• 1:15:51Nathan Labenz?

  Do you think OpenAI will build their own database at some point?

• 1:15:55Anton Troynikov?

  Yeah, I've been thinking about this. Certainly, they've, they've hired retrieval people. If you know where to look in the industry, you can, you can see it. I think that in general, they are unlikely to release something for developers like we've built. I think that it's possible that they will want their own retrieval system internally. Certainly, some of the things that have been teased or quote unquote leaked about the developer day implies the existence of at least a lightweight retrieval system. Personally, I hope that they run Chroma because we're building for enterprise use case for their scale. I don't think that they have a great reason to build their own, to be honest. I think that this is. Pretty significant undertaking on its own, right? Like it's it's truly, and I don't see it as

• 1:16:37Anton Troynikov?

  something that is in necessarily in the main line of what they have to build. I think that, first of all, I do believe that retrieval is like a really important component of getting to AGI. But I think like building the nuts and bolts of the retrieval system itself is probably not something you need to worry about too much. So my, you know, I I think it's possible. OpenAI has greatly expanded its team recently. I know that they've been working with retrieval, but I don't think it's in their best interest to to build something. And certainly I doubt that they'll build something they're going to hand to developers. They may have something that's like a like a service where you're like, okay, you know, upload, upload your data or like plug in your data here. But if they have that, then it's also good for us because the thing that they plug into should be Karma.

• 1:17:14Nathan Labenz?

  Yeah, it seems like ChatGPT Enterprise is like the first place where this might show up as kind of something that is added for them as a feature, but not necessarily a service.

• 1:17:25Anton Troynikov?

  Behind the scenes, OpenAI has been doing enterprise deals pretty early and sort of enterprise integrations. And we know that they go in with retrieval vendors on some of those deals. We'd like to be in on those two. I think that that's a pretty clear sign to us that, you know, they don't intend to actually build to build these systems for their customers. They just want their customers to pay for GPT, which makes total sense, right? Like if if if we enable people to pay more for GPT, we're happy to do it. But, you know, and again, like you can look at our relationship with Google as an example here. Google have highlighted us as a launch partner for Palm two. The Palm two public API is an end to end thing. And one of the reasons for that is just how developer friendly we are, right? We have great rapport with the developer community. Our tooling is the easiest to get up and running with, and people do use this in production.

• 1:18:07Anton Troynikov?

  And so, yeah, I think I think that like for us, it's about staying friendly. It's about delivering the best product. And then, given that the best version of our product makes more people use OpenAI's actual product as opposed to some ancillary piece of software. It makes sense for us to build this and them to build models.

• 1:18:25Nathan Labenz?

  How much do you think about Anthropic? That has not come up today.

• 1:18:29Anton Troynikov?

  Yeah, quite a bit. I mean, look, I've been playing with Claude 2 quite a bit. I know that they have the very large context window. They do the thing where they do PDF ingestion. I've been playing with it. I think that Claude are on a slightly different path to a lot of the other labs because their objectives as a research group are pretty substantially different. I think that things like Claude are intended to be test beds as well as commercial products, test beds for their approaches to a lot of these things. We haven't had much cause to speak with them yet. We did talk to them about like using a retrieval system in the context of training data. But that use case looks more like the search index use case where you want like a scalable search index and a lot of people use FIS for that. And I think that that's like fine, although FIS is kind of hard to scale. So I expect that we'll be deploying something there.

• 1:19:13Anton Troynikov?

  Naturally, over time, the sorts of tasks and the sorts of applications that the different general purpose labs serve will converge to a similar set. And people will be choosing among those vendors and given that they'll be choosing among those vendors and given that we're providing certain capabilities to the people using OpenAI and Google, I don't see why we wouldn't provide the capability to Anthropic 2.

• 1:19:32Nathan Labenz?

  I was gonna ask if you have any updates to your P Doom in light of the recent AI engineer slide that's made the rounds and.

• 1:19:39Anton Troynikov?

  Oh, that's so funny. I think there's absolutely no consequences for saying whatever number you want when you get asked that question. So people will just say fun stuff. I'm actually not really worried about figuring out Doom. I'm much more worried or interested in, I would say, in the question of can we actually get a reasoning system out of this approach or not? I don't know if you saw my fairly recent experiment with Game of Life. Did you see me doing that with GPT?

• 1:20:07Nathan Labenz?

  Hmm, tiny bit, but I'll say no.

• 1:20:09Anton Troynikov?

  I'll send you a link, right? So I was like, okay, like, can GPT apply its own stated rules in a deterministic fashion, even when the rules are extremely complex? Because people were a little while ago talking about like, oh, it can play chess. And then I was like, okay, well, how come it plays chess badly when you give it random positions? And then people were like, well, it's actually emulating bad chess players. I'm like, damn, that is a very strong statement of what it's able to represent internally, right? It's like, oh, it like not only knows the rules of chess, not only knows what optimal chess strategy is, but it also knows how a bad player would play. So it's able to like condition on that. That seems insane. That is a very strong statement to me. And so I came up with instead the weakest possible version of that statement that I could, which was, can, first of all, does GPT know the rules of Conway's game of life? And if you ask it, it will tell you.

• 1:20:54Anton Troynikov?

  And then, does it apply those rules consistently or correctly? So I actually cared less about can GPT play game of life? I hundred percent believe it's possible to get a transformer to play Conway's game of life. Hundred percent, I'm sure it's possible. Right, you just feed it enough data about the state transitions, it'll be fine. There's only like five hundred and twelve of them. It's really not that hard. What I wanted to do, what I wanted to find out there was, there's plenty of information in GPT's training set about the rules of game of life. And I'm sure it even has like lots of examples of the interesting states of game of life, like a glider. So it probably has plenty of information of like interesting state transitions. Oh, look, game of life can like do all this general corporate stuff. But what I'm interested in is I kind of just apply the rules of game of life consistently. Right, and then perhaps correctly. And I gave it the rules as a word problem, which was like N cells are alive, K cells are dead, the center cell is alive.

• 1:21:39Anton Troynikov?

  What is the next state of the center cell? Got that a hundred percent correct. Cool, that's like confirmation that it has at least one representation internally of Conway's rule, game of life. And then I presented the same information to it in a grid, the five hundred and twelve possible state transitions. And then I said, okay, well, it actually does pretty badly at that. It like, it repeats the rules to me and then it does it wrong, even when I tell it explicitly what to do. And even when I'm very careful to represent the grid as individual tokens for each cell, it doesn't do it right. And then I said, okay, well, it doesn't matter if it does it right or wrong. What I want to know is, does it have a internal set of rules that it's applying? Right, because you would expect that if it has rules for game of life, it'll apply them across representations and in different settings. And so I ran this experiment where I fed it ten by ten random boards, random binary

• 1:22:22Anton Troynikov?

  boards representing game of life states, and I asked it to iterate just once on those states. And what I wanted to measure was not, was not, does it get it right? Because we already know that it didn't. What I'm trying to measure is, does it do it consistently? So first of all, does it do it consistently for that representation? And then, does it do it consistently with the rules that it actually executed when you give it all this, all the state transitions individually? And the answer is no. It's consistent inside the 10x10 grid representation. So it tends to do the same things with the same state transitions. Although you can look at a very interesting graph that I'll send you, which is probably the most interesting part of my result. But it does not apply its own rules stated in one representation to this other representation, which to me points to no world model. But that doesn't answer the question. It's just a piece of evidence. It's just an angle of attack.

• 1:23:05Anton Troynikov?

  So these are the questions that I'm actually much more interested in right now. Can we arrive at a general purpose reasoning system? Because I think arriving at a real reasoning system is the thing that we require for this technology to really work. I think we can already get value out of it today. Don't get me wrong. We can get value out of it in domains, which are like subject to human interpretation as opposed to like exact results. But I think the real power of this is if we can use it as a general purpose computing machine, and I don't think we're there yet. I stopped arguing about Doom and I started thinking about much more practical questions.

• 1:23:35Nathan Labenz?

  Yeah, I will have to return to this in greater depth in a future episode. This has been a ton of fun. I'm like, you know, my quick reactions are it doesn't sound like no world model. It sounds like incomplete world model. And I guess I would also be like, how general is general?

• 1:23:51Anton Troynikov?

  Take a look at my thread, see what you think. I'm very curious about what you think. There's one particular piece of the result that I found tremendously interesting, which is it is conditional how random the model is on certain decisions that it makes. And that was very interesting. And you know what's interesting? How random it is, is conditional on how often those states appear. So the less often those states appear in general statistically, the more random models, the model state transition actually is, which is again evidence against for me, but you can argue otherwise. Let me send you the thread. This has been a lot of fun. I really enjoyed this conversation.

• 1:24:24Nathan Labenz?

  This has been a really good, you know, expert level discussion, which is what we try to deliver. So thank you for being a part of the cognitive revolution, Anton Troynikov. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.