Ep 14: Chroma CEO Jeff Huber on Vector Databases, Multimodal Embeddings & Building an AI Startup

On today’s episode we talk with Jeff Huber, the CEO and Co-founder of Chroma. We talk about what sets Chroma apart from its competitors, new developments in AI technology, and advice for listeners who want to get started in AI.

• 0:00Speaker 00

  This is Unsupervised Learning, a podcast about the latest in the rapidly developing AI landscape and what it means for businesses in the world. I'm Jordan Siegel and I'm joined by my co-host, Erika Brescia. On today's episode, we have Jeff Huber, the co-founder and CEO of Chroma. Chroma is an open-source vector database that has quickly emerged as one of the leading offerings in the space, with more than 7,000 GitHub stars and 4,000 Discord members. This is Jeff's third startup and he has previously had a product at Mighty Hive, which was acquired for 150 million. We talk about what sets Chroma apart from its competitors. I think that in some ways, we were maybe not smart enough to make a complicated thing and as a result, made a simple thing. We discuss new developments in AI technology like multimodal embeddings. Some humans have synesthesia. They hear color, they taste sounds or whatever, sort of like language model synesthesia.

• 0:43Speaker 00

  And Jeff shares his advice for listeners who want to get into AI. It is not too late to start a new company. Go do a PhD in this stuff. It is not too late to join OpenAI. It is not too late to join Chroma. Thanks so much and hope you enjoyed the episode. Jeff, we're very excited to have you here and thank you so much for joining us.

• 0:58Speaker 01

  Thanks for having me.

• 0:59Speaker 00

  So to start, you know, I'm sure our audience would love to just learn a little bit more about yourself, what got you into AI in the first place and ultimately what led you to start Chroma.

• 1:07Speaker 01

  Sure. So Chroma is about 14 months old now. We got started in May of last year. The thing my co-founder and I, I think, connected over, we've been friends for quite some number of years and we had both worked in applied machine learning contexts. So we had both seen the gap that it was to take a sexy demo, which tends to be fairly easy if you have clean data in machine learning, and building a robust production system you actually want to rely on, which tends to be incredibly hard. Each nine is probably an order of magnitude harder than the nine before. And, you know, this seemed like a big problem. It seemed like there were not very intelligent solutions.

• 1:49Speaker 01

  So, you know, even today, in most kind of traditional deep learning contexts, the state of the art is random sampling for putting new data into the training pool and like campaign-based data selection. So, hey, for whatever reason, our stop sign detector doesn't work when there's a willow tree over it. Okay. Like, let's get our data analysts to go find, you know, a thousand examples of this in our data set, historical data set, and let's get a sim vendor to try to create a bunch of permutations of willow trees over stop signs. We'll label which of data, we'll feed it into the model, we'll hope it gets better, and we'll hope it doesn't break anything else, which is often not the case.

• 2:29Speaker 01

  So this seemed pretty bad, and, you know, our belief was that the geometry and topology of the latent space itself. So looking at the data in latent space, in the way that the model itself understood its own data, was a really promising approach to be able to identify a clear path to, again, take your sexy demo and turn it into a robust production system. Now, obviously, 14 months ago, this was pre-large model boom. So I think that's a good thing, you know, we've been at this for a while. I guess maybe not that long in the big picture, but, you know, Stable Diffusion came out last summer,

• 3:09Speaker 01

  ChatGPT came out last winter, and, you know, I think the thing that we saw was how challenging it was to create any products with vector search internally. You know, we had built out some vector search stuff internally for ourselves to run analytics on top of that latent space, and just found everything very difficult to use, sort of overbearing,

• 3:30Speaker 01

  and then developed a thesis that, you know, this was really important to actually have an email to one of our investors in early January. And I kind of just laid it out. The title, some of the amount of the email was, "Help us walk through this idea maze", and it was, you know, "Large models are really important, embeddings are integral to the world. Every piece of data will be embedded to make it interpretable to language models", and, you know, a system which has a huge emphasis on developer experience does not exist. So that was kind of the nugget, which really came from talking to users, honestly, too. We, in the month of December, or basically end of December, early January, I,

• 4:10Speaker 01

  and this would be no longer possible, but I searched every user who had tweeted about embeddings or LangChain on Twitter in the month of December. And DM'd every user that had open DMs. And it was only like 150. It was not a big number. And then set up something like 35 Zoom calls. And just through the conversation, I think, found this to be true, which was people had a lot of, you know, kind of pain and heartache in rolling their own thing or using existing things that were sort of built for a different use case. And so Valentine's Day, as it turns out this year, we launched Chroma in open source. I don't know why it was Valentine's Day. Yeah, it was supposed to be the day before Valentine's Day, but like we had a bug we had to fix, so it got delayed by day. And so, yeah, it was Valentine's Day.

• 4:54Speaker 01

  We launched the project and, you know, it's been cool. It's been awesome to like, obviously, build the community. Building an open source company and project is really fun. It's just a good time. So, yeah, it's been a lot of fun.

• 5:07Speaker 02

  I think Valentine's Day is actually a great day when you're launching something in open source, right? It feels like a bit of a gift or a love letter to the community. I don't know. I'm actually not a big Valentine's Day person in terms of cards, but if it's like open source software, I can get off on a Valentine's Day.

• 5:26Speaker 01

  That's amazing.

• 5:27Speaker 02

  Statement. I will, you know, to your timing point, I think your timing is just absolutely impeccable because you were far along enough to kind of ride this wave. And Maya, if you reflect on how many folks were talking about this stuff in December versus now, it's amazing how far we've come. That said, and while most of the folks I'm sure listening know what Chroma and vector databases are, can you just give like a quick high level for some folks who might be newer to the space? Like, what is a vector database? And what is an embedding? And why are these important in the world of AI?

• 6:04Speaker 01

  The thing that we want people to do is be able to have language models learn and remember things outside of their own weights. So the weights of a language model are very large. They're expensive and difficult to update. And it's not even deterministic, right? You could try to teach a language model a certain fact by telling it over and over and over again in the training corpus, and it's still not, you know, landing in the training corpus. Removing information is basically untenable. We have no idea how to remove information surgically from the weights of a model. So what you want is an external knowledge store. You want the language model to be able to learn and remember things, but outside of its own weights in an external knowledge store.

• 6:48Speaker 01

  Something that you, a human, can look inside. You can add information to. You can update information and you can delete information. And it just works instantaneously in the same way that any other database would. So you remove a document. You no longer want the language model to know about that. It's done. The analogy that I use sometimes is like Neo in the Matrix. When Tank is like downloading instructions to his brain, to his brain. You know, and he had the famous line like, I know Kung Fu. And then goes into the dojo and fights Morpheus. It's awesome. So what is an embedding? What is vectors? Under the hood. I mean, an embedding is literally nothing more than a list of numbers. Literally just a list of numbers. What those numbers mean is the meaningful part.

• 7:29Speaker 01

  And so embedding models are machine learning models trained to put similar things close to each other in high dimensional space. So if you have a vector that's a thousand numbers long, that represents a point, a point in a thousand dimension space. And what we want is to have similar documents. It could be the documents about Paris. So other documents about Paris. It could be the documents about cozy coffee shops in Paris. And so other cozy coffee shop documents in Amsterdam and Berlin and other places are near that point in embedding space. And what it allows you to do is to be able to have this new fuzzy search that wasn't really possible before.

• 8:13Speaker 01

  You know, text search, obviously you can say, hey, give me Paris and you'll find everything that has Paris. Maybe it'll also give you Parisian and some other permutations of the word Paris. But it won't give you, you know, this kind of this fuzzy matching, this fuzzy search. So the fuzzy search capabilities is what allows the language model to both learn and remember things effectively. Because what you want to do is say to the language model in a prompt. You've used ChatGPT, you've probably experienced this, you want to go to the box and say, hey, what are the cozy coffee shops in Paris? And when you type that in, you could either, you could hallucinate something, so it could just make it up. It could just, you know, stream the tokens out of its own weights and just make it up. At best, you know, it might be stale,

• 8:55Speaker 01

  right? So this is data that was in the corpus when it was trained many years ago. And at worst, it's just gonna make up coffee shops. It's just gonna say, hey, go check out, you know, Rousseau's blah, and it doesn't even exist. It's on a fake street. And so what you want is you want to ground the language model in factual information and be able to find that factual information extremely quickly, and then surface it to the language model and say, hey, language model, the user asked this question, they asked about cozy coffee shops in Paris, which documents do we know about that should, that would help us answer that query? And so vector similarity search, vector search, and vectors under the hood enables that to happen.

• 9:36Speaker 00

  I'm curious, you know, obviously like nearest neighbors and vector search is one major use case. You have this amazing open source community since launching. Have you seen any other surprising use cases emerge? Or, you know, the community trending toward other use cases for vector databases?

• 9:51Speaker 01

  I mean, the primary use case, yeah, the era of things that we are in is the sort of Web 1.0 newspapers on the internet phase. So we're in the era of taking the things that we were already doing and doing them in a new way. And this is sort of what broadly chat your documents is like. There's a chat bot craze every five years for the last 15 years in Silicon Valley. And, you know, this is the latest instantiation of that. This one probably will actually work. Again, it's sort of newspapers on the internet. It's like in Web 1, we took newspapers, we put them on the internet, retrieval and regeneration, which is the task of chatting over a set of documents is sort of analogous to that. It's sort of obvious. I think really great. I think probably every document set on earth

• 10:34Speaker 01

  will have this ability to chat over it. It's just so. Useful to humans, but it is kind of the first obvious thing. I think we're starting to see signs of going beyond that. And I think that, um, obviously it's still very early, but I think that agents do pose a very interesting and important use case for memory. You've seen things like AutoGPT and maybe AGI, obviously in other projects, just do the craziest GitHub star art you've ever seen in your lives. And then you ask, is anybody actually using this stuff? And the answer is no, um, or not yet, but it will happen. It will happen. Um, I think that, you know, function calling stuff from opening I is making it a little bit more deterministic already. A bit, bit more useful already. It'll get better and better. Um, and I think like in broad strokes, you know, the idea is that

• 11:19Speaker 01

  any piece of software you use today can almost definitely get better with an agentic intelligence inside of it. So it's an intelligence that knows who you are, knows what you are trying to do, knows, um, what your preferences are and remember the remember those things. And that idea of like kind of an agent living inside of every piece of software you use today, um, I think will likely be extremely widespread, even just maybe a year from now or sooner. And so that's a native use case. You know, you could not imagine before. I mean, maybe you think Clippy is some like very early version of this, um, but, uh, you know, you kind of couldn't, you know, three years ago, you couldn't really imagine

• 12:03Speaker 01

  this happening. And so in that way, it's native and that way it's like more exciting.

• 12:06Speaker 02

  You know, I think Jordan talked a little bit about this amazing adoption that you've seen in the open source community. You know, when you kind of zoom out and look at the space, there have been a lot of native vector databases like Chroma coming out. And you've done an incredible job of standing out above the noise and it's, you know, it's showing up in the active community that you have. What do you think that really sets Chroma apart? And in particular, you know, how important do you think the fact is that people can just deploy and get, you know, Chroma up and running on their laptop and start building with such an easy on ramp? Like, how key of a differentiator is that?

• 12:50Speaker 01

  Yeah, I think it's hard to replay history and be too deterministic about what we did right, what we did wrong, and have really any conviction about that. I think in some ways our timing was extremely good. That probably was mostly due to luck, didn't go versus skill. I think that in some ways we were maybe not smart enough to make a complicated thing, and as a result, made a simple thing. And people when they're learning a new technology really want to grab the thing that's simple and easy. So that dedication to maybe conventional configuration, I think, has been useful. I think like a strong eye to design.

• 13:31Speaker 01

  So wanting to make everything from the landing page to the documentation to the API itself feel intuitive, it does what you think it should do, there's no sharp edges that you run into, it just works. It just works is difficult. Making a simple thing that just does what you think it should do is probably the hardest thing to make. It's maybe the classic kind of like, what is Michelangelo quote, what is David? I just removed everything that wasn't David. So I think those are things that we've done well. On the marketing side, I think we've done a bad job, actually. We posted a few hackathons, I think our Twitter account has tweeted like 40 times. So I don't know that actually, you should look at what we've done on the marketing

• 14:15Speaker 01

  side and say, hey, that's a textbook, that's a playbook for how to do something here. I think, again, we've gotten quite lucky in many ways.

• 14:23Speaker 02

  Well, I do think it speaks to the power of a great DX, though, right? You could say if you don't think your marketing's great, but the reality is the community's growing anyway, and that's because of the simplicity, the ease of adoption. And, I mean, this is kind of a tale as old as time. For the old school folks on the phone, you think back to the early MySQL days, right? The reason that MySQL became so popular was because it was incredibly approachable and easy to use and felt familiar, right? And, yeah, a bit of a throwback there.

• 15:01Speaker 01

  Yeah, yeah. It's a bit of an interesting, or it's a bit of a yellow flag to say that we believe in product over distribution. I think we also clearly care about distribution quite a bit. But, yeah, I mean...

• 15:19Speaker 01

  Think entrepreneurs are taught over and over again, if you built it, they will not come.

• 15:24Speaker 01

  And then sometimes that's wrong. I'm very surprised that that's been as wrong as it has been in this story thus far. So, but we're not resting on our laurels. We're working really hard to make the product better, serve developers better, figure out what new use cases are coming in the pipeline, new integrations. You know, I think we ultimately, we feel like the opportunity we have is to help define how a new generation of software is built and how it works. And, you know, that's an opportunity you're not handed that often in life. And so we take that really seriously. You know, I think, again, by no means are we, you know, resting on any laurels and are really thankful of the community for their involvement and help. I mean, the product

• 16:04Speaker 01

  is so much better because people have, you know, found the rough edges and told us, allowing us to fix them. So.

• 16:11Speaker 00

  I'm curious, you know, you talked a lot about developer productivity and that's something that Chroma has nailed for sure. Are there any other vectors for, no pun intended, that people are evaluating Chroma on compared to other vector databases, whether, you know, it's speed, whether it's supporting certain types of workloads better, whether it's transactional or otherwise? Just curious, what are the other considerations people are looking to in this space, other than just, you know, amazing developer experience?

• 16:39Speaker 01

  Yeah, it's a good question. Obviously, we have a new technology that's emerging. Approachability probably is the single most important metric. Beyond that, I think the jury's still out on what matters. Certainly just sort of all the databasey things. So, you know, acid, this, and speeds and feeds that, that's about matters. And it matters as these use cases scale and as they become, you know, more and more production grade and more and more large companies with large data sets look to onboard the technology into their organizations. All of these things are very relevant. And so, you know, we are of a concerted effort to, you know, move the needle in that direction month over month for probably years, decades to come. I think the more interesting

• 17:21Speaker 01

  question is not what is the same as before. So, these are things that are sort of the same databases have always had the considerations, but what is different? So, what is new about? You know, vectors that's different than relational databases or is different from, you know, other database technologies, key value stores, whatever. And I think for us, it comes back to developer experience. And so essentially, once you cross four dimensions, humans are sort of SOL when it comes to intuitively understanding the data. Relational databases have the advantage that it's just 2D data. You have rows and columns and you can make a table and you can put it on a screen. And screens are 2D and you can just get it very quickly. Once you have data that's, in the case of OpenAI embeddings, 1536 dimensions,

• 18:03Speaker 01

  being able to understand that and understand what's going on inside of it and get any sort of deterministic control or ability to understand and guide what's happening becomes much more difficult. And so I think at a high level, we believe that yes, we need to build a world-class, ubiquitous standard for the transactional side of the system. So the, the online database that's sitting in the loop of our product application, that's getting called and queried all the time, documents are getting created, updated, deleted all the time, recall stays very high, things move quickly, scales well. Those are sort of table stakes. They're not easy, but they're table stakes that have to happen. I think, again, the more interesting questions are how can we help both developers

• 18:43Speaker 01

  and also language models themselves be able to introspect, interpret and understand latent space? And so that looks more OLAP-y. So it looks more like analytical workloads over vectors. And if I have any hot take, it might be that this is like finally the data structure that H-TAP will actually matter for most workloads. And so for the audience, that's hybrid transactional and analytical processing, both, you know, individual record retrieval and updating, as well as being able to analyze the entire set of the database or all of the items in the database and doing so quickly and efficiently. Maybe vectors is a data type where that will actually matter in a big way, so.

• 19:28Speaker 00

  We love hot takes and we'll try to dig some more hot takes out of you in a bit.

• 19:32Speaker 01

  Okay.

• 19:33Speaker 00

  But you mentioned the classical databases and more on the relational side. You know, obviously, a lot of those databases are adding vector support. You know, you have PG Vector, you know, Mongo Atlas added similarity search, Redis. And there's sort of this debate going on as to what that means for vector databases in the future. I have a feeling I know which side of the fence you're going to lean on here, but just curious your thoughts. And, you know, as more of these providers add vector search, what does that mean for Chroma and other vector databases?

• 20:01Speaker 01

  A few, yeah, a few comments. In general, we're happy about it. It's helping to educate the market on this, what is this technology, that this technology matters, how to use this technology in general, what use cases are. And so we're generally big fans. I think that going back to developer experience, you know, just tacking it on to the thing that you're already doing to have checkbox compliance, buzzword compliance, to, you know, avoid users from churning or revenue cannibalization or whatever else, is not going to give you the best developer experience. And, you know, the best developer experience requires extreme obsession.

• 20:42Speaker 01

  And a product that's focused on that ultimately will do the best job. When it comes to open source, you generally, you know, bringing class of software, you have, you know, one project that gets sort of to runaway community love and adoption. And, you know, that's, that's, and that ends up what, that ends up being what matters the most. That's one comment. I think the other comment is, you know, things are still very early. And so, you know, anybody can throw a saddle on their horse and call themselves a racehorse and try to enter the race. But I think as time goes on and, you know, we have more time for folks to get down, get on down the track,

• 21:26Speaker 01

  it will be more obvious, you know, kind of which, which point, point solutions are again, are the best for the workloads. So.

• 21:36Speaker 00

  You know, you, you, there's a really interesting announcement of yours last week with the applied research team that you formed in Chroma to investigate, you know, unresolved questions in the field of LLMs. And I think our audience would love to hear, you know, what are some of those areas that you've identified as, you know, the worthwhile investigation areas to look into first?

• 21:57Speaker 01

  Yes, so Kroma will be building out or is building out a small applied research team. We don't, you know, we don't and don't want to train our own language models. We will not be training our own embedding models. You know, we are not competing with OpenAI or here Anthropic or anybody else. We are truly Switzerland when it comes to all those different, you know, sort of LM intelligence providers. We want to be able to guarantee to our users that we don't have a horse in the race. That we can be, you know, fair and unbiased when it comes to the tools and recommendations that we're making to users. That being said, in the journey of any developer adopting even the most basic use cases, they run into a series of questions every single time.

• 22:41Speaker 01

  So questions like, how should I chunk up my document to be embedded? Which embedding model or models should I be using to embed these chunks?

• 22:52Speaker 01

  How many nearest neighbors should I be retrieving? Five, 10, 15?

• 22:57Speaker 01

  And are these retrieved nearest neighbors relevant at all? Or are they from a part of the embedding space, which is too sparse or too dense? Both are failure modes in different ways.

• 23:08Speaker 01

  And so, again, early days, a lot of basic questions are still not answered or don't even have tools to help people evaluate. Eval also continues to be a really big problem. So, understanding, and this sort of goes back, obviously, to which embedding model should I use is ultimately it should come back to some eval metric. And, you know, there's not really even agreed upon ways of doing that yet. So when we think about applied research, those are the kinds of things that we are, I think, most interested in. The part that we're probably most interested in keeping our ear to the ground on, not necessarily actively working on, but the things that we're paying attention to are the ways that retrieval might come even deeper into the pipeline or stack. And so today, we sort of have one-shot retrieval.

• 23:52Speaker 01

  Where the beginning of a output, we do a retrieval, we put it into the context window, we click go, and we let the language model stream out and number of tokens. We think that it is likely in the future that retrieval will be much more ongoing. And so you will retrieve every token, or you will retrieve based on some sort of saliency metrics as you're streaming tokens. And I think that coupled with, you know, Sam Altman said it himself in the interview with Patrick Carlson recently, talking about the prospect of open source language models doing a lot of the kind of like vanilla summarization type work that we see GPT-3.5 Turbo, maybe GPT-4 or some degree today doing that those open source models would do those things in the future, potentially even the near future.

• 24:34Speaker 01

  The idea of distilling out, and again, this is not work that we will do, but I think the community real large will probably end up doing, distilling out reasoning machines from these very large memorization machines we've built, and being able to preserve a lot of the reasoning intelligence, sort of prefrontal cortex analogous style abilities, but having models that are much smaller to run, much smaller, are cheaper to run, cheaper to train, cheaper to fine tune, and ultimately more steerable and guidable because they rely less on kind of their own factual understanding of the world. And they rely more on, you know, the human to give them that context for them to be able to make good decisions. So that's something that we're very interested in paying attention to.

• 25:18Speaker 01

  But again, that's sort of more on the language model architecture side or distilling side. And I think there's already great work happening, you know, in the broader community around, you know, Apaca, Laura, and all of these other projects. So.

• 25:32Speaker 02

  Yeah, on the open source LLM front, and, you know, I think there is this debate about how this will play out over time and whether or not, you know, the non-open source vendors of the world are always going to have a lead on the open source projects. How do you think about that? Maybe another hot take is appropriate here.

• 25:55Speaker 01

  You know, maybe to make an analogy, like there are different kinds of cars that are good at different things. And sometimes you need a race car, and sometimes you need a pickup truck. And sometimes you need to go zero to sixty in 2.1 seconds. And sometimes zero to sixty in 20 seconds is fine. And the race car is way more expensive than the pickup truck. So I think that that's one thing I'd say. The other thing that I would say is people are underestimating how much cheaper and faster even, you know, GPT-4 level or future, actually will run in the future. You kind of the the information as I understand it is that these language models become at a minimum 10 times cheaper and faster to run every three years for the next decade. And that's compounding. So 10

• 26:38Speaker 01

  years from now, it will be 1,000 times cheaper to run and 1,000 times faster. And so those are the two curves to watch, right? It's like, number one, what is the cost and speed curve look like? And then number two, what is the capability curve look like? And then number three, what is the intersection point for different kinds of use cases along those two curves? There's maybe more like that's the econometrics way of viewing the problem. So I'll leave it at that.

• 27:05Speaker 02

  Yeah, I love that. Some great analogies in there, too. What do you think are the highest priority things out there to solve? Are the highest the highest priority gaps are any blockers? Or are there none right now?

• 27:19Speaker 01

  Yeah, I mean, we would love to see people doing more around agents. It's likely that they, you know, the internet and Twitter in particular are just like speed runs any hype cycle now. It's like all hype cycles have gotten compressed down to like 30 days or something like that, maybe 90 days. And so certainly agents, again, a la AutoGPT and others, went through this crazy hype cycle, and then they crashed pretty hard. And then now it's on the long road up, the long road building. And so, you know, I think people should not

• 27:50Speaker 01

  Should not take the crash as a sign to run away and should take the crash of any hype as a sign of it's time to build. And so I think agents are a good example of that.

• 28:00Speaker 00

  Yeah. I'm curious on the agent side, would love just your thoughts on the future of that space. You know, do you think that horizontal agents like AutoGPT and others will just get better? And those will be sort of the dominant interaction style for agents? Or do you think there will be many, you know, companies emerge with very vertical specific agents? And people will go to those in the future?

• 28:23Speaker 01

  I think an agent is only as good as its context ultimately. And so the question becomes, you know, it's almost like, you know, getting a language while to perform is less in some ways like doing engineering and maybe more like doing gardening. You know, you want to make sure the soil is really good. You want to make sure you're watering it, giving it sun, not too much sun. You know, it's more, it's more of an organic thing. It's like more, it's more cooking than, than bake, than baking. And so in that way, I think there will be a lot of art to set in the context. And so certainly one could imagine vertical specifications, people building interesting companies in that way. I mean, a very simple use case that is entirely obvious to, you know, probably everybody here

• 29:06Speaker 01

  even listening or once I say it, you'll be like, oh yeah, that's so obvious. Why'd you bring that up? That was a dumb use case is like the agent over your email inbox. You want the little thing to like sit in your email inbox and like help process your emails and reply to emails and maybe it doesn't send them, it just drafts them, that's fine. But you certainly want to remember your instructions. You want to be able to give a feedback about, hey, next time do this and hey, stop doing that. Or you want to be able to, you want it to be able to learn from the, your data exhaust, to look at all of your emails and learn your patterns and learn your habits and learn your tone and learn your vibe. And I think today, maybe it's still just too extensive for the average user, you know, it costs hundreds of dollars per month for the kind of the average email user to like, you know, run that kind of an agent over top of their inbox.

• 29:50Speaker 01

  But as language models get cheaper and more. You know, language, running, running language models on compute you already own. So running a language model natively on your phone or laptop will mean that that becomes much more prevalent and much more popular. So again, it seems like entirely obvious. Of course, you would want a little bot over your email. And yet, like, you see very little of it on, online, surprisingly, so.

• 30:12Speaker 00

  I realize that me asking that question is unfair, given no one can predict what's going to happen in the AI space right now, and it's just changing so rapidly. But I appreciate the, the prediction there.

• 30:22Speaker 01

  Yeah, 10 year predictions are okay. You know, one month predictions are fine, but like two or three year predictions, those are the real killers, you know, because everyone's always wrong, so.

• 30:31Speaker 02

  Do you follow up on the agent question? I think the email use case is interesting. What gets me even more excited is more of the kind of web control, like me having a little thing that lives on my phone that I can tell to go do things for me. Like, how do you think, I, I, I understand this is semi unfair, given, you know, your point about the short term and then the very long term predictions. I guess two questions on that. Like, one of them is, do you think that people are going to have different, different agents doing these different things for them? Or there's going to be, you know, one agent that both helps with your email and because

• 31:11Speaker 02

  it learns so much in the way that you interact, then can be more useful as it goes out and does other things in the, the world for you? Like, is it, is it a multi-agent world for that use case, or?

• 31:25Speaker 01

  I'm sort of biased here, but I'm pretty firmly in like multi-agent camp. You know, smaller, more opinionated agents that can work in a coordinated fashion with other agents to solve problems. I think the one agent to rule them all is a bit of a fool's errand, at least in the short term. Also, as I talked about before, the context isn't really transferable, or at least translatable to other agents. So, you know, it's not that one agent acquires all the context, and that context is not transferable to any other agent. You know, you can plant, another plant in the same garden, and it can benefit from the soil, water, sun, et cetera.

• 31:59Speaker 01

  And so, yeah, I mean, I think like. You know, we need to be semi-practical about, you know, the state of the world and, you know, how this stuff will emerge. And I think that relying on, you know, the lesson of at least traditional deep learning, which again, maybe is wrong to apply here, but lesson of traditional deep learning was to like make the machine learning model do the smallest thing possible, because that's what gave you the most reliability. And then you set up like all the fences and guardrails around that smallest thing possible to get, again, reliable behavior. So yes, large models are clearly a really important capability jump, or in some ways a new thing because of general purpose. But I think that in all likelihood, to get to the level of reliability that people will want, certainly in consumer experiences,

• 32:43Speaker 01

  we will kind of go back to some level of like guard railing or fencing. And, you know, smaller models that are more focused in what they need to be able to do and know about, I think pose a much better path to making that happen, making it work. Could be wrong, though.

• 33:00Speaker 00

  No way of knowing. We'll find out soon, but I would love to sort of transition to where Chroma is going from here. You mentioned sort of publicly, you're working on a distributed version and a cloud version and obviously a lot going on. But I think our audience would love to hear a little bit more about the plans for the future.

• 33:17Speaker 01

  Sure. So essentially, since launch, we've been focused on making Chroma better documented, more fully typed, more fully tested, more stable, more portable, faster. In some ways, the API service itself has not changed significantly since launch. However, people have taken that API that we built and extended it into things that we, I never believed were possible, which is a cool thing to see. You know, good abstractions in software are ultimately maybe the hardest thing to find. And, you know, it seems like we have found a decent abstraction or a very good abstraction in software. So you know, what's next?

• 34:01Speaker 01

  I talk a lot about the journey from demo to production. I think today, Chroma serves the demo or small production use case very well. We do not serve the large production use case as well yet. And we do not serve the, you know, even small production use case with a low dev ops burden. Well, yet either. So Chroma has no hosted offering yet. So our goal is that developers can use the same API that they learn in a Jupiter notebook. Or, you know, some like local node file, JavaScript file.

• 34:39Speaker 01

  And carry that through all the way to production and, you know, millions of users, tens of millions of users and beyond. So in that direction, we need to offer a distributed version of Chroma, open source, Apache too, like everything else. As well as our own hosted offering that uses that open source project to give developers a serverless, initially a serverless experience around vector retrieval, vector search. So that's the first step. That's what we're really focused on right now. I think a lot of other stuff is, again, a bit more up in the air as we've already talked about. We have a lot of DCs about how challenging it can be for developers to reason about Intuit, understand and interpret

• 35:25Speaker 01

  vector data and think there's a large set of workflows and tools that can aid and assist developers in building applications reliably. And so those are certainly things that we want to be able to do. This also touches on the applied research stuff we've talked about, which, how should I check my documents? Which embedding model should I use? All of the above. You know, we want any product builder, any web developer or mobile developer to be able to build a world-class application on top of language models, you know, powered by retrieval, powered by vector search, without needing to be an expert in search or embeddings or machine learning or infrastructure or any of these things. And we think that is entirely possible.

• 36:07Speaker 00

  Awesome. I know a lot of our companies are very excited to get their hands on all that. You mentioned a few times the embedding model side of things. There's obviously a lot of various options for developers today, whether it's OpenAI, Audit 2, or Palm, or others. What do you sort of see the future being in the dynamic between Chroma and the embedding model space and sort of maybe just a little bit more insight into the area of research there from the applied research group?

• 36:32Speaker 01

  I think when it comes to the embedding models themselves, you know, some people who you talk to in industry believe that they're already state-of-the-art, maybe as good as they're going to get by themselves. You know, other people believe that there are more capabilities to be had. Obviously, today, primarily, people are using single modality text. And in many cases, single language as well. So they're doing just text, just English. I know a few different providers have multilingual support, so you can embed in various languages and then retrieve in various languages, and it all just works because it's all just some latent space, which is pretty cool. You know, multimodal embeddings are also pretty cool and pretty crazy.

• 37:17Speaker 01

  So, you know, the simplest version of this is, you know, searching cozy coffee shops in Paris and getting back photos of cozy coffee shops in Paris. And it just works because it just, they are semantically similar, and it's been trained this way, and it just works. There's probably even crazier ideas, like, you know, taking a Led Zeppelin song and figuring out what SQL query correlates to that Led Zeppelin song, which should be interesting when that comes about. Unclear if useful, but really cool. So, and that, you know, that's sort of like, you know, humans have, really, some humans have, like, synesthesia, right? They like, they hear colors and they, they, you know, they taste sounds or whatever. And it's sort of like...

• 38:02Speaker 01

  Language model synesthesia, which is really interesting. Again, I think the use cases we'll have to see, but it seems pretty cool. It will allow visual embedding specifically alongside visual transformers or other architectures that do visual stuff will enable this stuff to go into the real world. And hopefully not in the gray goo, paperclip, Terminator sense. Hopefully only in positive and useful ways. But the text domain is obviously limited to a screen. And I think if anything, the venture community and the software community overrates the power of language models because we are all to some degree word cells. Right? Like we all, our hands live on our computer.

• 38:47Speaker 01

  Our hands live in our keyboards. Our hands literally hover over characters on a keyboard all day. And then our jobs mainly are to type down characters and read characters. And so in that way, we see the world through those eyes. But if you ask even a simple question like can GPT-4 put together a Lego instruction set? As I was just doing the other day with my four year old. You know, probably not, like probably would get very confused. Would maybe operate at the level of a two year old if best. I mean, that will change in probably a fairly rapid fashion. But I think we're still early on that side of things. So yeah, those are the kinds of things that we're excited about with embeddings. I think also some of the things that are very obvious things to do,

• 39:31Speaker 01

  which you've already seen in either OpenAI cookbook examples or in some closed source offerings is the idea of fine tuning over your embedding space. So an embedding space is generally trained on the internet's data. It's not trained on your data. And if all of your data is healthcare data, it probably will be in a fairly small part of the embedding space because it's all healthcare data. And so what you can do is you can give a you can give either a small neural net or even just a layer transform feedback on this is a good match, this is a bad match, this is a good match, this is a bad match. And what you get is a way to take that general embedding space and shift it and shear it and stretch it such that it matches what you want better.

• 40:17Speaker 01

  And I think that that will be totally boiler like it'll be totally a best practice that everybody will end up doing. Because today, people are so amazed by the magical power that it is putting together these few building blocks and be able to do something new. It is it is amazing, genuinely, it's amazing. But eventually, you know, they're going to be tasked with, okay, it does well 80% of the time, but 20% of the time it does not work. We need to fix that. And so you start chasing that further and further and further. And, you know, I think as you do that, you'll want to do a lot of these more like productionization type things, like fine tuning of your embedding space. So.

• 40:55Speaker 02

  Yeah, makes sense. I'm I'm particularly excited about what this the multimodal world will bring us. I think that's a great call out. Let's move on to our rapid fire questions. We're ready for some more Jeff Huber hot takes. What's overhyped and underhyped in AI?

• 41:14Speaker 01

  Yeah, this is like Tyler Cowen, overrated, underrated. But you're not actually giving me specific things. You're just telling me to make it up from scratch. That's even harder. Overhyped, large context windows.

• 41:24Speaker 02

  Why?

• 41:25Speaker 01

  I don't think they're necessary. I don't think most tasks require that much context. You just need to be thoughtful about the context that you provide. And large context is not free. It takes longer, it costs more, and people eventually do care about cost and speed in real world settings. And so I think that it lowers the. Bar to experimentation, which is very good. Makes it easy to experiment about what's possible. But I think that they're not a panacea that Twitter pretends that they are. And even large context windows, there's a recent thing like yesterday, the day before, showing that like it's even suspect how attention scales over context windows. And so, you know, I don't claim to be in deep AI research on these topics and I'm sure

• 42:09Speaker 01

  a lot of these problems will be solved over time, but I just don't think they're a panacea. Underrated, I think at this point, we are maybe the bottom of the hype cycle on agents. So maybe that's underrated.

• 42:19Speaker 02

  It does not feel like that in VC land for what it's worth.

• 42:22Speaker 01

  Okay, all right.

• 42:23Speaker 02

  Maybe different.

• 42:26Speaker 01

  Different spheres.

• 42:27Speaker 02

  Different spheres, yeah. All right, final one, which I know is is a really tough nut to crack and probably a topic of a whole nother podcast or several, but how at all do you think AI regulation should be put in place?

• 42:43Speaker 01

  It is easy to have fear-based reactions to any newly sufficient and mystical technology and to rush into regulating it, putting laws around it that will have unexpected and potentially dire consequences as a result. Ironically, it may be better, clearly you don't want to be too late, but you also don't want to be too early. And certainly the tendency is probably more when it's this fear-based to be more too early than too late. I think that in the same way that I'm more suspicious about artificial superintelligence

• 43:28Speaker 01

  happening in the next 10 years or wherever, I think that also bounds my upper bound on what I think regulation needs to or needs to not do. You know, I also don't have a huge amount of faith in the regulators to make good decisions. And that's in some ways not even a political comment, it's just a reality. And so, you know, industry self-governance is obviously pretty well trod in many sectors. And it seems like there's some effort to do that in this one as well. Clearly, you don't, you know, the case of a bad actor, like stepping out of form and, you know, doing a bad thing.

• 44:14Speaker 01

  Still happens. That still happens in the case of regulation too, but frankly. I think one theory was that AI could be regulated in the same way that, you know, nuclear bombs were regulated because the energy footprint of, you know, centrifuging uranium meant that it was detectable kind of at the global level to have enough power to train, you know, not train, train you to, but to have enough power to centrifuge and still enough uranium to make a bomb. It was detectable. And so we kind of knew, you know, the US intelligence community knew who was doing this. And there's some sort of reason by analogy people are trying to make right now with a certain density of compute.

• 44:56Speaker 01

  And, uh, you know, I'm not even sure that larger models are more dangerous.

• 45:01Speaker 01

  You know, the sort of like implicit in this conversation is that, okay, more parameters will mean more capabilities, more capabilities mean more danger.

• 45:09Speaker 01

  I think that's not even, again, updating your prior to the last 30 days, that's not even obvious by itself. So I think this is a long, rambling answer, but I think that, um,

• 45:21Speaker 01

  I think people should be a little bit on the, on the margin, the average person should be less afraid. And the, um, what would the average person in technology be less afraid? Maybe the average person not in technology should be more afraid.

• 45:38Speaker 01

  And, uh, you know, I think we need to be careful about having, there's this, there's this term called like, do not imitate the eschaton, um, which is basically like every human tends to believe that they're living in like the greatest, but also the last generation. And, you know, this goes all the way back to, um, you know, scares about famine or global cooling in the seventies or, you know, any other disaster. And I think that that kind of thinking ends up leading to really, really, really bad outcomes. Because all of a sudden, when you believe that you are the sole arbiter of the future of civilization, you will make drastically

• 46:20Speaker 01

  Oftentimes terrible decisions to try to preserve civilization, quote unquote. And so I think being a little less eschatological and a little more humble and rooted in reality would benefit certain people.

• 46:35Speaker 02

  Awesome. That was a lot of great insights and a very thoughtful answer there. Okay, one final one and then we'll wrap up. Is there anything we didn't talk about you wish that we would have covered?

• 46:46Speaker 01

  The talent side of this is really interesting. Is it too late to learn this stuff? Like, you know, has this ship already sailed, either for all the application that you use cases, or, you know, if anybody listening out there wants to become an AI researcher, you know, have gotten to a kind of runaway research and there's no way to ever catch up, or no way to make an impact. And to answer my own question, I think that it is still day zero. And I'm sure you all would agree, but it is not too late for us to start a new company. It is not too late to go do a PhD in this stuff. It is not too late to, you know, join OpenAI. It is not too late to join Chroma. Like, it is still early days. So lots of fun stuff to be had.

• 47:32Speaker 02

  Totally agree.

• 47:33Speaker 00

  Yeah, we love that. And we've seen plenty of really smart entrepreneurs that have dove in really recently and are already doing amazing things. So very much agree with that. This has been awesome. I mean, thank you so much, Jeff. I'm sure our listeners will want to continue to hear from you and would love to just hear a little bit about where they can find you and follow you.

• 47:53Speaker 01

  Definitely. Twitter probably ends up being the best place. I'm Jeffrey Huber on Twitter. Chroma's Twitter account is trychroma, T R Y, chroma. If you have a question about using Chroma, come in the Discord, you'll see me, get your questions answered.

• 48:09Speaker 02

  And you said 40 tweets from trychroma so far, right?

• 48:13Speaker 01

  That's right. It's fairly, you know, low commitment. We like to be high signal, high signal, low noise. So it's like, it's an easy follow.

• 48:20Speaker 00

  Love it. Well, Jeff, thank you so much. This has been amazing and really appreciate you hopping on with us.

• 48:26Speaker 01

  Thank you all. It was a great conversation. Appreciate it.