E5: The Embedding Revolution: Anton Troynikov on Chroma, Stable Attribution, and future of AI

We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. (0:00) Preview (1:17) Sponsor (4:00) Anton breaks down the advantages of vector databases (4:45) How embeddings have created an AI-native way to represent data (11:50) Anton identifies the watershed moment and step changes in AI (12:55) Open AI’s pricing (18:50) How chroma works (33:04) Stable Attribution and systematic bias (36:48) How latent diffusion models work (51:26) How AI is like the early days of aviation (56:01) How Disney inspired the release of Stable Attribution (59:53):Why noise can lead to generalization (1:01:04) Nathan’s KPI for The Cognitive Revolution (1:01:59) Other use cases for embedding (1:03: 19) Anton touches on the applications for biotech (1:04:35) Anton on doomerism hysteria and what actually worries him (1:11:43) - Nathan sums up a plausible doomer scenario (1:20:17)What AI tools does Anton use and why? (1:22:55) Anton’s hopes SPONSORS: SHOPIFY | OMNEKY Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive *Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Twitter: @CogRev_Podcast @atroyn (Anton) @labenz (Nathan) @eriktorenberg (Erik) Join 1000's of subscribers of our Substack: https://cognitiverevolution.substack.com/ Websites: cognitivervolution.ai trychroma.com/ omneky.com Show Notes & references: -https://same.energy/ (Beta) -Wright Brothers Bio (https://www.amazon.com/Wright-Brothers-David-McCullough/dp/1476728755) -Ajeya Cotra article in Less Wrong (https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to) RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Anton Troynikov?

  ML models are kind of weird in the way that they think quote unquote about the world and they might represent something quite differently to what a human would. They might focus on quite different things because they are mechanistically pursuing an objective. They are not trying to encode any meaning. The Wright brothers were wrong about how the right flyer worked. They just had intuitions about it. And if it flew, that's an object demonstration. It doesn't really matter if your intuition is wrong if the thing works. And then there's this Cambrian explosion of different types of aircraft because we all we had to go on was intuition. We didn't have engineering principles yet and we didn't even really have the tooling to develop those engineering principles. That's where we're at today, in my opinion, with machine learning. And one of my favorite things about

• 0:40Anton Troynikov?

  Kurzweil is I went on the website not long ago and it was like the 10th anniversary of the singularity is near. And that just, I don't know, that made me laugh.

• 0:51Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg.

• 1:13Speaker 02

  The Cognitive Revolution podcast is supported by Omnikie. Omnikie is an omni-channel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms with the click of a button. Omnikie combines generative AI and real-time advertising data to generate personalized experiences at scale.

• 1:33Nathan Labenz?

  Anton Troynikov is a co-founder of Chroma, the AI-native, open-source embedding database. In the midst of an AI gold rush, Anton is selling shovels. Embeddings, for any who don't know, are numerical representations, generally speaking, a single vector, or simply put, a list of numbers, that encode an input. That could be text, an image, an audio file, or even a DNA sequence or protein structure into a lower-dimensional latent space.

• 2:05Nathan Labenz?

  The embedding process is itself learned, such that semantically similar inputs cluster together in latent space, and the resulting embeddings can be used to compare inputs, to power retrieval systems, and for additional processing. Embeddings are a key piece of a huge number of AI-based projects right now. By embedding a data set, whether it's a user's email history, or Google Drive documents, or even a company's entire knowledge base, developers can enable semantic search at runtime and include the most relevant background information in language model context. This dramatically improves performance by reducing hallucinations and increasing personalization. We talked to Anton about embeddings,

• 2:49Nathan Labenz?

  the trade-offs and optimizations involved with working with embeddings at scale, their recent Stable Attribution project, which flexes the power of Chroma by taking a stable diffusion-generated image and attempting to determine which of the 5 billion images in the Lyon training set had the most influence on its creation, but which also quickly became a flashpoint in the broader debate about how artists and other rights holders should be credited and compensated for their contributions to generative models. We also covered the big picture of recent AI developments. Anton is a deep thinker with a sophisticated conceptual understanding of latent space. I learned a lot from this conversation, and I hope you do too.

• 3:32Nathan Labenz?

  Anton Shoynikov, welcome to the Cognitive Revolution.

• 3:36Anton Troynikov?

  Glad to be here.

• 3:38Nathan Labenz?

  So tell me first about vector databases. I think our audience is obviously interested in AI. They most probably have at least a superficial sense of what a similarity score is, what a dot product is. But take me to how that works as a database, and then we'll get into some of the optimizations there as well.

• 4:00Anton Troynikov?

  Yeah, sounds great. So let's kind of start at the beginning.

• 4:05Anton Troynikov?

  What a vector database allows you to do is if you can represent your data as a vector, it makes it possible to perform geometric operations on your data. So traditional databases have, let's say, algebraic discrete data structures, right? Like a SQL table, you select from where and it has concrete properties and it returns exactly that result. With a vector database, data is represented as vectors in high dimensional space. So you can perform operations that have geometric meaning instead, like, you know, distance, you already mentioned similarity scores, you can talk about densities meaningfully, you can, you know, fit surfaces to them, you can cluster them in a way you can't in a, in an algebraic, you know, form of data structure, discrete data structure.

• 4:47Anton Troynikov?

  So what, I think the reason that they're exciting right now is because we now have powerful models that can transform all kinds of different types of data into vectors. And it can transform them into vectors in a meaningful way. So traditionally, that's been text and a lot of these embedding functions have been kind of hand rolled. But now with the success of, you know, transformer models and attention based models for, for modeling text, the vectors that you get out from those kinds of models are much more meaningful. They, they represent structure like, like a human would impose structure on the resulting vectors. So, you know, sentences that, that talk about similar things, they end up close together in vector space.

• 5:32Anton Troynikov?

  And that's kind of, what that's done is turn on a lot of experimentation and development with using embeddings as kind of the AI native way to represent data. And so I'll give you an example, a typical workflow that you see is you want a model to answer questions about specific information. So ChatGPT, for example, it's trained on an enormous amount of data. You can ask it questions about anything, but it might hallucinate, it might not have the information that you want. And certainly it might not have your personal private information that it didn't, you know, it wasn't trained on because it's not available anywhere else. You still want it to be able to answer questions about it because it's good at general question answering as a task.

• 6:16Anton Troynikov?

  So what you can do is you take the corpus that you want it to answer questions about. And then you embed it, you can embed it using, there's a bunch of open source models, you can also use OpenAI's API. Here's API and transform that data into vectors. And so you have the data available as vectors with the actual text associated with it, with that representation.

• 6:40Anton Troynikov?

  And then what you do is you query it and you query it by again, taking a query, embedding the query text, performing a nearest neighbor search or an approximate nearest neighbor search in the in the vector database, returning relevant documents. And then what you do is you take your question and you take the relevant documents that it found. And then you ask the model specifically to answer the question with the documents that you've put in its context. And then it's much more accurate and it's answering questions about the thing you actually wanted to answer questions about. With data that wasn't available to it at training time. And this is a really powerful technique. It pretty much suppresses hallucination. It allows you as an organization to use large language models without fine tuning them, which is expensive and slow. And it confers a lot of other advantages as well, which which maybe we'll get into in time, but that's that's kind of the basic idea.

• 7:25Anton Troynikov?

  Vector databases have been around for a while. Pretty much any service that you use that has a recommendation or like a similarity thing. So the one that I always think of is Pinterest. It'll show you similar images and the way that that's being done is because they have like these images encoded into vectors and they've got a large vector database. So that like when it wants to find nearest neighbors, it'll it'll look for that and you get visually similar images. Because their embedding function produces visually similar images for vectors that are close together. But the use cases for these things are exploding now. Because the embeddings for all kinds of data became available and it's super cheap and you could just hit OpenAI's API. You don't need to have a run model anymore. And people are building all kinds of really cool applications on top of this stuff.

• 8:06Speaker 03

  Yeah, okay, there's a lot to dig in there and follow up on. For one thing, you know, just to contextualize a little bit, I would imagine a lot of our listeners have experienced querying SQL databases. So I would love to get a little bit of a sense for kind of how the, you know, DBA life is different as we move into a vector world. I think you spoke, I'll just give a couple, you know, additional notes too on like, there is so much energy around these kind of chained retrieval and language model systems right now. That's really at the heart of kind of these dreams that everybody is dreaming.

• 8:51Speaker 03

  Everybody sees ChatGPT and everybody says the same thing that you said, which is, well, man, if this had my data, you know, if this could go read all my emails, if this could remember every text message I've ever sent. And then obviously, the, you know, the business use case is even bigger and probably a bit, you know, a deeper need because now you've got organizations and all of our Google Docs, you know, and our whole now, I mean, how many companies have rolled out these knowledge bases or the wikis and, you know, but do those things, can anybody even locate the information in there? Generally not, right? So this, this vector database kind of sits at a really critical point in that value chain. And you've started a company that is a vector database company.

• 9:34Speaker 03

  So it does, before we go into even more details, so let's talk about your company specifically.

• 9:37Anton Troynikov?

  Absolutely. So I want to say one specific thing here. Chroma uses a vector database as a technology, but the thing Chroma really is, is a platform for basically embedding knowledge that your machine learning enabled applications can use. You can think of us as the thing that acts as the storage layer for applications that use large language models in the loop. So we're much more than a vector store. And one of the things that we have to really learn while we were building this product was the affordances of the existing vector stores on the market don't really suit this use case very well at all. And we actually built this product in-house and we were using it in our own product experiments for a while.

• 10:20Anton Troynikov?

  Chroma started by first, Chroma basically started on the principle that you can understand a model's behavior based on how it sees its data, basically. And then embedding is a representation, is a model's representation of its input data. And so our early, all of our product experimentation was around like running algorithms and embeddings, representing them, showing them in different ways, manipulating them in different ways. And we needed something really easy to use for developers. We needed something that would allow us to rapidly experiment, maintain development velocity while still being very performant. And so when it became clear, and you know, OpenAI did a lot of good work here in marketing the power of embeddings to people. And these sort of early use cases showed up with document question answering. It became clear that wait

• 11:03Anton Troynikov?

  a minute, you know, there's thousands or hundreds of thousands of developers in the world that are going to need something exactly like this. And we essentially took what we built, packaged it up to make it a lot easier to sort of just get up and running and then gave it to people and people really seemed to love it. Things have been really exploding over the, I mean, it's kind of crazy to me because it feels like it's been out for a while, but we launched it last Tuesday was when I announced it on Twitter. So it's kind of been a lot since then.

• 11:29Speaker 03

  Yeah, time is doing very weird things for me as well. I, earlier this week, I realized, it was on Wednesday that I realized, wait, we just did that on Tuesday? And it was like, oh my God, so many things happen in such a small period of time in the AI world that is dizzying to say the least.

• 11:50Anton Troynikov?

  Yeah, it's, I think, I feel like there's been a few step changes, right? Over the last couple of years. I think, I feel, so first of all, I think GPT-3 was the watershed back in 2020. And then stable diffusion, DALL-E, but stable diffusion was the next really big one because it allowed people to see that, wait a minute, you don't actually need to own a huge model. I can totally just run this on my laptop and get results and build things around it. That was a huge watershed moment because it was open source and like small enough to run on an M1 Mac. You can do it yourself. Someone was crazy enough to get it to run on an iPhone, which I thought was amazing. And then the next thing that just recently happened was ChatGPT. And technologists will tell you the same thing. People have been like following closely for a while will tell you the same thing. They're like, oh yeah, the model was already available. It just needed fine tuning. Nobody

• 12:33Anton Troynikov?

  did a blah, blah, blah, blah, blah. It doesn't matter. What matters is it's packaged and given the right affordances so that people can easily understand what it is they can do with it. And then they'll start doing that and then they'll start experimenting. So I think those are the two big watershed moments that happened in the last like while that that that really makes it look like a amazing playground for experimentation right now.

• 12:55Speaker 03

  So you mentioned a couple of things that I just wanted to elaborate on a bit. One, It sounds like you guys are a neutral database, vector database technology. So I can use any different provider. You mentioned OpenAI. Their pricing, just to give context on exactly how cheap it is, because you said it is is cheap. The cost to embed all of the transcripts of every podcast that Joe Rogan has ever done, a hundred thousand pages of text, forty dollars to embed that hundred thousand pages. And if you wanted to go, this is from Boris Power, who said OpenAI.

• 13:34Speaker 03

  If you wanted to go to the full web scale data set of ten to the eleven pages of text, then you're looking at, he says, only fifty million dollars to embed. So, you know, that's not nothing, but to embed the entire, you know, web scale databases, it actually is amazingly cheap. So that once you have a document, you know, or whatever it is embedded, then you could also store those embeddings. You don't have to, unlike the other APIs, you know, that are generative and doing something new each time. The embedding is kind of a one shot. You know, there's no temperature setting. You're going to get the same thing out every time. So then you can store it and that's where your platform comes in.

• 14:19Speaker 03

  You mentioned that there are some shortcomings of the things that were out there. So what shortcomings did you see and how did you address those in the product that you built?

• 14:26Anton Troynikov?

  Yeah, there's a great questions. I think the first shortcoming was just how hard it was to get started with anything out there. Like we evaluated literally everything else in the market and it just didn't fit our needs of like rapid development, rapid experimentation. It was either too hard to like deploy and keep running because it was really designed for heavyweight workflows from day one, or it like didn't really provide the kind of price performance point that we needed. Or it was just like frankly, too complex. Corba for a long time was just me and my co-founder Jeff Huber. And we couldn't spend like an entire full-time engineer's worth of work just running the vector store part. We needed something else.

• 15:08Anton Troynikov?

  And the other thing is, is we found the abstractions were pretty wrong. There was also like a bunch of missing features, which which we just couldn't figure out why they were missing. We just implemented them. There was a bunch of other things too, which which kind of made it obvious that the affordances here were wrong. Like none of the other products on the market are really like AI native. That's the way we think about it. And that's because like. You need to, like, if you have this app that's LLM-enabled and it has like a store, you need a much more like transactional approach. You need good inserts, updates, leads. You need to, you know, you need to have it talk to the embeddings API properly. You need to have it return the results in a way that the model is going to pick up easily and then stuff them into the context window, all of these things.

• 15:48Anton Troynikov?

  And we also saw in parallel the growth of what we think of as application development frameworks for AI, LLMs, our friends LangChain, for example, where we were a day one integration partner when Chrome launched with them. Like, we saw that, okay, the thing that we build has to actually fit what people are doing. It doesn't, it shouldn't fit this like abstract idea of vector database. It needs to be an app development platform. It needs to be the storage layer for app development and then it needs to have the right affordances to think about this so that developers can actually think about it in the right way. Those are kind of the main differences. That said, like, it's pretty performant. We're making it go even faster pretty soon.

• 16:25Speaker 03

  Pre-AI, what were the main use cases for vector databases that people had built all these other things for?

• 16:33Anton Troynikov?

  Yeah, look, the two big ones, as I mentioned earlier, are recommender systems and semantic search. Prior to embeddings, embeddings are a concept that basically just means take data from one high-dimensional space and encode it as a vector in a lower-dimensional space. That's all embeddings really means. It's actually not even a deep learning concept. It predates it by a long time. It's an idea in mathematics, which, you know, the name is like loosely used now, but that's pretty much what it means. And so there were a lot of, like I mentioned, especially for natural language processing, like hand-tuned methods to convert words into vectors or convert like corpuses of documents of text into vectors. And then, you know, you would try to hand-engineer

• 17:16Anton Troynikov?

  these embedding functions and then store them in a vector database and then query them and look them up in a similar way to the way I've described. And what people focused on in that era was building these really large-scale databases. It's intended to work at very large scale. So we're talking about like... By, you know, being able to query billions of documents out of the box, being able to like search over billions and billions of images out of the box for sort of these like web scale, huge applications. And at that time, like AI just wasn't in the picture. Deep learning really wasn't in the picture, so.

• 17:54Speaker 03

  So do you also have like a sort of SQL like aspect to the database? Where, you know, you have a, could I, could I do like a sort of equivalent of like a where clause? Where I'm like, but I'm only interested in these, do you have kind of traditional B-tree indexes along with the, the vector space? Okay.

• 18:15Anton Troynikov?

  That's right, that's right. It allows us to, so we, we have the underlying algorithm that we use for the vector index. And the index is the thing that actually stores the vectors and the relationships between them. And then there's a metadata store and the metadata store stores the data in like original, more sequentially form. And we have them talking to each other in such a way that you can take something that filters the query that you perform in this very traditional way, and then turns that into a query onto the vector store. So I can say, you know, find me all the matches to this query, but only from documents that contain this phrase. And like, you can just do that with Chrome.

• 18:50Speaker 03

  So one of the things you said about the technology that caught my ear was the, was the phrase approximate nearest neighbor search? So tell me about that. That, that sounds, and this also starts to preview a little bit the, the project that you've released and made some noise on the internet with, which is stable attribution.

• 19:12Anton Troynikov?

  Exact nearest neighbor search is, has like N squared complexity. Each time you add a new vector and you want to query it to find out what the nearest neighbors are, you need to perform an N squared calculation. Like it grows in the number of vectors quadratically, which is actually for certain, for certain sizes of data is totally fine. Roughly, if you have like, I don't know, a couple of hundred data points is fine. You should just be doing exact nearest neighbor. It's just one matrix multiplication and that's very fast. NumPy calls out to like very fast numerical libraries. You can just do a matrix multiplication, keep it all in memory. It's nice and fast. You don't have to think about it. The issue, of course, is quadratic growth is really bad. So by the time you have, you know, north of 10,000 data points, that n squared is really starting to hurt you.

• 19:56Anton Troynikov?

  And this is where approximate nearest neighbor comes in. And there are several of these algorithms, but basically all approximate nearest neighbor algorithms are flat in the number of, or approximately flat in the number of vectors in the index. So you no longer have this n squared term. Your query times, regardless of how many vectors are stored in the index, are going to be identical. But there's a trade off. The trade off is you won't necessarily get all the nearest neighbors every time. You might lose some. And that's a, that's called a precision recall trade off, and it's very similar to other aspects of machine learning in that way. But in practice, in many applications like these, that's a trade off that you can make. And the trade off isn't very severe. It's not like you're going to lose half. It's more like you might lose one in every 100 queries.

• 20:40Speaker 03

  Yeah, interesting. Okay, so that makes a ton of sense if you're working at a 5 billion vector scale. How do you think about kind of, when you said 10,000, I was thinking, I'm sure I have more emails in my Gmail than that. And so what would, how should I think about kind of these sort of mid-scale personal data sets where I probably do want the full thing, right? Like, I don't want to miss the one thing that I'm, that I most need. So yeah, tell me, in that mid-scale, what happens?

• 21:14Anton Troynikov?

  Yeah, so you can, especially for smaller data sets, you can tune for more recall. You can make it very unlikely that you'll miss anything, but still get that good query performance. You can also trade off processing upfront and sort of insert, you know, time to insert something with recall and query performance. Depending on what your workload looks like, basically. If you have a lot of inserts, if you want it to be fast, it's probably a good idea to, you know, tune it to be a little faster so that insertion speed, little bit rest recall. If you're like, insert, if you have like a set of data and you really need to insert at once, do all the preprocessing on that side. You'll still get great recall.

• 21:51Speaker 03

  So the tuning, it's interesting, the, there's multiple meanings of tuning. You're speaking about a pretty classic database tuning paradigm, which is like, when exactly do we do stuff? And do we pre-calculate? So I had listened to your conversation with Paki on the Not Boring podcast not too long ago. And actually, as I was listening to that, I thought you were tuning some sort of learnable, you know, like fine tuning, some sort of aspect of either the embedding or, you know, the way like which, so you're doing that as well, like which dimensions, you know, would be kind of primary. So okay, so tell us about that side as well.

• 22:33Anton Troynikov?

  Yeah, so this is the other, this is the, again, like I mentioned, one of the most important parts is you can perform geometric operations. And because of the way that embeddings work, those geometric operations have meaning. So for example, um, kind of the simplest example of this that I can think of is you can cluster vectors geometrically, but those clusters will have semantic meaning. So if you have a large corpus of text and you look at what the clusters might be in it, and you look at the topics that are in those clusters, that's information that like belongs together because the model is trained to keep things that seem together together, right? Clustering is like this basic first approach thing you can do. Another thing you can do, and this is actually great for the question answering application. We've been examining this in some detail lately.

• 23:16Anton Troynikov?

  You can find a direction in the vector space that represents some human understandable property. And it's actually surprising how often this comes up. But for example, you could be talking about a topic, but you could be talking about it in the context of a question, like who was born in year, blah, blah, blah. And then over here, you could be talking about it like, oh, this king, et cetera, was born in blah, blah, blah. And so if you find this like question, answer vector, you can do interesting like re-mappings and bring things closer together. And because, um, because it's just a vector space, it's this well-understood, fairly straightforward mathematical structure. The way to like transform one thing that you want into a different thing that you want,

• 23:59Anton Troynikov?

  or like make it work better, fine tune it to get better results for you is actually very straightforward compared to traditional machine learning. You don't need to do backprop, you just need to estimate some transforms, which is something that we know how to do.

• 24:10Speaker 03

  That sounds pretty similar in some ways to like classifier guidance almost, right? Like you're sort of identifying a direction that is the the direction from statement to question, essentially, and then you can kind of move around, you know, you could you could place that vector anywhere in theory and kind of move from different, you know, statements to their corresponding questions. Obviously, it's going to be a little messier than that, but.

• 24:39Anton Troynikov?

  But there's also all kinds of other things you can do directly by working with this. For example, if you have a system to get user feedback about relevance of the return results, you can reweight your your space by applying a trend, a learned transform on top of it to get ever more relevant results from human feedback. In a very straightforward way, there's no model retraining that you need to do. Your model is doing what it's best at, which is like understanding and recomposing knowledge. And actually, that that takes me on an interesting tangent here, which I really do want to indulge briefly. Right now, we're seeing like, oh, chat GPT and then GPT three point five and GPT four soon, which are these large models that know things and like they're trained on very large corpus of data. They're very expensive to train. They're expensive to fine tune.

• 25:24Anton Troynikov?

  You don't really know what knowledge is in them, but they're very good at general purpose tasks. They are, you know, they can do all kinds of tasks efficiently for you with like zero-shot learning or or like multi-stage prompting. And I think those models are going to be around. I think they'll continue to be there because they're such great, like they're almost like utilities, really. But on the other side, I think what we're going to see is these smaller leaner models, which are actually trained to find and compose knowledge in response to queries rather than store knowledge in their own weights. And we've seen early signs of that. There's a paper called Retro or there's a system called Retro. It's from Meta. Oh, is it from Meta? I think it's from DeepMind. Sorry, I misspoke. Which where they trained a model to do exactly the sort of stuff we're talking about. They

• 26:05Anton Troynikov?

  trained it instead of sort of storing in knowledge in its weights, they trained it in such a way that it has to search for knowledge in this databank. It can't get knowledge from anywhere else except in this databank. And that and you know, that's, I think that's one direction in the future. And we're starting to see the very first stage of that happening right now. And and the sort of stuff we're building with Corma really enables that.

• 26:24Speaker 03

  So those transforms that you described, those are just like one layer, like you're you're basically just learning one additional vector?

• 26:34Anton Troynikov?

  For a while, there's been folk wisdom, and it's actually even in the OpenAI cookbooks, OpenAI cookbooks, there's been this folk wisdom that a linear transform, which is just a matrix multiplication, is enough to transform one embedding space into another embedding space, as long as the semantics are broadly the same. And there was recently another paper about this, where they demonstrated that empirically, models tend to learn the same representations of the same thing, so mapping between them is a straightforward operation. And that kind of means that that's very promising for being able to do things like that and integrate it really well with what we're building.

• 27:06Speaker 03

  Like, relative representations, is that the paper you have in mind?

• 27:09Anton Troynikov?

  I think so, yes.

• 27:11Speaker 03

  Yeah, that was a fascinating one. You see these kind of, I mean, obviously, it's a higher dimension than projected back down to an even lower dimension for visualization, but you see these kind of shapes where you're like, oh, wow, that's very strikingly the same, subject to a rotation, or a dilation, or whatever. I thought that was pretty awesome. The clarity that they achieved in demonstrating that, I thought was pretty cool.

• 27:34Anton Troynikov?

  It was great empirical work, and honestly, I think more machine learning research needs to take that empirical approach. I think machine learning people need to take more cues from biologists and really observe what's going on first. There's one extra interesting thing about, you brought it up just now, you saw how these embeddings seem to be invariant to rotations or scaling, right? There's a very interesting, there's a very interesting potential approach here, which is, obviously, not obviously, but in principle, you could find ways to maybe decode people's information from the vectors. So if you're a provider of, say, cloud vector storage or something, and people are sending you their data, it might be a risk of you being able to read that, and people don't want to know that.

• 28:18Anton Troynikov?

  But, most of the operations that you perform in vector space are, like you mentioned, invariant to translation, rotation, and scaling. But that means that you can put your, even if I know that your data came from, say, OpenAI's Ada2 model, if you apply a random transform, and maybe add some noise, it's very, very, very difficult for me to recover that transform, but I can still perform the same operations that you want me to perform in that space. And we haven't evaluated this in depth, it's an idea that I had fairly recently, but you can think of this as a form of homomorphic computation, because I can perform computations for you without really knowing what your data contains. And I think that's a really interesting thing as well, and that might be a whole direction on its own, in a little while.

• 29:01Speaker 03

  I mean, the security, I'm working on a couple of projects where I'm like, boy, that actually could be a great solution to an otherwise thorny problem of where exactly are we sending our client data and who is seeing it in what form? So yeah, that's, that's a really interesting insight and a great connection. I mean, there's so many, the convergence of so many things is just striking to me. We just did another interview with the authors of BLIP, the computer vision model. They just came out with BLIP 2. And BLIP 2 reduced the training computation requirements by 95%. They were able to train the whole thing on a single machine in under 10 days.

• 29:44Speaker 03

  And the reason that they were able to do that is because they kind of created an ensemble approach where they had a pre-trained vision model and a pre-trained language model. And they just trained this connector model between the two. I think they said it was 200 million parameters. And what it predicts are text embeddings that are derived from the images. And so then they're able to inject this representation of the image, skipping the embedding layer, just going straight to injecting the embeddings as kind of a, you know, prefix to the prompt

• 30:21Speaker 03

  in a, in a way that is, well, the thing that I thought was kind of most fascinating about that is they're accessing a part of that embedding space that no text could in fact access. We talked quite a bit about like, is there any way to sort of reverse that to like, what text would you have had to put in to get those embeddings? And they were like, yeah, there's really no text that would create similar embeddings. We're just finding these, you know, kind of weird spaces, but the language model knows what to do with it. But, you know, it's hard not to see the analogy between like the vision model being sort of the eyes and their model being kind of, you know, some sort of relay layer. And then you get the language model seems like an executive function of some sort. You're kind of doing a similar thing, right? Like you're connecting.

• 31:06Speaker 03

  I mean, is that how you sort of casually would describe it to people? Like, the language model is kind of the, you know, the executive function or the sort of top-level processing and then you're helping get into deep memory?

• 31:19Anton Troynikov?

  I think that's a, I think that's a reasonable analogy, yes. I think that's a reasonable analogy. I think another paper that's in that vein, you've just jogged my mind, is Robotics Transformer, RT1, came out from Carol Hausmann's group at Google Brain Robotics. And what they do is, yeah, they have like a perception subsystem that sits on top of everything and then they have this planner and the planner just basically learns for these multi-head embeddings to like output plans and actions. It's really, really cool. It's kind of similar to the system that you described.

• 31:47Speaker 03

  It's all coming together pretty fast.

• 31:51Anton Troynikov?

  It's, it's really an interesting time to be working. I think that like, yeah, I think that there's also just untapped veins of research here. The reason, you know, we noticed this very early on, even before we started Chroma, that the incentives between, let's say, academic ML AI research and industrial academic AI research, and I don't mean universities versus, versus like industrial research labs because they're mostly doing similar work now. I mean more like the sort of work that production machine learning deployments are interested in versus the sort of work that maybe pushes you towards AGI is fundamentally different. And one of those big differences is in academic research, there are accepted community benchmarks

• 32:33Anton Troynikov?

  and your aim as a scientist is to demonstrate the performance of your model on these accepted benchmarks, right? But the benchmarks are static, whereas in the real world, the data's always changing. And, you know, questions of like, should I train my model? What do I train it on? Is it working better or worse? Monitoring it, measuring it are much more salience than demonstrating performance on benchmarks. And kind of, you know, we founded Chroma with that observation in mind, that there's actually a lot we can do and mine some of these restraints of research if we just focused on them in the first place.

• 33:03Speaker 03

  Going back to nearest, you know, approximate nearest neighbors, we talked about kind of various tuning you can do where you can kind of pre-compute some stuff depending on your workload, where you can learn these transforms to get more and more relevant over time. Do I still have to worry if I'm doing that, that there's some sort of systematic blind spot in my system? Like if I'm, you know, it just to start to bridge to stable attribution, if you've applied all these techniques. And I upload, you know, an image that I got out of stable diffusion.

• 33:39Speaker 03

  Is there any, is there any way that we could start to assess like, whether or not there's some sort of systematic bias there? Or is there, is there some part of the latent space that's like, you know, being kind of unfairly not, uh, you know, not brought to the fore in the attribution?

• 33:55Anton Troynikov?

  I think so. Um, I think so. And actually, again, there's an experimental feature we're deploying into Chroma very shortly, which will tell you the relevance of, uh, what's been retrieved. So you can actually do something about it. You probably, you probably know there's like a two-step retrieval process as well. Uh, it's called Hyde, where instead of just embedding your query, what you do is you send your query to a language model. You let it hallucinate whatever it wants, a response, but it looks more like a response. Then you embed that and do the query, but it's expensive. It's expensive to do two like large language model calls. We did some napkin math not long ago and we figured out that, for example, one company who we've been talking to, if they did it, um, if they did it even the vanilla way, it would be hundreds of thousands of dollars per week.

• 34:38Anton Troynikov?

  Right? So reducing the cost of this is actually really important. And so what we realized pretty quickly was we could make this Hyde approach more efficient if we could algorithmically figure out when the results would be relevant or not. And so we're deploying that very shortly and it's a very general idea that I think people also integrate into their sort of app development flow. And, and it again, because all this stuff is geometric and nicely continuous, we give you a probability instead of just a, just a threshold of yes or no. So, yeah, but again, pretty much everything we're talking about, more research, more work needs to be done.

• 35:11Speaker 03

  Tell us, give us the, you know, the brief overview of stable attribution. And then, you know, those who want to go deeper on that topic specifically, definitely recommend the show that you did with, uh, Pacquiao on the Not Boring Podcast feed. But give us the, you know, the kind of short overview and then I have some questions that you didn't get to cover on, on that conversation.

• 35:29Anton Troynikov?

  Absolutely. So the short version is fairly straightforward. We saw a need and a possible technical approach to figuring out, or at least starting to figure out how images in stable diffusion's training set influence particular generations that are created. Uh, and the way that we did that is by doing various, um, by extracting various information, both about like,

• 35:52Anton Troynikov?

  Like just similar images to a given generation, the training set, but then also like figuring out what the right similarity is. And the reason we took this approach is again, because there are optimal approaches, but they are all computationally intractable. For example, like one more optimal or more principled way to do this might be to remove examples one by one until, uh, until, and like retrain the model every single time until, you know, the image has gone far enough away, you say, okay, those images influenced the most. But of course, every training run of stable diffusion costs about a hundred and sixty grand. So it's computationally infeasible. We found a different approach. We think it's principled and we applied that and we leverage other, you know, various properties of how diffusion actually works alongside like similarity and latent space.

• 36:36Anton Troynikov?

  We construct like a, what we think is a principled similarity metric and go from there.

• 36:41Speaker 03

  Cool. So give us just a little bit more of the principles behind it at least, and then, uh, then I'll get into a couple of questions.

• 36:48Anton Troynikov?

  Yeah, sure. Uh, assuming, you know, assuming the audience here knows basically about how latent diffusion models work. They're trained, uh, in their training set, they have image text pairs, um, and they're trained to reproduce, uh, and the same encoded latent as the training example, um, through the diffusion process, they're trained by iteratively adding noise to the training examples latent vector representation and then training a network that reverses that noise. Uh, and what's great about it, what's actually the big breakthrough is this is very efficient, both in terms of how much data is used by each dimension in the vector, but also because you can train in parallel, um, each denoising step. That's the big thing. So, you know, there, there's a few parts to this. The starting point is like, okay, well, given that the model's training objective is minimized,

• 37:32Anton Troynikov?

  if it can exactly reproduce the latence of all of its training examples, and because we know that the latent space is continuous and smooth by definition, because that's how backprop works, you have to be that, then it's not unreasonable to say that training examples that are near, uh, the generation in latent space in some way are influential to the output of that generation. So that's where we started from. Um, we took a look, we found out a bunch of interesting stuff. Uh, we found that for example, you know, there, there is a bunch of papers that demonstrated that actually stable diffusion totally can reproduce exactly, or very close to examples in its training set, which people denied for a long time. I don't know why. Uh, now, now they tell me like, oh, you can, we've just a tiny number of examples. Like, it's not what you were saying before.

• 38:13Anton Troynikov?

  But anyway, we found that, you know, noise in the training set is definitely, uh, something quite interesting here because the clip representation of images and text, um, might differ for given example. And you can imagine why the same image might have a lot of different captions, even in different languages. Uh, and then a lot of the data is just noisy. So we took that into account. And we took into account, um, the sort of the attention mechanism, uh, of the diffusion model itself by constructing attention maps. And attention maps are actually like, if you, you look at them at a slightly different lens, they're actually just, again, a matrix multiplication. They're a linear transformer that vector. So when we apply the attention map, we get, you know, it's as if we're transforming the latent space and then we can do similarity search in that new latent space. That's like a very short overview.

• 38:55Speaker 03

  Cool. So yes, again, for more, uh, depth on that, go to the not boring, uh, Anton Teich's packy about AI, episode six. And there is quite a bit more, uh, detail there. So just kind of zooming out from the technology a little bit. I naturally went online and tried it. And I tried it with a mix of images that I had made with stable diffusion. I know that it's not really meant to be used on natural, you know, images that I, that I took on my phone. Uh, but of course I tried that as well and it does work. And, you know, either way, you do get images back that are similar. The, the divergence definitely between the real images, you know, like stuff on my phone and what comes back is, is bigger.

• 39:41Speaker 03

  Um, and I think you have a explanation for that, uh, around kind of, or why it doesn't come back with so similar images, even though I, I mentioned that they must exist, right?

• 39:52Anton Troynikov?

  Yeah, sure. Here's, here's the interesting point. The similarity is not perceptual human similarity. It's models, it's how the model perceives the difference. That's an important thing to note. It's, when I say influence, it's not necessarily the same way that an image might influence a human artist where they would draw motifs and inspiration from it. It's more like. This very mechanical, raw interpretation, like these vectors are the same, right? These vectors are similar. So to a human perceptual system, once those images are decoded from their vectors, they might look pretty different and you won't know what's similar about them. But we know for a fact that, like, ML models are kind of weird in the way that they think, quote unquote, about the world. And they might represent something quite differently to what a human would.

• 40:36Anton Troynikov?

  They might focus on quite different things because they are mechanistically pursuing an objective. They are not trying to encode any meaning. And I think that's what produces surprises. I think one way to think about what you get back with inserting images that are not from stable diffusion is if stable diffusion had produced this. So if it were possible for the model to produce this, or put another way, if you like, if the model had produced this image with a corresponding caption that was appropriate to it, here's what would have been in the training set. It's like, it's a long chain of counterfactuals. It's not really meaningful. So here's, here's the thing, right? In, and you can go and look at this, anyone can. If you go to GitHub and you look at the stable diffusion code, whether version one or version

• 41:22Anton Troynikov?

  two, as provided in the open source repo, there is a watermark function there. That watermark is not present in most hosted versions of stable diffusion, and we don't know why. Or at least if it is present, people don't tell us what the watermark is, and we don't know why. That was our plan originally to sort of reject non-stable diffusion generated images, but it's just not present and we felt that better to release and get feedback than not.

• 41:45Speaker 03

  Yeah, you know, actually, I think there's an interesting, it is an interesting experience when I used my real, you know, taken on my phone images. The results were kind of, you know, I don't want to use this term, but it's the only one that's coming to my mind. I know it's loaded. But it almost did feel like the pieces that were kind of collaged together to make my actual real image. Like, I tried one where it was just my kids saying goodbye to their friends at our front door. And what I got back was not like... All, you know, kids at front doors.

• 42:24Speaker 03

  But you could see these like elements that were very resonant across the images. And I felt like, you know, there's almost something here product wise that is kind of interesting. Like it's a sort of diverse image to image search that like picks up on these kind of notable aspects.

• 42:46Anton Troynikov?

  Have you seen, I think it's like same vibe or something like that. It's like essentially that it's like a mood board search engine that I think uses similar principles under the hood. Um, I think there's a bunch of ways that you could take this. I'm actually like, I'm curious, you know, if we do in keep doing experiences direction, maybe we will right now. Obviously we're focused on other things. I think this is a great chance to turn like generative models into something besides a one way process. Like if you can manage to to do these kinds of relations, you can really, you know, start to use them more as in like a search or an iterative or iterative conversational search type of approach, which I think is is really interesting.

• 43:26Anton Troynikov?

  Um, but yeah, like it, it, it, the, the thing that you're getting back is like, okay, this is how the model sees your image almost. This is like what it would have tried to do if it was generating it.

• 43:37Speaker 03

  It is cool because it it does have this sort of. It it sort of reflects back to you like, these are the things that made your image distinctive in a way that you couldn't necessarily have articulated yourself. But like, oh yeah, I see like my kid in this picture is like turning over his shoulder and looking a certain way. And like this other image is like quite different in all the other respects, but it has that same like pose. So there's some like feature that it that they have like very, you know, very much in common, even though like the other features are are very much not in common. I actually would recommend that people go try their real images on your search. Even though it's not what you necessarily meant it to be.

• 44:19Anton Troynikov?

  Please don't yell at me if you do that.

• 44:21Speaker 03

  Yeah, FYI, it's same dot energy is the.

• 44:26Anton Troynikov?

  Same dot energy, same dot energy. I really like that. I think it's fun.

• 44:29Speaker 03

  Okay, so let's return to the the intended use case. So you put an image that you made with stable diffusion into the product. It does its searching and, you know, principled extrapolation of the search comes back with images.

• 44:47Speaker 03

  Sometimes I see a few, sometimes I see a lot. I was really left wondering, like, how are you deciding how many to include? Is there like a certain cutoff? Or are you kind of doing like a top P type of thing? What do those distributions look like?

• 45:02Anton Troynikov?

  Yeah, that's a really interesting question. It's something we've tried to instrument, but it's again, one of those things that you need a lot of data to figure out and a lot of compute, which we are not applying to this right now. But it's interesting. You're right. Sometimes you get not many matches and sometimes you get quite a lot. And I think that that depends on where you land in this like modified similarity space that we talked about. Like you mentioned earlier with the language model, where it can like generate reasonable text from places that are nowhere in its training set. You can think of this almost as like a dial maybe from the model reproducing stuff to the model generalizing. Right? If you, if you can kind of maybe look at that from that lens.

• 45:45Anton Troynikov?

  And the thing is, is the stable attribution algorithm really doesn't take model generalization into account. We're trying, we're like leaning really hard on this like reproducibility rather than than generalizability. And there's a lot of questions to explore there. And I think that they will get explored because we're going to look for ways to continuously improve the performance of the models outside of the scaling laws. Although a lot of, you know, the labs who have the most compute, they will keep pushing on the scaling laws. The labs who are looking to just, you know, do something interesting or even even at the product level, people are looking to do something interesting. They will experiment with stuff like this and be like, oh, okay. I noticed that like this part of the generative space doesn't have much in it, but people are asking for a lot from here.

• 46:28Anton Troynikov?

  Maybe we should like go out and get stuff that that people want from this part of space. And then like get people to actually make images for it. It's an interesting possibility for the future.

• 46:38Speaker 03

  So when I do get those results, is it, is it fair to say, I think you said this on the, on the backy show, that the, you know, everything influences everything in this space, right? Cause like everything's involved in gradient descent at some point. So in theory, if you had the kind of fully principled. Answer, they would all be, you know, infinitesimal, but probably non-zero.

• 47:07Anton Troynikov?

  Yeah, it's an interesting question. I'm not sure that that's quite right. I don't think that they would necessarily be infinitesimal. I think that different generations will draw from different parts to greater or lesser degrees because of the training objective. Further, things that are further away should naturally not influence the generation from a particular conditioning caption as much as things that are, you know, related to that conditioning caption. I think that like, I think that's also a pretty interesting question because one open question is, in what way does generalization actually influence the output? Like, we can't, we haven't looked at that. I think that you could do this kind of research with a toy model.

• 47:50Anton Troynikov?

  And really see like, oh, you know, just have a, just have a tiny diffusion model, make, make pictures of flowers or something. And actually just, just literally just burn a bunch of compute on figuring out like, okay, this is like proportionally how the data set actually does influence things. I think there's another really interesting approach here, which you can take with a toy model, which is basically like compare the stable attribution approach with like the gold standard and see how good or bad we actually are. If we had, if we had GPU time, we would just do that. And then I'd be like, okay, yeah, everybody was right. This sucks. Or I could be like, oh, actually, no, this is really great. And I have numbers now. But until then, it's, you know, these questions are pretty hard to answer.

• 48:26Speaker 03

  So in the processing running today, does it generate a score for each image that it finds? Like they're ranked, right? Does that have a clear relationship to like the whole? Like do those sum to one?

• 48:43Speaker 03

  Or do they sum to anything that you know what they sum to?

• 48:46Anton Troynikov?

  I mean, it's, it's, there's, there's some metric in space. It's, we treat it more like an ordinal thing. And then you sort of, you know, you can get a proportion out of that by you say, you take the top end and then you divide evenly according to, uh, according to the similarity metric, but it's not that meaningful. It's much more of an ordinal question. You can say, okay, these, these are like closer, these are further apart.

• 49:08Speaker 03

  So then what is happening when there are no results? Like you also have a threshold below which you just are like, nothing is close enough to this?

• 49:15Anton Troynikov?

  There's a few filters going on behind the scenes. We have a threshold where we just stop looking. You know, it's pointless to get things that are far away. We say that, okay, really only this can be regarded as influential. It's an arbitrary choice. It's heuristic. And again, that's like, okay, well, here the model is really like extrapolating and filling the gaps.

• 49:34Speaker 03

  So I'll, we'll leave aside for the moment the like, why, you know, at least the deep dive into the why behind you did this, behind why you did this and the, you know, the vision for, you know, sort of an economic dimension to this attribution that could, you know, one day come to exist. Again, you talked about that in the recent show, so I don't want to rehash it. But I wonder like, to what degree this is separable? Like I was just running this thought experiment where I'm like, okay, let's say I have a stable diffusion image and, you know, Greg Ruckauski is in my prompt and it's amazing. Like you go on these sites and you look at the prompts and it still feels like one out of every three has like Greg Ruckauski, you know, by name.

• 50:14Anton Troynikov?

  I think it got stuck in people's heads as well early on. It's amazing how like the very early things that people discovered, they just like become gold standards, even though it could have been anyone, it could have been any choice. There's plenty of concept artists working today. It's kind of, it's, it's just one of these odd artifacts of internet culture.

• 50:29Speaker 03

  I guess I'm wondering like, if you had something, if you were able to kind of peel back, right? And they're doing this now with stable diffusion, like they, you know, they've got this opt out mechanism and all that kind of stuff. To what degree is everything sort of meshed together and not really separable? So like a thought experiment would be, let's say you took all the stock photos and then you added all Greg Ruckauski's real art. Do you have any intuition for whether I could still get Greg Ruckauski stuff out of that? Or do I need like a richer, denser space?

• 51:03Anton Troynikov?

  Yeah, look, this is, this is exactly the kind of research that needs to get done. This is exactly the kind of research that hasn't been done to this point because we've all, we've been working towards capabilities of these models. And we need to start looking at like, okay, like controllability, engineering principles. The way that I think about this is in the early days of aviation, we had like intuition about what makes airplanes fly. Right? We had, you know, the first powered flight. The Wright brothers were wrong about how the right flyer work. They just had intuitions about it. It flew. That's an object demonstration. It doesn't really matter if your intuition is wrong if the thing works. And then there's this Cambrian explosion of different types of aircraft. People were putting two, three wings on them.

• 51:44Anton Troynikov?

  People were, you know, experimenting with different materials, different fuselage shapes. You got things that look like birds, things that look like crazy fans in a tube. Because we all we had to go on was intuition. We didn't have engineering principles yet, and we didn't even really have the tooling to develop those. That's where we're at today, in my opinion, with machine learning. We have a whole bunch of intuitions. We kind of know how things work. And what's really interesting, and this happens very frequently, is like some paper will assert. And, you know, even with stable diffusion, even even with like plane diffusion models, they were like asserting, oh, this is like Lagavine thermo diffusion dynamics. And the paper came out the other day. It's I think it's called cold fusion or something like that, or cold diffusion. Where they're like, no, it's not. Here's here's an object demonstration that that's not true.

• 52:28Anton Troynikov?

  We're in that era of just experimentation. None of this stuff is like really well understood. And we're seeing the early days of people trying to develop principles. The scaling laws were the first thing. Where it's like, okay, well, we're going to empirically demonstrate that you need to increase data as much as you increase compute, otherwise it's worthless. That'll develop further until we probably get to a point where we can actually design the things that we're building for the mission that we're intended to perform like we design aircraft today. Because we've, you know, and we still don't know everything about flight. Flight is a very complex discipline. But we now have the tools and we have enough experimental sort of evidence and we have enough, you know, laws of physics that we can really like build pretty amazing aircraft. Uh, but it took a long time.

• 53:09Speaker 03

  Yeah, for what it's worth, uh, the, it sounds like you might have read it, but the David McCullough Wright Brothers biography is pretty short and absolutely fantastic. I could not recommend that. Would you send me that? It's very striking just how, you know, these two brothers cared about basically nothing but doing the work. There was. And their family, but they really did not. There were a lot of folks out there who were like hyping themselves up as the great experts in flight. And they never actually flew anything despite their, you know, supposed expertise.

• 53:48Speaker 03

  And these two dudes just, they put their heads down and they just did the work until they actually had the goods.

• 53:55Anton Troynikov?

  Yeah. And then they, then they milked it, then they milked it through intellectual property lore just as far as they could. Until Curtis bought them. Uh, it's, it's kind of a, it's kind of a Greek tragedy, honestly.

• 54:06Speaker 03

  It's interesting. That, that's not as much covered in the book. So maybe I'm missing, uh, some of the less flattering aspects of their story. And if you have any, any take on the kind of technology angle, you know, I was thinking also as I was doing this, you go through all these prompts and you just see, you know, 4K unreal engine, you know, Pixar style, you know, red brand cameras. And, you know, envisioning your kind of imagined future of like people getting, you know, some sort of economic remuneration for their contribution to this. Do you think that like those sorts of technology layers also have a claim like?

• 54:49Anton Troynikov?

  It's interesting. I think that's interesting. That's a really interesting point.

• 54:54Anton Troynikov?

  So first of all, I, like for me, as somebody who works in this discipline, prompt engineering kind of feels like, and I've said this elsewhere, it kind of feels like trying to pick a lock with a wet noodle. Like there's a behavior that you, the model you have that the, that you want the model to do. But you're, the only interface you're using to the model is like literally typing text into it. When what you really want to be doing is like have feedback mechanisms inside it, like, you know, the control systems, like we have in industrial machinery or in aircraft. So like, what actually is adding 4K doing? Open question. Nobody's like, it hasn't been answered. This is more of that kind of research that I'm talking about that needs to be done. Like what I, like, why does 4K matter? It's kind of like with, with instruct GPT, why does telling it that it's the world's expert in something?

• 55:38Anton Troynikov?

  Make it produce better answers. What's actually happening? And the danger is anthropomorphizing these models. It's like, oh, it's thinking better. It's, it's not a person, it's a machine. Dissect the machine, see why it's doing that. Where is it going in its latent space? All these models are, are encoding and decoding the latent space. To answer your next question, like, you know, how do you attribute rights holders to equipment? I mean, you know, it's, that's fairly settled. You don't. Um, although, you know, could be interesting. Personally, I'm not a huge fan of adding additional copyright law or additional like strengthening derivative work IP law. One of the reasons that we did this project was we didn't want Disney to own everything forever in the inevitable copyright backlash. Um, because you can imagine like, okay, derivative work copyright law

• 56:21Anton Troynikov?

  gets strengthened and then the next day Disney says, well, anything vaguely superhero looking is in the Marvel universe. So I'm sorry, you can't draw pictures of that anymore and put them online. So that's one of the reasons we actually released this is just, you know, it's actually aligned incentives here. I hadn't thought about that angle. I think traditionally sort of the medium is, is elided. Uh, I don't think that people really have claim to images based on the equipment used to capture them. Could be different here. I don't know. I don't, I can't say that I have an opinion.

• 56:52Speaker 03

  Yeah, I don't really either. And it definitely, you know, I would agree. It seems quite settled when it comes to, you know, the photographers own their work and they, you know, they bought the camera and that's kind of the end of that transaction. But this does feel like it could be different, you know, the, certainly just the number of times you see that as you go through the prompts does kind of suggest like, gee, there is something to that that is, you know, highly desirable that they've kind of created that, you know, people have a hard time getting without them. So I wouldn't be surprised if the, uh, you know, the red camera makers attached to, uh, something at some point.

• 57:29Anton Troynikov?

  Look, like, I mean, I think maybe it's something similar to how photographers will share, you know, their equipment. They'll say, this is how I captured this photo, right? They still captured the photo, but there's some credit to the equipment they used so that other photographers can learn from them. I don't know, like, could be.

• 57:47Speaker 03

  Yeah, it's a lot of free advertising for them right now as well, so they they might not want to opt out.

• 57:52Anton Troynikov?

  Prompts are like this first, the like zeroth stage of of like attribution. Like, if you're typing Greg Gregor's Kowski, you should probably like credit the guy somehow.

• 58:01Speaker 03

  The the role, the importance, you know, the the problems associated with noise, I think. Been one of the concepts I've been thinking about the most since I listened to you talk to Packy. And, you know, you said that, for one thing, these massive web skill data sets, like. You have a lot of image text pairs where there is high similarity. And that basically means like the text describes what is in the image. And then you have a lot where, you know, it's kind of like. A totally different thing that may have no almost nothing to do with what is actually in. The image and that just serves to add a ton of noise to the to the data set. And then you're kind of extending that to say like that noisy.

• 58:46Speaker 03

  Portion of the data set is helping it generalize. I don't have a great intuition for that, so I'd love to understand your thinking there a little better.

• 58:54Anton Troynikov?

  Yeah, so like in, let's say, old school machine learning, even though it's been around since twenty fifteen. Pre in in the prehistoric times and the times of our forefathers. One thing that people noticed very early on was that neural networks were very prone to overfitting. And the way to prevent overfitting was to inject noise into the model's training loop. So what you would do is like dropout was a very common thing where you just like disconnect connections between neurons while it was training. So that it didn't like end up relying on any specific connection. Too much or, you know, there are all these all these little tips and tricks and techniques which kind of got obliterated by by just scaling and just make just make the model bigger. Uh, feed it more data. But at the time, this this was like a huge issue, right?

• 59:37Anton Troynikov?

  And so the way to maintain generality so that the model, like, once you evaluate it on your test set, still performed well, was to inject noise. Like, you, you know, you'd fuzz the images, you would like flip them around, try to do like plausible things that might exist in the real world. And so that's the analogy that I draw here. Noise seems to lead to generalization in some way. It seems to allow it to, when the text and the image are not very related, it seems to, it seems to grab something conceptually here from those images. It's like, it, it, it almost picks like supporting structures out of them without necessarily attaching the associated meaning. That's just an intuition that I have.

• 1:00:21Anton Troynikov?

  I can't say for sure that it's real. Um, another way to look at it is like, you can imagine one image, many captions. It allows the model to come to the same image for many conditions. Right? Imagine, imagine you only actually had one image, but you just trained the model on one image with many, many different captions. It would always end up in that image regardless of what, what like you put into it.

• 1:00:45Anton Troynikov?

  Maybe that's also helping like weight the kinds of generations that you get like it suddenly, it suddenly is able to connect one image concept with many textual concepts. Things like that might, might be helping it. Again, this is exactly the kind of research that needs to be getting done. Well, hopefully I want it to get done. I don't know if other people do.

• 1:01:04Speaker 03

  To target rich environment right now, that's for sure. If, uh, if we go, you know, an hour and I am able to, uh, that's almost like my KPI for this show is how many times I can get my, our guests to say that that research needs to be done. I think we're at like three, so I'm doing all right today. So just briefly going back to your, your product with Chroma and the vector database. And you had said at the top, you know, there's all these sorts of, all these different kinds of things can be represented as vectors. So we've largely talked about text, which is going to drive this like retrieval future where, you know, the AI is going to have seemingly like photographic recall of all of our emails and our, you know, interactions and so on.

• 1:01:45Anton Troynikov?

  Even better than that, put, put that information together in a way that actually answers the, the query that you have, answers the question that you have, or like fits the context that you want it to operate in.

• 1:01:54Speaker 03

  So what else is is getting embedded these days?

• 1:01:59Anton Troynikov?

  Oh, boy. I mean, you guys have probably obviously heard of Whisper before. You have a podcast, so yes. Whisper is audio embeddings. Any anything, basically, any modality that you can stick in your own network onto, and preferably a transformer at this point, onto, you can get embeddings from it. So we've got audio already. We've got video. I've been speaking to a couple of people who have been working on that and it's actually starting to get commercialized too, like video embeddings. So you can do like semantic search inside a video. You can be like, find me all the, find me all the like footage from this party that's actually fun to watch and it'll just do that for you. So video, you know, and I think it'll just, I think it will keep going. I think that we'll see it over in robotics, for example, like chains of actions are also embeddable, like, oh, the person like perform this task. That's, that's, that can, that's embeddable.

• 1:02:44Anton Troynikov?

  Lots more like that. Basically, all kinds of modalities will continue to get embedded. And I think what will continue to happen as well is, so we've talked about clip, contrastive, language, image, pre-training. We'll see more models trained that way. So right now we have like Whisper, which is text to audio in the same space. And we've got Clip, which is text and image in the same space. And we'll have video and video and text in the same space. But then you can also think of like audio to video. You can think of images to audio. That whole matrix of things and each one of those entries in that matrix is a potential application of one kind or another, which is a really exciting thing.

• 1:03:20Speaker 03

  Are you seeing things also like from biology starting to get embedded? I was thinking genomes and protein structures?

• 1:03:26Anton Troynikov?

  Yeah, biotech, I think, is waking up to this. Biotech is a very interesting field to think about this in because in biotech, you kind of have to get it right. It's okay if your semantic search returns things that only kind of look like the thing that you wanted. In bio, you really need the protein to be the one you were looking for. But the other part of this is in bio, most of those embedding functions that we talked about at the top of the of the recording are still hand-rolled. Very few are actually, you know, trained and modeled. And I think We've seen the success of this in these other domains that we've talked about, and I think biology is starting to wake up to this whole possibility.

• 1:04:07Anton Troynikov?

  And so I think in the next year or two, we're going to start seeing a lot more work in this direction. And I've been talking to sort of biotech founders and companies and people working in this space, and they're all very interested to see what can be done here.

• 1:04:20Speaker 03

  You know, I've been using the phrase Kurzweil's Revenge increasingly often over the last year or so. And that's probably a good bridge to kind of the last topic for today. Are we all gonna die or what's gonna happen?

• 1:04:35Anton Troynikov?

  Yeah, that's a good bridge. That's a really good bridge because you brought up Kurzweil. And one of my favorite things about Kurzweil is I went on the website not long ago. And it was like the 10th anniversary of the singularity is near and that just, I don't know, that made me laugh. So, look, I don't think we're gonna die. I have yet to see a convincing argument that we're all gonna die. Every argument that I've seen in that direction requires some sort of system of basically incalculable and definitionally like incomprehensible power, which, okay, like if you're gonna invoke magic, you might as well just do that at the start. I think that there are dangers. I definitely think that there are dangers. I don't think the dangers look like

• 1:05:18Anton Troynikov?

  AI super smart overlords pursuing human unfathomable goals. If you're concerned with that, you should probably work on sun alignment because the sun is giant and incomprehensible goals and is very dangerous to humanity, but it actually exists today. What I'm more worried about is, and these are like cycles in history and you see them repeatedly. We see them with the inventing of the printing press and then radio and then television significantly destabilizing to society. And often either like very

• 1:05:49Anton Troynikov?

  Very violent period. So the Thirty Years' War is the one I always come back to. And the Thirty Years' War, in part, was triggered by the printing press. You know, Martin Luther was able to spread his ideas very quickly throughout Europe because they could be printed. And that caused, you know, players in power to like be able to attach to this movement and they produce dischism. And then parts of Europe were fifty percent depopulated in a matter of a few years, which is incredible. It's still, per capita, the Thirty Years' War is still the worst conflict in Europe, in European history, per capita. You know, and then you have the rise of totalitarianism, which was, you know, you put the radio, you put a radio in everybody's home. Suddenly, a person who wants to be a dictator doesn't have to walk physically from city to city or just convince you through text. You could hear it, or raiders.

• 1:06:32Anton Troynikov?

  And I think it takes time for societies to develop, basically like what I think of as memetic antibodies. Like, I think, I think if someone, I think if someone heard like Hitler ranting on the radio today, they'd be like, damn, this guy's crazy. Although, you know, we do have that. There's a few people who are still pretty persuasive. But it takes time. It takes time. And there's, there's also this bizarre thing that occurred in my lifetime, which, which still deeply confuses me.

• 1:06:57Anton Troynikov?

  Back when the web first came out, everybody kind of knew not to trust it. Because people were like, oh, anyone can put anything on there. Like, don't trust anything you read on the web. Like, people can just lie. And that doesn't, that has never changed. Like, you can just still go on the internet and lie. And there's like that famous meme macro. But people seem to have regressed. People like seem to believe what they read online more readily now. And it's almost because it got legitimized as an information distribution mechanism that people like lost that immunity. And the reason that I bring this up in the context of AI is because, like, you can imagine very concrete use cases for even something like ChatGPT, where you can have like finely targeted down to the individual propaganda.

• 1:07:41Anton Troynikov?

  We've seen the success of like social media in doing this, like ad targeting, being able to be really mobilizing for like elections in the United States and in other countries. I think at the point where you can like create text that's individually targeted and kind of like start hacking people's actual preferences to get them to vote for you or to like incite violence or any number of things, that's, that's kind of dangerous. But it's not the AI, it's the people running the AI. There's another interesting thing, uh, which I've been speaking to a few people about, and I'll, I'll sort of mention in the abstract, which is like, there are dangerous things in the world which are still gate-kept because the knowledge of how to do those dangerous things is very difficult to come by. But if you have this like reasoning machine in

• 1:08:25Anton Troynikov?

  your pocket, you don't have to be smart anymore. You can ask the smart reasoning machine that you have to do the dangerous thing. And so that's another, like, that's another real possibility. I called it a knowledge capability, um, somewhere. Uh, and I've, I've been getting, uh, I've been trying to get ChatGPT to tell me how to make a neutron initiator, which is a classified component of a hydrogen bomb. They won't do it. Um, these things never let me get my work done. These are things, these are things that are more worrying. And then, these are just the things we can see. I think the destabilizing effects of new media technologies are always unexpected as well. Uh, and I think we're in a very early stage, so we need to kind of be vigilant about this stuff. Um, I think that's, that's kind of the perspective. I don't think we're gonna die, but, but I think, you know, it might be a

• 1:09:09Anton Troynikov?

  dangerous and, and in the, uh, in the sort of the cursed sense of interesting times.

• 1:09:15Speaker 03

  Yeah, that seems to me, that's kind of why we called this show The Cognitive Revolution. That seems to me kind of baked in at this point. You know, the, from my survey of the AI landscape, you know, we've talked from a couple of different angles about how kind of the same architectures are working for everything. All this progress is happening in parallel. And, you know, we're talking to folks like yourself who are chipping away at kind of, you know, and I don't think, by the way, I don't think anybody really thinks right now that the core technologies, like the fundamentals, are gonna stop right where we are, you know, or that there's gonna be no new insights or no, you know, architectural improvements. I expect that too. But even if I kind of rule that out, I think, jeez, the core stuff

• 1:09:58Speaker 03

  right now seems powerful enough to be transformative. It hasn't been productized, it hasn't been, you know, sanded, the rough edges are still there, they haven't been sanded down. It hasn't been, you know, connected to all the other things, really, new Bing, you know, I'm still on the waitlist, for God's sake.

• 1:10:17Anton Troynikov?

  You missed the fun part.

• 1:10:19Speaker 03

  I know. Well, yeah, for better or worse. Uh, but. These things are all gonna happen, right? And so the impact, to me, seems like pretty baked in. I think we're in for a wild ride that we're really not ready for.

• 1:10:30Anton Troynikov?

  I don't think there's ever been a time in history where someone's been like, okay, guys, get ready, we're gonna press this button and everything's gonna change. But I think that actually speaks to our resilience as a species. It's that that's still literally true. Every single time that's happened, we've come through it. And, you know, some people take the perspective that this is a unique risk. I think that the actual risks that exist rhyme pretty strongly with other times we faced them. So I, you know, I put that not in general as like a, as like a statement like, oh, we'll be fine. We've always been fine before. It's like, no, I think we'll be fine because the actual risk we're facing rhyme with risks we have actually faced before. The way, the way what you put right now, I think, is actually a good point. And people working in like broadly speaking, the realm of AI and AI safety.

• 1:11:14Anton Troynikov?

  The sort of the consensus is, is even if we stopped training and developing new models today, we would keep finding new uses for the models we have today. Like, if you see how people experiment with GPT three and three point five, like discovering that you could ask it to think step by step makes the answers better. It didn't require training in your model. You just literally like, oh, this, this thing can do that. They call it a capabilities overhang, if you're in this sort of space, safety space. So, yeah, we could literally stop training right now and still not know everything we can do already.

• 1:11:44Speaker 03

  Yeah, there's a lot of capabilities overhang is vast. I'm really uncertain whether or not we're all gonna die. But I definitely don't rule it out. And I'd like to hear your kind of response to, you know, I'll just give kind of a superficial, like, high level version of it. But I would say, you know, if there's like a canonical argument at this point, I would say it's probably Ajay Catra's post, which is called, "Without Specific Countermeasures, The Most Likely Path to AGI."

• 1:12:21Speaker 03

  Likely ends in disaster.

• 1:12:24Anton Troynikov?

  Could you also link me to that post after this chat? I genuinely read a lot of this literature. I read L E Izer's thing, A I Doom. I like, I've read, I think people are always surprised by how much less wrong I've read and how much alignment forum I've read for, given what my position actually is. I'd be curious to read that, but you know, what, what is the argument that is advanced there?

• 1:12:47Speaker 03

  Well, there's a lot, so you, you know, a lot of them, they can be subtle, but this, the way that I like to summarize it for folks is, you know, what she describes as the most likely path to A G I at this point, she calls human feedback on diverse tasks. So kind of a generalization of the reinforcement learning from human feedback to more and more different kinds of tasks and, you know, likely multimodal, you know, types of tasks. The challenge that she sees that does seem to me really hard to dismiss, not to say that I'm like sure that it plays out this way, but I really can't see a great reason why I should be confident that it won't, is just simply that we are not reliable raiders.

• 1:13:32Speaker 03

  You know, the people doing the evaluation have these flaws and, you know, we know from kind of the heuristics and biases literature, you know, the cognitive.

• 1:13:42Anton Troynikov?

  We know that people are flawed. In fact, there's a, Richard Noe from OpenAI has this great paper called Alignment From A Deep Learning Perspective, and he brings up a lot of these great examples, actually. He brings up examples where like the model actually fools the human raider, uh, cause it looks like it's performing a task on camera, but it's actually just faking it. So yeah, like, okay, so far, so good.

• 1:14:02Speaker 03

  At some point, it seems like, you know, these models are getting pretty smart already, right? Like they have increasingly, they're, it's like now in debate whether or not they have, you know, effective theory of mind, so on and so forth. It seems like it's not too much of a stretch to get to a point where the models will maximize their feedback score by not being straightforwardly, you know, truthfully, honestly doing the task, but instead start to develop a model of, well, here's what reality is, but here's what this person is likely to give me a score for. And so then you kind of create this incentive for the thing to begin to deceive. And if you create that in an environment where we don't have the interpretability.

• 1:14:48Speaker 03

  Chops right now, obviously to detect if it's there, certainly with any reliability, then it seems to me like you really are playing with real fire.

• 1:14:58Anton Troynikov?

  There's a leap of logic to be made here. So we have a model. It's performing a task that's valuable to humans. We're rating its performance on the task. First of all, if it's performing the task that we asked it for, and we're rating it on that task, what is the difference between it doing it deceptively or correctly if we can measure the outcome, right? And I don't mean in the sense that, oh, human raiders, et cetera. I just mean in the sense that like, it's either doing the thing we want it to do or not. And so in the example, in the like deceptive examples from that paper that I mentioned, well, yeah, okay, it wasn't really raising the ball, but also there was nothing riding on it raising the ball or not, except our benchmarking of this model. That's the first part of it. In engineering, the real task is, does the machine perform the task?

• 1:15:41Anton Troynikov?

  Instead of just in a test environment, but is it actually doing what it's supposed to be doing, right? And this is where I come from. In the thing that people call, seem to call alignment in AI research is called control theory everywhere else. And these things are dynamical systems that can be understood. They have actually some really nice characteristics that allow us to understand them as dynamical systems. And we know how to build controllable dynamical systems. It's why the F-16 can fly at all is because we know how to do this. The other leap of logic that's being made here is like, okay, well, it's deceptive to human evaluators. It can hack their reward function. How does it follow from there that it's that we're all going to die? What are like, what is the path that actually happening? I've never, I've never heard a clear example of this

• 1:16:21Anton Troynikov?

  that doesn't invoke some sort of other form of calamity that if you're worried about, humans could just do that. You don't need to worry about the AI. You could just worry about the calamity itself. I get this thing where like, oh, you know, it develops an undetectable nerve toxin that triggers once it's implanted in every human being on Earth. Well, you know, okay, so why not a human with a machine that's optimizing in the same way produce the same outcome? Or, or, you know, this thing that it builds like these nano self assemblers. And there's this famous post like, create an identical strawberry thing where it like, you know, does these nano assembler things. It's like, okay, but like, first of all, first of all, as far as I can tell, nano assemblers are science fiction. A lot of people like Drexler, I haven't actually seen a demonstration of this technology.

• 1:17:06Anton Troynikov?

  But second of all, like, okay, like, you don't need the AI to be worried about the nano assemblers at that point. Uh, so that it's like not clear to me that an AI that, okay, is like reward hacked some humans is any more dangerous than an out of control bulldozer. Like, we know, we know how to deal with this stuff. We ought to think about it in those terms. I think the real danger here is anthropomorphizing the system and saying, no, it's like it's thinking. It has goals. It doesn't have any goals. It's a, it's a computer system. It can, it can be anthropomorphized to have goals, but concretely it's doing matrix multiplications and applying nonlinearities to them. And that's a system, it's a, that's a nonlinear, dynamical system that's continuous and differentiable. And we know how to deal with those.

• 1:17:51Anton Troynikov?

  It just doesn't strike me as a credible argument. It only strikes me as a worrying argument when you start to turn these machines into people.

• 1:17:58Speaker 03

  Well, I worry about the whole spectrum and I get, you know,

• 1:18:03Speaker 03

  ambivalent on this in the classical sense of having strong feelings both ways. Like I'm, I love working with all the AI technology. I love studying it. Uh, I'm super excited about the upside. I do, you know, when you talk about like, what about just like a bad actor with an AI? That's very much, I think, of concern, uh, no doubt. There's a project out there right now called, um, I can't exactly what it's called, but it's basically text to regulatory RNA sequence, which starts to sound pretty insane. Um, I asked a model one time to write some hard science fiction based on that premise. And it got me to some pretty strange places pretty quick. Obviously, we have so much time, we can only go so deep on this today.

• 1:18:49Anton Troynikov?

  There's one, there's one canonical example of that, right? There was the, there was that famous paper where they're like, they asked it to produce effective pharmaceuticals. And then they turned the reward function around and just started producing deadly poisons almost immediately. Like, that's, that stuff is more concerning. Um, ultimately, in order to kill us all, whatever does that has to obey a physical law.

• 1:19:10Anton Troynikov?

  It's a large leap for me in the physical world, especially, and I think this comes from my background as a roboticist, in that it's really hard to do things. It's a lot easier to do things with information than it is to do anything in the physical world. Regardless of how often people analogize things like, oh, DNA is just like binary encoding, or it's like CRISPR is like programming biology. It's not really like that in the real world at all. To give another analogy, right? Like, anyone can make sarin gas. The problem with sarin gas is not making it, it's deploying it. It's getting it into places. That's why we don't have constant sarin gas attacks. Like, whatever this thing is that's supposedly gonna kill all of us, has to be able to act in the physical world in a way that I just, like, haven't seen any system capable of doing yet.

• 1:19:55Speaker 03

  Well, I hope I never meet a robot that has the regard for me that New Bing has had for some of its early users. That I can say for sure. So, we're running out of time. I would love to check back in, you know, at some point down the line and see how both your company is developing, whether or not you've continued this stable attribution project, and we can also check in again on how, how much existential confidence or nervousness we have. But a couple just real quick hitters to close us out today. One, AI tools in your life. What are you using? What has changed your workflow? What do you recommend?

• 1:20:36Anton Troynikov?

  Yeah, ChatGPT has sped up my programming work a whole bunch because whenever I work with an unfamiliar library or get a weird bug, I ask ChatGPT. Sometimes it hallucinates stuff, but at least helps me look in the right direction. I use like coding assistants. I've been using Copilot. I recently tried Codemium. It's pretty good. Definitely sped me up as an engineer, mostly because like I no longer have to like Google or go to Stack Overflow or look for documentation of stuff that I'm unfamiliar with. And what that means is it's not replacing me as a programmer. It's getting the stuff, it's getting the things that I really don't know how to do out of my way, so that I can focus on the things that I'm actually really good at. That's been a huge, huge productivity boost. Yeah, I think that's really the main ones that I'm using today.

• 1:21:19Speaker 03

  Also recommend Replit and their Ghostwriter and Ghostwriter Chat.

• 1:21:24Anton Troynikov?

  Yeah, yeah. We really like Replit at Chroma.

• 1:21:28Speaker 03

  Okay, quick hitter number two. Hypothetical scenario. If a million people already had a Neuralink implant.

• 1:21:40Speaker 03

  And it would allow you to type as quickly as you can think. In other words, you now have thought to text. Would you get one?

• 1:21:50Anton Troynikov?

  I think, and I've thought about this, I think if we end up living in a society where somebody makes me implant a computer in my head, or it's like socially unacceptable to not implant a computer in my head, I'm going into the woods.

• 1:22:02Speaker 03

  That's a no. You're a no on.

• 1:22:03Anton Troynikov?

  It's a, it's a strong no. Listen, I've seen, I've seen how software is written. I don't want it anywhere near my brain.

• 1:22:09Speaker 03

  Yeah, it's wild. I do think, I kind of expect to see that day. You know, I, I used to think my grandparents, who were born around 1930 and, you know, my grandmother's still alive, she's going to be 90 this year. I used to think that they saw way more change in their lifetimes than we ever would. And that perspective has started to change recently. And now I'm like, yeah, maybe I, uh, better be getting emotionally prepared to have a hole in my head and a little implant.

• 1:22:36Anton Troynikov?

  Look, I know what's going to happen here. What's going to happen here is I'm going to have some neurodegenerative disease when I'm like 85 years old. And they'll be like, install Neuralink and it just fixes you. And I'll be like, god damn it. Remember when I went on that podcast?

• 1:22:48Speaker 03

  You, you'll remember it after they do the implant. That's when you'll, uh, that's when you'll have the memory.

• 1:22:53Anton Troynikov?

  That'll be the real tragedy, I think. You're right.

• 1:22:55Speaker 03

  All right, last one, you know, you're not so concerned about the, the doom scenarios. Well, give us your big picture hopes, positive hopes for, and also to the degree that you have them, downside fears for AI for the rest of this decade.

• 1:23:12Anton Troynikov?

  Yeah, I think that's a really good way of framing some things. So one of the problems that we faced historically as a species is the thing that we talked about earlier, where nobody tells us, like, we're going to push a button and everything's going to change. But I think AI gives us the possibility of having tools that allow us to rapidly adapt to our situation. And kind of the fundamental thing that humans have going for us is our adaptability. So a prosthesis, an extra tool that allows us to adapt even faster. Makes it easier for us to deal with problems as they emerge. It actually could make us safer in the long run. And the way that I think about that is like, right now, for example, to suppose, suppose that there's like some frightening thing like, like COVID, right?

• 1:23:53Anton Troynikov?

  In COVID, you needed a PhD or an MD to understand like what to do with it, right? And there's only so many people in the world that really understand like what recombinant DNA really means. I certainly don't. I'm not a biology person. But if we had tools to like, get people to the research front, like a PhD in a box where it was like exactly tailored and individualized to you and turn you into a researcher just because you wanted to work on that stuff. That sounds like we could be much more agile and nimble as a species to deal with anything that might come up. That's like, that's a hope. The, of course, the downside is everyone kind of, every new media technology has this other property where at the start, everyone believes it's going to be for education, but it ends up getting used for pornography and propaganda.

• 1:24:31Anton Troynikov?

  And I think this time is no different. And the propaganda part worries me and the sort of this new idea of like knowledge capability through having a reasoning machine in your pocket. Kind of worries me a bit. The thing that doesn't worry me is like, I don't think that like. You, you, you could probably reason your way to some very dangerous things, but again, those very dangerous things require you to be able to act in the physical world. So it's, we're talking about like. Fairly organized actors being dangerous rather than individuals? Hard to say. But yeah, like overall, I think. Also, I think one thing that's going to happen, I think one other dangerous thing which we didn't really discuss is there's a very real possibility that another AI winter will happen. And I, I actually asked Sam Altman that question a little while ago. There was some sort of event.

• 1:25:12Anton Troynikov?

  Honestly, over exuberance about the capabilities and then they fail to deliver. That's what's happened every other time in AI winters, right? Like, you know, the previous one was like expert systems and one before that was neural networks. Now we're back in neural networks again. That's kind of worrying, um, because another, another fallow decade when we could have these incredible technologies is actually like. I don't know. That, that seems really bad to me. We should, we should try to like. Be fairly clear eyed. That said, uh, I'm still bullish because there's, like we said, there's like these massive capabilities overhang. We haven't even started to explore, so.

• 1:25:45Speaker 03

  That's all the time we have for today. Anton Chernikov, the company is Chroma, the embeddings database. Thank you so much for joining us on the cognitive revolution. Thanks for having me, guys.

• 1:25:56Speaker 02

  The cognitive revolution podcast is supported by Omnikie. Omnikie is an omni-channel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms with the click of a button. Omnikie combines generative AI and real-time advertising data to generate personalized experiences at scale.