The Four Wars of the AI Stack (Dec 2023 Audio Recap)

Note for Latent Space Community members: we have now soft-launched meetups in Singapore https://lu.ma/aie-sg, as well as two new virtual paper club/meetups for AI in Action and LLM Paper Club https://lu.ma/ls. We’re also running Latent Space: Final Frontiers https://lu.ma/latent-space-final-frontiers, our second annual demo day hackathon from last year https://twitter.com/FanaHOVA/status/1627142249988657153. Edit from March 2024: We did a followup on the Four Wars on the AI Breakdown https://www.listennotes.com/podcasts/the-ai-breakdown/the-four-wars-of-the-ai-Fhtlb4M2yzI/. For the first time, we are doing an audio version of monthly AI Engineering recap that we publish on Latent Space! This month it’s “The Four Wars of the AI Stack”; you can find the full recap with all the show notes here: https://latent.space/p/dec-2023 https://www.latent.space/p/dec-2023 * [00:00:00] Intro * [00:01:42] The Four Wars of the AI stack: Data quality, GPU rich vs poor, Multimodality, and Rag/Ops war * [00:03:17] Selection process for the four wars and notable mentions * [00:06:58] The end of low background tokens and the impact on data engineering * [00:08:36] The Quality Data Wars (UGC, licensing, synthetic data, and more) * [00:14:51] Synthetic Data * [00:17:49] The GPU Rich/Poors War * [00:18:21] Anyscale benchmark drama * [00:22:00] The math behind Mixtral inference costs * [00:28:48] Transformer alternatives and why they matter * [00:34:40] The Multimodality Wars * [00:38:10] Multiverse vs Metaverse * [00:45:00] The RAG/Ops Wars * [00:50:00] Will frameworks expand up, or will cloud providers expand down? * [00:54:32] Syntax to Semantics * [00:56:41] Outer Loop vs Inner Loop * [00:59:54] Highlight of the month This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, partner and CTR resident, and that's the both partners. And today I'm joined just by my co-host, swyx, for a new podcast format.

• 0:17Speaker 01

  Yeah, it's a bit uncomfortable because we have to just stare into each other's eyes lovingly. But in our end of year survey last year, a lot of listeners were asking us for more one-on-one time, more opinions from the both of us as hosts on what's going on in AI. You know, both of us are very actively involved. And I don't think this year will be any different. This year, there's lots more excitement to come. And we're trying to grow Late in Space in terms of the types of formats and the amount of value that we deliver to our subscribers. So one thing that we've been trying, experimenting with is this monthly recap that I started doing around August of last year, where I basically just take the notable news items of the month and then I sort them and categorize them according to some order that makes sense

• 1:01Speaker 01

  and write them down in the newsletter. And this last December recap was particularly exciting because it seemed like it popped off in a number of areas, particularly with the AI breakdown. Our friend NLW featured it on his podcast. And I figured we can just kind of go over that as a way of setting the stage for 2024, but also recapping what happens in 2023.

• 1:23Alessio Fanelli?

  Yeah, and people always ask me if December is like a slow month. But I think you almost broke Substack with how many links we had in the thing.

• 1:31Speaker 01

  No, we actually did. So a lot of people commented to me about the formatting issues within the newsletter that I sent out. And I know that they are there, but I couldn't fix it because Substack was broken by us with how long it was.

• 1:41Alessio Fanelli?

  Oh man, but we had this kind of like four main buckets called the four words of the AI stack. Data quality and I guess like data quantity as well in a way. The GPU rich versus poor, which we have a whole episode about with Dylan Patel. Multimodality, we're actually recording tomorrow with LumaLabs about their new 3D model. So we went from text to image to 3D video. I wonder what's next.

• 2:06Speaker 01

  And we're going to release Hugging Face as well, as I guess. Thinking about calling it multimodality one oh one. Because the first modality beyond text that you should really pay attention to is vision.

• 2:16Alessio Fanelli?

  Right, yeah, and then the rag ops war, I think that's a.

• 2:20Speaker 01

  I don't know what to call it. I don't know what, if you were to call it anything else, this is my.

• 2:24Alessio Fanelli?

  Or I don't know, but I think beginning of last year, that was like kind of the hottest space because there wasn't much open source model work. And I think over the last maybe like four or five months, everybody's so focused on fine tuning Lama two and like a DPO to improve these models. Max trial and all these things and people forgot about our friends at Langchain, Lama Index and some of the things that were maybe top of mind. Vector DBs, you know, it seemed like everybody was releasing a vector DB early in the year.

• 2:52Speaker 01

  Yeah, I think that I I'll be very surprised if any new vector DBs come out this year. With one exception, which is something I'm keeping my eye on, which is turbo buffer. I don't know if you've seen them going around. Yeah, all the smart people seem to be adopting turbo buffer as the first serverless vector DB.

• 3:07Alessio Fanelli?

  Yeah, no, and we're gonna have definitely Jeff and Anton on the podcast at some point. I know they're gonna be, they're gonna be fun.

• 3:16Speaker 01

  I should also mention, the reason I selected these four wars was a process of elimination of wars that I think didn't ended up not mattering. So for those who don't know, inside of my writing, I often include footnotes that are in themselves, just essays in footnotes. And so I think it's also notable, the things that people thought were hot, that were less hot than expected. So it was agents, definitely less hot than at the start of twenty twenty three. And then this one is very controversial, non-selection by me. I think open source AI is not a battle in the sense that I don't think there's anyone against open source AI. Everyone is like on one side. There's no like opposing side apart from regulators. But in my mind, when I think about like for engineers, engineers are all universally in favor of open source models.

• 4:01Speaker 01

  So there's there's no battle here. Everyone just wants it to improve. So it's not like interesting to write about. We just want more open source.

• 4:07Alessio Fanelli?

  Yeah, the only battle is people offering inference on it.

• 4:11Speaker 01

  Yes.

• 4:12Alessio Fanelli?

  Killing each other in the process.

• 4:13Speaker 01

  Yes, I classified that as a GPU rich versus poor war. But maybe there's a better way to classify that and you can give me some feedback. On that, it's a struggle to try to categorize the world. Code models as well. I was very struck by a conversation I had with Poolside, I so Kant from Poolside. So they haven't been on the podcast yet. They're kind of stealth still, but they had a very, very notable fundraise. I think they had like fifty million dollars raised.

• 4:35Alessio Fanelli?

  I think even more, yeah.

• 4:36Speaker 01

  For a seed, spending most of it on GPUs. And my conversation with I so he was like, hey, you know, like, Replit was like one of our podcast's early biggest winners. Replit didn't really follow up with, like, they announced like their one point five model, but it's not really why they use beyond Replit, you know? There's Star Coder, there is Code Llama, but like, it's not really for how important code is. It doesn't seem like as big of a battlefront as just general function calling, reasoning, you know, these other kinds of domains. And so I thought it was just interesting to note that even though we as a podcast try to pay particular attention to developer tooling, to code models.

• 5:15Speaker 01

  We interviewed Cursor, Find, Replit, Codium and Hugging Face. These all seem like very small compared to the amount of money being thrown, the amount of heat in the other domains. And I don't know why that is.

• 5:26Alessio Fanelli?

  Yeah, I think it's maybe the fragmentation of the tooling, you know, like most people in code are using VS Code, Cursor, GitHub, one of the three. So there's maybe not as much experimentation versus with text, people are just trying everything. It's hard to try a code model, you know, I see code models being released, but like, it's not super easy to just plug it into your workflow. So I think engineers like myself are just lazy and like, hey, I'm having great success with whatever I'm using.

• 5:53Speaker 01

  Yeah.

• 5:53Alessio Fanelli?

  I don't really wanna wanna go there.

• 5:55Speaker 01

  Special case form of code is SQL and the semantic layer data engineering type things. We also had to guess on there from Seek and Cube, and we also talked to a bit of Databricks, a bit of Julius.

• 6:06Alessio Fanelli?

  Yeah, and we have Brian from.

• 6:07Speaker 01

  And Brian and Brian from Hex. Does he count? I don't know.

• 6:11Alessio Fanelli?

  Yeah, I know.

• 6:11Speaker 01

  Yeah, yeah, yeah. I guess the Hex notebooks, yes. Hex magic, yes. Rexis is a different beast. Anyway, but yeah, I think people who like come to AI engineering for the AI might actually end up finding themselves in data engineering in the end, in like traditional ML engineering in the end. They might have to discover that they're, you know, doing Rexis and all the stuff that gets swept under the rug in a demo becomes their job. And I'll probably say like, just because we didn't select a theme for last year, doesn't mean it wasn't important. It just wasn't top of mind yet. And maybe I think that will be an emerging theme this year.

• 6:46Alessio Fanelli?

  Yeah, I think that's kind of the consequence of the low background tokens, like the end of the low background tokens once.

• 6:53Speaker 01

  Can you explain what you think low background tokens? This is our November recap.

• 6:56Alessio Fanelli?

  Yeah, the comparison that our friend Jeff Huber at Chroma brought up is steel before the atomic bomb creation. So steel before no radiation in it. After all the testing, a lot of steel had radiation embedded in it. So it was really precious to get low background steel, meaning with no radiation. And same with tokens, you can assume that any internet content from three years ago, it's just internet. It's like people writing, it's not models writing. Instead now, anything we're gonna get on Common Crawl, updates and things like that, you never know. If it's human written or not. And I think that will put more work on data engineering, right? Because even basic stuff like checking if a text says, as a model created by OpenAI, you know, it's gonna be important.

• 7:40Alessio Fanelli?

  So people had just been blindly taking all the data sets offered by Luther and Common Crawl and all these different things, assuming that all the data in it is good. I think now, how do you build on top of it? And we've seen the New York Times lawsuit against OpenAI. We've seen data partnerships starting to rise in different companies. I think that's gonna be one of the bigger challenges and maybe we'll see more of the work that Databricks has done to build the Dolly 5K instruction tuning. Just first party creation of data. It's like you got people sitting at their desk every day. If everybody wrote five, you know, Q and A pairs or things like that, you would have a massive, unique dataset for your model. So.

• 8:18Speaker 01

  Yeah, for people who missed that episode, that was one of our early episodes as well. And Mike Conover since left and to start Brightwave.

• 8:25Alessio Fanelli?

  Yeah.

• 8:25Speaker 01

  Which I'm sure we'll have him back this year.

• 8:27Alessio Fanelli?

  Yeah, they're doing a lot of interesting stuff. I think the next episode will be very cool.

• 8:31Speaker 01

  So how do you want to tackle this? Do you want to just kind of go through the four wars?

• 8:34Alessio Fanelli?

  Yeah, let's do it. You created this like Wikipedia-like infographic for each of them.

• 8:40Speaker 01

  Yeah, I should say like the inspiration for this actually was during the Sam Altman leadership battle. People were making mock Wikipedia entries for the debate and for like who's on the side of the, you know, D cells and who was on the side of the EX. So I like that format because. It's very concise, it has to list the key players and it's kind of fun to think about like who's on what side. And think about what is important and what people are battling over. And I think it is important to focus on key battlegrounds as a as a concept. Because there's so many interesting things you could be talking about in AI and they're not all equally interesting. So how do you decide what is interesting? I think it's money, it's power, it's people, it's, you know, like impact, that kind of stuff.

• 9:24Speaker 01

  And so, yeah, that's what I ended up doing. And fun fact, the way I did this was I actually edited the HTML on Wikipedia and then I just screenshotted it just to get the formatting.

• 9:33Alessio Fanelli?

  Good old developer tools. Developer tools is all you need.

• 9:38Alessio Fanelli?

  So the data war, belligerence.

• 9:40Speaker 01

  Yeah.

• 9:41Alessio Fanelli?

  On one side you have journalists, writers, artists. On the other side you have researchers, startups, synthetic data researchers. I guess like, maybe we want to talk about what are the axis of war? So like one of them is attribution, right? Like, I I think there's a varying spectrum of how comfortable people are about this data going into a model. So some people are happy to have your model trained on it, no matter what. Some people are happy to have your model trained on as long as you disclose that it's in the model. Some people just hate that you trained on their model. And some people, like the New York Times, want you to destroy any artifact that might have touched your article. So that's kind of what we're fighting on.

• 10:23Alessio Fanelli?

  I just want to make it clear that it's not just like you should never use the data or you should always use the data. I think people are just trying to figure out what's the right form of attribution and how do I get paid? As somebody whose data ended up being in this training. I think we're giving everybody a lot of great tokens with latent space because we do full transcripts on on everything. And we're happy for people to train models.

• 10:45Speaker 01

  Oh, yeah, yeah. Please train a latent space model.

• 10:46Alessio Fanelli?

  Yeah, we would love it. So that's kind of what we're fighting on. Anything that people should keep in mind about this war and like maybe some of the campaigns that are going on?

• 10:56Speaker 01

  I think the New York Times one is probably going to go to Supreme Court. It is very, very critical. It is landmark war that will probably decide what fair use means in context of. And I recommend, I think The Verge did a good analysis of this. Platformer maybe did a good analysis of this. There are like four criteria for what fair use is and everyone basically converges on to the last criteria, which is, does your transformative use of my copyrighted material diminish the market from my content? It's very hard to say. I would suspect that yes, in some capacity, in some amount, but good luck proving that in a court of law. And I think a negative ruling on OpenAI would seriously stall the progress of AI.

• 11:38Speaker 01

  And that's bad for humanity, but good for content creators and writers. So obviously, which we want them to be adequately compensated and recognized for their work. There's like no easy outcome here, apart from the existing copyright system, which is also somewhat broken. And it's just a very, very tricky, challenging case, I think.

• 11:56Alessio Fanelli?

  It's funny because we had something, I was a community moderator at a website called Rap Genius, which was a lyric sanitation. And there was like a similar thing in maybe like 2014, where like the music labels basically came to the website and it's like, hey, this is not fair use, you know, like you cannot reuse the lyrics to the song. And eventually the website made deals with the record labels to like be able to do this. And then Google was stealing the transcripts to put in like the enhanced thing.

• 12:23Speaker 01

  And they proved it.

• 12:24Alessio Fanelli?

  Yeah, yeah, we did all the like, basically like the things on the I, some I's we put the dots, some I we put like the accent and that's how it.

• 12:32Speaker 01

  Oh, I thought it was, I thought they just varied the spacing or they like use a different kind of spacing in the Unicode.

• 12:38Alessio Fanelli?

  I think it was the I thing, but maybe, I mean, this is like almost 10 years.

• 12:42Speaker 01

  So Rap Genius proved it by injecting some data poison into their corpus. And then Google reproduced it faithfully. And so therefore they proved that Google was scraping Rap Genius. Did Google have to pay Rap Genius money in the end?

• 12:54Alessio Fanelli?

  I don't think so. There was also another issue with Rap Genius that we had that it got blacklisted by Google for like, there was like a lot going on. But anyway, this is not a Rap Genius special.

• 13:04Speaker 01

  Yeah, I mean, ultimately, like, I think that we do need quality data. I think that if this case is contained to the New York Times, the New York Times' worst outcome is that they will substitute it with Washington Post. And they substitute with the Economist or like the second or third ranked newspaper that is the most friendly to AI. And then the New York Times will realize that actually their words are not as not that much more valuable than other words. Then the the value of the content comes down very, very dramatically. I think it will be interesting, but uh yeah, I I do I do think it's overstepping their bounds to call for the destruction of all GPTs. Yeah. That's probably for sure. Then the bigger problem I have is with Stack Overflow and Reddit, which I named as the side of the New York Times. They have effectively shut down their APIs in order to try to train their own models.

• 13:48Speaker 01

  Probably same as Twitter, actually, I should probably have put Twitter, I put Twitter on the wrong side, maybe, I don't know, Twitter is on both sides. Elon is on every side, the side of chaos. Yeah, what what this is, is basically every UGC, users generated content company of the two thousand and twenty teens. Now has a giant pile of user content that becomes valuable data. That used to be open for researchers to scrape and train models. Now all of them are locking their walls, right? Behind their wall gardens and then trying to train their own models to boost their benefits. So this is a locally optimal outcome for them, but a globally suboptimal outcome for humanity. Because why should we care about the closed garden of Reddit?

• 14:27Speaker 01

  You know, the Reddit model, the Stack Overflow model, the X model, as opposed to like it being a part of a data mix of twenty percent Reddit, twenty percent Stack Overflow, twenty percent X. That seems like a much better outcome for the world, but everyone is acting in their very narrow self-interest in trying to make their own model, which is probably going to suck. Right, so next war, after you get data. We we should mention synthetic data. Oh, yeah. Yeah, so what happens when you run out of human data? You make your own.

• 14:56Speaker 01

  So I would say like when I went to New York, that was the number one discussion out of every single researcher's mouth. There is a lot of research coming from both, I guess, the big labs as well as the academic labs on like what good synthetic data looks like. I don't know if you've like talked to any startups around that, I just talked to Luis Costicato. Yeah. And he is promising a very, very interesting approach to synthetic data generation. I think his phrase for it is like pre-trained scale synthetic data. As opposed to what the news research and the other open source communities have been doing, which is fine tune scale synthetic data. And so he wants to create like trillion token data sets that are all synthetic. And I'm like, okay, that's interesting, but also at the same time, these are all just downloads from GPT four or something else.

• 15:40Speaker 01

  Lewis is very aware of that and he's he has a way around it. I don't really understand it, but he claims that that's a good way around it. Andre Karpathy at NERIPs highlighted this paper from DeepMind where they were bootstrapping synthetic data that could be verifiably proven correct. So specifically in math and in code, where there is a correct answer. So yeah, that makes sense. You know, you can you can solve the synthetic data problem that way. But like, what about, you know, beyond that? There's just no answer.

• 16:08Alessio Fanelli?

  And wasn't part of the issue also that the way that the phrases are constructed and like all of that and synthetic data ends up kind of like making mode collapse even worse? Because one thing is like right or wrong, right? The other thing is like every sample is read in the same way, you know, or like as a similar, since it comes from a certain model, kind of as a similar route of structure.

• 16:29Speaker 01

  Yeah, so I mentioned this in the best papers discussion with John Franco. So the the basic argument is you already have a flaw distribution from a language model. You are resampling that flaw distribution to double down on that flaw distribution. Like, there's no extra information from humans. So on principle, how can this work? And so the only conclusion there is you don't need it to emulate a human. You need it to emulate a useful assistant, however you define it. So I think the goal of synthetic data is less to emulate human speech because that is basically solved. It is now more to spike the distribution in useful ways. And that's a phrase I borrowed from Kanjou from Imbue. But anyway, so I think that synthetic data will be a giant theme for this year and not least because the human data is being locked up behind walls.

• 17:13Speaker 01

  So like, it's a very, very clear trend. This is probably the most amount of money after GPUs will be spent here. So one war I did not put here was the talent war, right? Like the war for PhDs and smart people. And but when you break down what the talent people do, one is they make models and, you know, they run in friends on GPUs or they run training runs on GPUs. But the other is they clean data, they find data, clean data and format data. And like, so yeah, these are all just proxies for like the kind of talent that is flowing back and forth. And ultimately, I think you have to focus on what they're working on, like the the visible output of what they're working on, which is data.

• 17:46Alessio Fanelli?

  Awesome. Alright, let's talk about the GPU inference war. I think this is one that has been heating up. And we actually have a bunch of these folks coming on the podcast in the next few days.

• 17:57Speaker 01

  Yeah, yeah. Are we calling it Compute Month?

• 17:59Alessio Fanelli?

  Yeah, we can figure out a name, but we have Model, Together, Replicate. There's a lot coming up. But basically, the mixed role released, the M O E model was kind of the spark of the war. I think the price went down like 90% in one week.

• 18:14Speaker 01

  Yeah, I wrote 222 times, but yeah, the one divided by 222 is whatever the...

• 18:18Alessio Fanelli?

  Yeah, yeah, yeah. And then there was like the benchmark drama between Together and AnyScale. On like whether or not, which one was faster and like whether or not the benchmark was really reflective of performance.

• 18:30Speaker 01

  Yeah. This is very surprisingly ugly in a way that I think usually people try to respect each other's work and play nice and say nice things when people release stuff. Even if it's a competitor, you say nice things or you don't say anything at all. AnyScale, for some reason, they released a benchmark that on which, of course, AnyScale looks the best. Why would you release a benchmark where you don't look the best? But then, basically, everyone featured in that benchmark didn't like it, of course. I do think there's some like methodological thing. So for anyone doing benchmarks, you have to understand that there's a real, real, real difference between like a public benchmark that is meant for just limited testing compared to, okay, if you're load testing us or if you're seeing what a real enterprise customer would see, you have to give them a heads up.

• 19:14Speaker 01

  You have to get a different API key, a different endpoint, and you test the real infrastructure, not the demo one. This is very common for infra companies, and I think AnyScale just neglected that. And it hurt their credibility, like AnyScale is not new at this game, like they should have done that. But what was interesting was this benchmark drama reached even beyond AnyScale. And we're gonna have Sumit on, and he's gonna talk about like why he weighed in. Because Sumit doesn't represent any inference part, he just works at Meta. But he felt like this was a, this is a very interesting debate. And I think we'll see more of this. You have been a data investor for a while, like data-based companies always do this. And I think now we're just seeing this kind of fight come into the inference space.

• 19:53Alessio Fanelli?

  Yeah. I think the hardest thing is the end customer cannot replicate it. You know, so like if you give me like a Postgres benchmark, I can run Postgres on my MacBook, you know, and run similar ones. I think with models, it's just impossible. So people tell you... This is the benchmark and you're like, okay, I have to go sign up to every single cloud now to try it. It's just not easy. And we talked about this in benchmarks one-on-one, which is same with model benchmarks, right? Just like, oh, this model is so much better than this. And then it's like, did you train on the questions? And it's like, what? Uh, oh, I don't know. Uh, so, and again, it's hard for people to just like, run the models and test them, you know? So like, there's a lot more weight, I think, in AI on benchmarks than there is in traditional software.

• 20:36Alessio Fanelli?

  Because nobody buys upstaged over Redis Cloud or whatever, just based on a benchmark. They try them and check performance and whatnot, because they have real production scale workloads. Here, it's like, nobody's really doing anything with these models. So it's like, whatever any skill says, I guess is good. But then customers are gonna go try it and just decide for them what the right thing is.

• 20:57Speaker 01

  Yeah, and I think it's important to understand it is not just about cost. I think what the price war represented was a race to the bottom on cost. And you're like, okay, Deep Infra, which is a company, the name of the company is Deep Infra. Deep Infra has promised to just always be the lowest cost provider. Like, okay, fine, that's a, that's a good value proposition, but you're not only optimizing for that in a production application, right? You're optimizing for latency, that's one thing. You're optimizing for uptime, that's something that you can only earn over time. You're optimizing for throughput and like, other forms of reliability. It starts to tail off beyond that. But there's three or four dimensions that really, really matter, right? Like, if, if you're not table stakes on any of those things, you're out. You're just out. Actually, there was a really good website that was released just this week called Artificial Analysis. Did you see it?

• 21:41Alessio Fanelli?

  Yep.

• 21:41Speaker 01

  Yeah, so this is what the industry needs, which is an independent third-party benchmark, pinging the production API endpoints of all the providers and giving a third-party analysis of, of what this is. I actually built a prototype of this last year.

• 21:55Alessio Fanelli?

  Yeah, I was gonna say.

• 21:56Speaker 01

  But I didn't like maintaining it.

• 21:59Speaker 01

  I'm glad someone else is doing it just because I don't want to keep up with all these things. But still, like, I think it's a public service that somebody should do. And so I'm glad that they did it. I think they did it very well. So yeah, I think that is where the, I guess, the inference drama is ending for now. I don't think, you know, I haven't seen any continuing debate there. The only other thing that, you know, I did some extra work on this for the recap, which is like, are they losing money? You know, are they pricing correctly their tokens from Mixtral? And I actually managed to go into Dylan Patel's write-up of the Mixtral price war. And I think I reasonably worked out that you can serve Mixtral, and the lowest you can possibly charge if you like, take the most aggressive

• 22:43Speaker 01

  amortization of all your CapEx and all that, is fifty to seventy-five cents per million tokens, which is what perplexity prices their Mixtral at. And perplexity is a very smart player. They're not even an inference provider. They're just like doing this for fun. But they're like, we don't want to lose money on this. We will provide it at cost. This is what cost is to us. So that means perplexity provides it at fifty-six cents per million output tokens. That means any scale, which is fifty cents, OctoAI, fifty cents, Abacus AI, thirty cents, and deep infra, twenty-seven cents, they're all losing money. Because we think that the break even is fifty-one cents.

• 23:18Alessio Fanelli?

  And even that is like a full batch size and

• 23:22Speaker 01

  No, no, no. I assume fifty percent utilization. So like, very, like, you talk to practitioners, very, very good is sixty percent. Average is like thirty, forty. So I just, I say fifty, right? You assume fifty percent, batch size sixteen, hundred tokens per second generation, that's also very, very high. These are all very favorable numbers. Like, probably the real number is closer to seventy-five cents per million than fifty cents per million. Anyway, anyone charging under fifty, definitely losing money. So then it's like, okay, either you don't know what you're doing, which in which case, good luck, or you know what you're doing, and you're purposely losing money for something. And what is that? And I don't know, but I think it's an interesting aggressive strategy to pursue if you are doing it on purpose.

• 24:05Speaker 01

  So this is something that like the classical like Walmart would have a lost leader, like they they they really, really on purpose lose money on things. So that they get you in the door to try things out. Like, I don't know if that makes sense to you as a as a VC.

• 24:17Alessio Fanelli?

  Yeah, it's like the, well, it's like all the, you know, the candies are placed at the cash register. Because maybe you just went to get the thing on discount and then you buy a Kit Kat, whatever, and they make money on the Kit Kat. They all have the Pokemon trading cards at checkout now. So if you bring your kids or buy the discounted whatever for you, then you end up spending more. But to me, the thing is like, what where's the checkout register where you upsell people with these things, right?

• 24:43Speaker 01

  Yeah, I don't know how you also.

• 24:44Alessio Fanelli?

  That's that's really the big thing. I don't know. I'm curious to see. I don't think a Klaflair still has a life. I wonder what they're gonna charge.

• 24:51Speaker 01

  Mixtral.

• 24:52Alessio Fanelli?

  Our workers. Yeah.

• 24:53Speaker 01

  They cannot serve Mixtral. Their GPUs are too underpowered. Klaflair AI is like very good marketing for very, very underpowered inference, right?

• 25:03Alessio Fanelli?

  Yeah, well, well, I don't know. I think it all depends on like what is gonna be needed, right? So that Mixtral 7B right now, I I check.

• 25:11Speaker 01

  But they can also serve Mixtral.

• 25:13Alessio Fanelli?

  Yeah, yeah, yeah. I wonder, but I think they don't want to get into this race right now, probably.

• 25:18Speaker 01

  No.

• 25:19Alessio Fanelli?

  So, yeah, I'm curious, going back to the lost leading, it's like, is there gonna be a better model that comes next? That they hope that you already integrated their thing with, you know, if you if you're using Together to serve Mixtral? And then something else comes in that you're gonna replace Mixtral with? Hopefully, you're still gonna use Together and they're gonna get better unit economics on it. I don't know. It's a good question.

• 25:43Speaker 01

  It's a good question.

• 25:44Alessio Fanelli?

  Thank you, VCs, for, you know, paying for all of our inputs.

• 25:48Speaker 01

  No, no, no. I I like, I think these are, you know, everyone in here are grown adults. They're they're smart investors. I'm sure there's some kind of long-term strategy here, and I'm I'm trying to figure that out, like, assume that people are smart. And then what what will smart people do?

• 26:00Alessio Fanelli?

  Yeah, I I think it's the same with Uber, right? It's like, how could it have been so cheaper at the start? You know, like you look back at all DoorDash, all these things, it's like.

• 26:10Speaker 01

  And like last year was a great year for Uber.

• 26:12Alessio Fanelli?

  Yeah, no.

• 26:12Speaker 01

  All my Uber friends are like suddenly very, very rich again.

• 26:17Speaker 01

  One thing I will mention on like the engineering sort of technical detail side is the rise of mixture of experts is something that we covered in our podcast with George and now with Mixtral. And it represents the first successful, really, really commercially successful sparse model. And sparse in a very interesting way, in a sense that the divergence between the amount of compute you need at training versus the amount of compute you need for inference continues to diverge. But also in a weird way where you need to keep all the weights of the MOE model loaded, even though you're not necessarily using them at all times. So basically, what I think that is, is like, I think that that is going to impose different

• 27:00Speaker 01

  needs on hardware, different needs on workload, different needs on like batching optimization. Like Fireworks recently announced Fire Attention where they wrote a custom CUDA kernel for Mixtral on H100. It's like super, super domain specific. And they announced that they could, for example, quantize from like 16 bit down to 8 bit with like no loss in performance. All these magical details emerge when you take advantage of like very, very custom optimizations like that. I think like the rise in MOEs this year is going to be, going to have very meaningful impacts on the inference market and how it's going to shape how we think and price for inference. It may not be that we have this sort of input token versus output token paradigm for, for long, particularly because we have things like different forms of batching, different forms of caching.

• 27:44Speaker 01

  And like, I don't really know what that looks like, but I'm very, I'm very curious. I see a lot of opportunity here if I was an inference provider player. Like that's something I would be trying to offer to people as, as a way to differentiate because otherwise you're just an API.

• 27:56Alessio Fanelli?

  Yeah, yeah, no, it was in a way counterintuitive because most of the struggles with inference as well are just like memory bandwidth, you know? So we have now models the scale worse, a higher batch, you know? But I'm glad I'm not in that business, I can tell you that. As far as, there's so much work to be done and like so many low levels of the stack. You're already trying to provide value to the customer on like the developer experience and all of that, but you also have to get so close to the bare metal to like make this model. Like writing a kernel, imagine if you had to write, you're like a CPU cloud provider and you have to like write instruction sets. It's like just, nobody would get in that business, you know?

• 28:37Alessio Fanelli?

  So I salute all of our friends at compute providers doing this work. I mean, together it's doing so much for like three dial and like fresh attention too and one on so.

• 28:46Speaker 01

  Yeah, so and that's something that I would I would leave as the last part of this sort of war of GPU rich versus poor. The GPU rich people are the model trainers and the infra providers. They're they're say like, we have the GPUs, comb use our GPUs, you know, and then we we provide you the best inference, right? And that's that's what we've been discussing so far. On the other side, on the GPU poor side, are like all the alternative methods, right? The modulators, the tiny corps, the Q Loras, and and all the other type of stuff. I even put consistency models in there because, you know, any efficiency or distillation method where you you reduce your inference or GPU usage by like 25 to 40 times, it's a GPU poor friendly approach.

• 29:27Speaker 01

  So I would also put Apple and MLX in there, and that's also like, Apple is finally making moves in inference, and that will be a game changer for local models, because then you just don't need any cloud inference at all. You just run it on device, which is fantastic. And then obviously, RWKV and Mamba and Strikty and Taina from together, like all those like emerging models, I don't know, there's something I've been worried about for a latent space. How much attention should we give to the emerging architectures? Because there's a very good chance that, one, these things don't work out, two, they take a very long time to work out, and then three, once they work out, they're like for limited domains and like not super usable. So I don't know if you have opinions on that. I can follow up with one conclusion that I've had, but I want to

• 30:10Alessio Fanelli?

  Yeah, no, I want to hear.

• 30:11Speaker 01

  Throw that question open to you. And so the one conclusion is RWKV and the state space models, including Mamba, have historically just been pitched as super long context models. And I'm like, that's not something I need, because I'm okay with 100K context. I'm okay with RAG and recursive summarization, all those, those techniques to extend your context, like rope and yarn and all these, all these things. So like, why do I need million context models? Why do I Need ten million, one hundred million, one billion models like whoa, why? So the easiest argument is, oh, you can consume very, very high bit rate things like video and DNA strands and then you can do like syn bio and all that's good stuff. And I'm like, okay, I don't know anything about that.

• 30:54Speaker 01

  Like what happens if like you hallucinate one wrong chain in your, you know, the DNA strand that you're trying to synthesize? Good luck.

• 31:03Speaker 01

  I don't know. That's why I've been historically underweighting intentionally. Our coverage of state space models and the non transformer alternatives until Mamba. Mamba really changed things where basically for the same amount of compute, you can get a lot more mileage or a lot more performance for the same size of model. Now it's an efficiency story. Now it's a GPU poor story. It is no longer a long context story. It is just straight up, we are strictly more efficient than transformers. I'm like, oh, okay, I can get that. Does that change anything? I don't know.

• 31:33Alessio Fanelli?

  No, that makes sense. I think people look at the slope, right? Which is like, oh, you're going to get the context higher and higher. But in reality, it's like if you kept the context smaller, instead look at the anti-slope, so to speak. It's like same context is like a lot less compute.

• 31:46Speaker 01

  Yeah. So that was not clear to me until Mamba. And so I think that's interesting. There's a concept that being trying to call the sour lesson. You know, the bitter lesson is stop trying to do domain specific adjustments. Just scale things up and it's going to work. That's general intelligence. General intelligence dislikes any attempt to imbue with inside of it special intelligence. Like if you have like any if switch case or if statements or like if finance do this, if something do that, don't bother, just just scale things up and it's going to do all of them simultaneously all better at once. That's the bitter lesson. The sour lesson is a parallel, is a corollary, which is stop trying to model artificial intelligence like human intelligence. Right? The neuron was inspired by the brain, but doesn't work exactly like the brain.

• 32:30Speaker 01

  Machine learning uses backpropagation. The brain does not use backpropagation. We keep trying to create alternatives to transformers that are that look like R N Ns. Because we think that humans act like R N Ns. We have a hidden state. And then we process new data and we update that state. But maybe artificial intelligence or machine intelligence doesn't work like that. Maybe we just fail every time we try.

• 32:53Speaker 01

  So that's the sour lesson. Every time we try to model things. And my favorite analogy, I actually got this from, I think, an old quote from Sam Altman, who was like, you know, like, we made the plane, the airplane. It was inspired by birds, but it doesn't work anything like birds. Right? And it works very efficiently. Like, it's probably the safest mode of transportation that we have. And it works nothing like a bird. So why should artificial intelligence work like human intelligence? And that is the philosophical debate underlying my continued cautiousness around state space models. I feel very vulnerable saying this because I don't think there's any justification. Once you look at the empirical results or like the mathematical justifications for these things.

• 33:35Speaker 01

  But there is some grounding in philosophy that you should have when you think about, does an idea make sense? Is it worth exploring?

• 33:42Alessio Fanelli?

  Yeah, I think now there's a lot of work being put into it, right? And I think transformers have shown enough success that people are interested in finding the next thing. You know, so before it wasn't clear if transformers were really gonna work. So people are kind of working on them. But yeah. Okay, maybe in the 2025 recap, we're gonna have.

• 34:04Speaker 01

  Yeah, I mean, we're trying to do one before that. So we actually have a link. I don't know if you know this. Shreya Rajpawal from Guardrails, she's married to Karan from Hazy, yeah. And so now he's started one of the other state space model companies. I forget the name of it, so as you'll see, I'm sure like this will be an emerging topic this year as well. So we don't have to wait till next year.

• 34:22Alessio Fanelli?

  Yeah, no, I think we're gonna have maybe the sour lesson, you know, overview.

• 34:27Speaker 01

  Well, I mentioned this in the Luther Discord and then they were like, okay, so what is the spicy lesson? And what is the salty lesson?

• 34:34Alessio Fanelli?

  I want the sweet lesson. Sounds better. Cool. Talking about GPU port, let's do multimodality. Or I feel that stable diffusion was like the first GPU poor model.

• 34:46Speaker 01

  Yes, absolutely. I should, I don't know if I mentioned that. I just didn't mention it. Stability, I think in 2023, you know, they shipped incremental things. I think, I don't know if stable diffusion two was out there. But everyone's talking about XDXL turbo, which is a form, which is an alternative to consistency model, but looks like a consistency model. They ship video diffusion. They shipped a whole bunch of stuff, but just wasn't as big as 2022 when they, you know, made a huge impact with stable diffusion.

• 35:10Alessio Fanelli?

  Yeah, I mean, it's hard to, it's hard to.

• 35:12Speaker 01

  It's hard to top that.

• 35:14Alessio Fanelli?

  But yeah, Midjourney has been doing great, obviously. I actually finally signed up for a paid account last month. So

• 35:20Speaker 01

  Midjourney, yeah, yeah.

• 35:21Alessio Fanelli?

  I'm part of the 200 million dollars a year that they're getting.

• 35:24Speaker 01

  What's confirmed is, I think, like a Businessweek article or Economist or Information article, that this team has now reached at least 200 million ARR, completely bootstrapped. I think their employee count is somewhere between like 15 and 30 people. I don't know if you know exact numbers. I have heard rumors that their revenue is actually higher than that, that was what was reported. But it's between the 200 billion to 300 million range, which is crazy.

• 35:48Alessio Fanelli?

  Yeah.

• 35:49Speaker 01

  Especially if it's like primarily B to C.

• 35:52Alessio Fanelli?

  Yeah.

• 35:52Speaker 01

  Which it looks like it is.

• 35:54Alessio Fanelli?

  Yeah, yeah, yeah. It's like B to Fiverr to B. I think there's like a ton of.

• 35:59Speaker 01

  Oh, you think there's a lot of Fiverr?

• 36:00Alessio Fanelli?

  You can see the.

• 36:01Speaker 01

  Midjourney specialists.

• 36:02Alessio Fanelli?

  Yeah, you can like get in Discord and see what people are generating, you know? And you can see a lot of it. It's like product, placement, ads and a lot of stuff like that.

• 36:11Speaker 01

  And DALL-E 3 doesn't seem to have any impact on Midjourney.

• 36:14Alessio Fanelli?

  DALL-E 3 got so much worse after the GPT-4.

• 36:17Speaker 01

  Really?

• 36:17Alessio Fanelli?

  The all-in-one. Well, first of all, before you could generate four images. And then like very good vibes. Now the vibes are like boomer vibes.

• 36:25Speaker 01

  Oh, no.

• 36:25Alessio Fanelli?

  Every time I generate something.

• 36:26Speaker 01

  The images I have here are DALL-E 3.

• 36:28Alessio Fanelli?

  Every, every time I generate something on DALL-E, it looks some like some dusty, old, yeah, like.

• 36:35Speaker 01

  I think it's a skill issue. I think you've.

• 36:37Alessio Fanelli?

  No, but that, that, that was the great thing about DALL-E 3, right? It's like it made the problem better for you.

• 36:43Speaker 01

  Yeah, yeah, yeah.

• 36:44Alessio Fanelli?

  Like before, like literally like when it first came out, I'm like, hey, make a coliseum with like llamas. And it was like this beautiful thing. I feel like now it's not. I don't know. Again, it's a model, right? So it's like, maybe I just got unlucky. I'm in the wrong latent space, you know?

• 36:58Speaker 01

  Yeah, there's, there's a lot of players in this. I don't even think I put like some of the players I were really excited about. Like, you know, the Imogen team split out to be, to create Ideogram, you know, that was a few months ago. And I didn't even put it here because I forgot.

• 37:10Alessio Fanelli?

  It's too much, right? Can't, can't keep track of all of it.

• 37:14Speaker 01

  I would just basically say that I, I do think that I used to, at the end of 2022, start of 2023, I was not as excited about multimodality. Obviously, I'm more excited about it now. I used to think that text to image was more like hobbyist kind of work, but 300 million dollars a year is not hobbyist.

• 37:31Alessio Fanelli?

  Right.

• 37:33Speaker 01

  It is not like, not just like not safe for work because Midjourney doesn't do not safe for work. So it's real. It's a new form of art. It's citizen art. It's uh exciting. It's it's unusual and interesting. And you can't even model this as an investor. You can't even model this on an existing market. Because like, there's just a market of people who would typically not pay for art, and now they pay a little bit for art. Which is digital, not as good as a human, but it's good enough. Are you sure on that?

• 38:01Alessio Fanelli?

  Yeah, I'm surprised I haven't seen a return of digital frames that were very popular during the NFTs boom.

• 38:07Speaker 01

  Yeah, the very, very first latent space post was on the difference between crypto and AI in this respect. So I called this multiverse versus metaverse. Crypto is very much about metaverse. Let us create digital scarcity. And let us create tokens that are worth, that are limited edition, that were something. And then you display it probably in your, in your PFP as your, as your representation of yourself. And what AI represents is multiverse, which is a very positive sum instead of zero sum. Where like, if you like a thing, okay, I'll choose a different seed and I'll make a completely equivalent second thing, and that's mine. And that means very different things for like what value is and where value accrues. I mean, I still cling to that insight, even though I don't know how to make money from it.

• 38:51Speaker 01

  Obviously, Midjourney figured it out. I think Midjourney like made the right approach there. The other one I, I think I'll highlight is Eleven Labs. I think they were another big winner of last year. I don't know, did they announce their fundraise? I think so.

• 39:04Alessio Fanelli?

  Uh, I don't know.

• 39:05Speaker 01

  Rumor is, yeah, rumor is, I, I can say it. You don't have to say it because I, I only heard it from my friends. Rumor is they're now unicorn and they just focus on voice synthesis, which again, did not care about it at the start of twenty twenty three. Now we have used it for parts of latent space. I listen almost every day to an Eleven Labs generated podcast, the Hacker News Daily. I don't know what the room for this to grow is, because I always think like it's, it's so inefficient to talk to an AI, right? The, the, the bit rate of, of a voice created thing is so low. It's only for asynchronous use cases. It's only for hands-free, ice-free use cases. So like, why would you invest in like voice generation? I don't know, but like. It seems like they're making money.

• 39:44Alessio Fanelli?

  Right, yeah, yeah. Yeah, I mean, Sarah, my wife, yeah, she uses it while she drives to talk to ChatGPT. Just like.

• 39:51Speaker 01

  Yeah, so, so ChatGPT uses their own TTS.

• 39:54Alessio Fanelli?

  Yeah, yeah, yeah, yeah.

• 39:55Speaker 01

  Okay.

• 39:55Alessio Fanelli?

  But you can see the modality.

• 39:57Speaker 01

  You should bring Sarah in at some point, but.

• 39:58Alessio Fanelli?

  Customer interview. We're doing a bunch of like home renovation, so maybe she's like driving to Home Depot and it's like, hey, what am I supposed to get to replace the sink, you know, or

• 40:09Speaker 01

  Okay.

• 40:10Alessio Fanelli?

  all these sort of things that maybe were like Google searches before. Now you can easily do ice free, hands free.

• 40:16Speaker 01

  Uh, yeah, a lot of people have told me about that and I just say, when I'm by myself, I always listen to podcasts. So I don't have time for ChatGPT and ChatGPT, you know, probably the number one thing they can do for me is give me like a speed adjustment. So I can listen as well.

• 40:30Alessio Fanelli?

  That's funny.

• 40:31Speaker 01

  Yeah, anyway, so like, uh, I'm curious about your thoughts on like how as an investor, I think this is the weirdest AI battlefront for investing. Cause you don't know the time.

• 40:40Alessio Fanelli?

  It's funny because there was a bunch of companies doing synthetic voices a while ago, and I think the problem, a lot of them got through like good ARR numbers, but the problem was like a repeatability or use case. So people are doing all sort of random stuff, you know, and the problem is not, it's kind of like mid-journey. The problem is not that there's not maybe a market of interest. It's like, how do you build a venture-backed company with like a scalable go-to-market that like can go after a customer segment and like do repeatedly? I think that's been the challenge. I don't know how Eleven Labs is doing it. But you could do so many things with text to voice that is like, how do you sell it? You know, who do you call? Like, that's like the hardest thing, right?

• 41:21Alessio Fanelli?

  If you're raising like a Series A, a Series B, it's like, how are you going to invest this money in sales and marketing to get revenue back? It's kind of like the basic of it and it can be challenging. That's why sometimes investors are like, you're making money and that's great for you, but like,

• 41:35Speaker 01

  How, there's no industry.

• 41:37Alessio Fanelli?

  It's hard to like just tie it together, you know?

• 41:39Speaker 01

  I would be interested in, because I feel like there's a category of companies in the early 2010s that did this. Meaning they offered an API with no idea how you're going to use it. Uh, I'm thinking Twilio. Twilio has a cohort of like sort of API first companies that are all like sort of Twilio inspired. I think there's a category or a time in the market when it makes sense to just offer APIs and just let your customers figure it out and it's actually okay. And then there's some times when it's not okay. And I think the default investor mentality right now is that it's not okay if you don't know what your customer is doing.

• 42:11Alessio Fanelli?

  I think Twilio is a funny example because I think in the middle 2010s Uber was like 15% of Twilio's revenue.

• 42:18Speaker 01

  But like I'm just, I'm talking like move yourself back as to like Twilio seed investor, Twilio series A investor. They had no idea. Uber wasn't even around.

• 42:24Alessio Fanelli?

  But I think the, the thing now is like text to voice is not new, you know? Like that's really the thing. It's like what's new now is that you can generate very good text to then feed into that model.

• 42:35Speaker 01

  Yeah.

• 42:36Alessio Fanelli?

  So that changes why the market is interesting, you know? But if you really think about it, the models today are a little better. They're maybe like 50% better than they were three years ago. But the transformer models under the feet of what to say, they're like a billion times better. A lot of people use it for like automated, you know, customer support, things like that. Before you had like scripts they were reading, now you can have a transformer model converse with the customer. So it makes it a lot more useful in cases, but we'll see how.

• 43:06Speaker 01

  Yeah, we'll see.

• 43:06Alessio Fanelli?

  That changes.

• 43:07Speaker 01

  Okay. The last thing I'll, I'll mention here, why is this a war? Which is OpenAI and Gemini and Google are working on everything models versus each of these individual startups all working on their selected modality. And so this is a question of like, uh, the big tech company is going to actually win because they can transfer learning across multiple domains as opposed to each of these things being point solutions in, in their specific things. The simple answer is obviously everyone will win.

• 43:33Speaker 01

  Because the AI market is so huge, you know, there's a market for the Amazon basics of like everything, you know, one model has everything. And then there's a market for, no, like the basics are not good enough. I'll need a special thing. Do you have an opinion on when does one market win over the other? Or is it just like everything's gonna win?

• 43:49Alessio Fanelli?

  Yeah, the it's interesting. I think like it works when people wouldn't have used the product without the Amazon basics, you know? So like, maybe an example is like a computer vision, you know, like, I mean, we have.

• 44:00Speaker 01

  Yeah, vision is so important now.

• 44:02Alessio Fanelli?

  Yeah, it's like, you know, before people were like, why am I bothering trying out to set up a computer vision pipeline and all of that? Now they can just go on GPT four and put an image and it's like, oh, this is good. I could use this for this. And then they build out something and maybe they don't use GPT four V. They use Roboflow or whatever else. That's kind of how I think about it. It's like, what's the thing that enables people to try it? You know, so in a way, the God model can do everything fairly okay. It's like DALL-E and Mid Journey, you know, all these different things. And maybe like the mixture of inference wars are like another example. It's like, I would have never put something in my app at like two dollars per million tokens, but I did it at 27 cents per million token, you know?

• 44:44Alessio Fanelli?

  And now it's like, oh no, I should really do this. It's a lot better. Yeah. So that's how I think about how the God model kind of helps the smaller people then build more business.

• 44:54Speaker 01

  Yeah, creates a category. Yeah, rag and ops.

• 44:56Alessio Fanelli?

  Yeah, last but not least, where to begin? We had almost all of these few on the podcast, so.

• 45:03Speaker 01

  They're they're honestly the easiest to talk to because they look like DevTools.

• 45:06Alessio Fanelli?

  Hmm.

• 45:07Speaker 01

  And you are a DevTools investor. I worked in DevTools. I I think they're also more mature as businesses. There's more of a playbook that is well understood by the customer, like, yes, I need a new stack here. Maybe not. Okay, so my my biggest problem with putting databases versus frameworks versus ops tooling in the same war is that they're not really a war. They work cohesively together, except when one thing starts to intrude on another thing. And that's why I very consciously put together this sequence, which is databases on the left, frameworks in the middle, ops companies on the right. What's the first product of Langchain, Langsmith, which is an ops thing?

• 45:41Alessio Fanelli?

  Hmm.

• 45:42Speaker 01

  Right? So so now suddenly the framework companies are not so friendly with the ops companies. Cause they are trying to compete with the ops companies.

• 45:48Alessio Fanelli?

  Yeah.

• 45:48Speaker 01

  And what are also he's trying to do? The ops companies are trying to produce SDKs that compete with frameworks. Okay, then what are the database companies trying to do? First of all, they're fighting between each other, right? There's the non-databases all adding vector features. We had some people approach us and we had to say no to them because there's just too many. And then there's the vector databases coming up and getting $235 million to build vector databases. You know, obviously you're an active investor in some of these things, so you cannot say everything. But just on databases alone, one of the biggest debates of 2023, where do you stand on the whole thing?

• 46:20Alessio Fanelli?

  That's the million dollar question. Well, one, in the start, there's kind of like a lot of hype, you know? So like when Langchen came out and Lama Index came out, then people were like, oh, I need a vector database. They search vector database and it's like Chrome, Pinecone, whatever. But then it's like, oh, you can actually just have PG Vector in Postgres. And you already have Postgres. Did you know it could do that? People are like, no, I didn't because nobody really cared. So like, there's not a lot of documentation. Same with MongoDB, Vector, Cassandra, all these things.

• 46:48Speaker 01

  Elasticsearch.

• 46:49Alessio Fanelli?

  You can actually put vectors and man hangs in everything.

• 46:53Speaker 01

  It's a different kind of index.

• 46:54Alessio Fanelli?

  You know? And I think like Jeff and Anton also, what they always talked about even early on is like, this is like a active learning platform. This is not just like a vector database. It's like, what do you do with the vectors? It's like, what's most helpful? It's not where do you store them? So that's kind of the change.

• 47:11Speaker 01

  I think that was old Chrome, by the way. I don't know if that's the new current messaging.

• 47:14Alessio Fanelli?

  Well, but I think I'm just saying like, to them, it's never about this is the best way to put a vector somewhere. It's like, this is the best way to operate on the vectors. And the store is like part of it, but there's like the pipeline to get things out and everything. You have to build out a lot more. So I think 2023 was like, create the data store. I think 2024 is going to be like, how do I make the data store useful? Because the vector storage just commoditized. So there needs to be something else on top of it.

• 47:44Speaker 01

  Unless they can come out with some kind of like new distance function or something. They teased a little bit of what they're working on at the AI Engineer Summit, which, yeah, density and whatever other fancy formulas Anton is cooking up. But yeah, I think I tweeted about this maybe like two, three months ago, and I think I pissed off Chrome a little bit. But the best framing of what Anton would respond to here is what people are embedding within vectors is a very different kind of data from what is already within Postgres and MongoDB and all the others. In some sense, it's net new data. And that actually struck a chord with me because that's how I started to understand structured versus unstructured data. That's how I started to understand, you know, one of my kind of heroes is Mark, who's CTO of MongoDB. This guy was the former GM of AWS RDS.

• 48:29Speaker 01

  And for those who don't know, GM is like, you're the mini CEO of that business. And when you work at AWS RDS, you run a one, two billion dollar a year business. And now, and then he quits being Mr. Postgres of AWS to join MongoDB, the enemy. When he gave that speech of like why he did, he was like, actually, if you look at the kind of workloads that's happening, Postgres is doing well, obviously. Structured data always going to be there, but unstructured data and document type data is just rising exponential rate even faster. And like, for him to say that means different things. Anybody could have said that. Anybody could have pointed, made a chart that showed what he did. Anybody could have said that. But for him to have said that, I think it was a very big deal. Cause he, he's rich. He doesn't have to work, but he like believes

• 49:12Speaker 01

  in this so much that he was like, I'll just join MongoDB. So I'm like, okay, there's a real category shift between structured data and unstructured data. I believe it. I don't, I don't think it's just that you can put JSONB inside of Postgres and be done. It's not a NoSQL database. Okay, fine. So what is this new thing of vectors? And how do you think about that as a new kind of data? And I, I think if there's a third category of something beyond unstructured data, I don't know what it is, like context or memory or whatever you call it, whatever you call this kind of new data, that might belong in a new, new category of database and that might create the new MongoDB of this era. And it could be any one of these guys. Right now, you know, Pinecone has, has the lead. I think they're a 750 million dollar company.

• 49:55Alessio Fanelli?

  Uh huh. Valuation.

• 49:56Speaker 01

  Yeah. And then all the others are much smaller. So like, if this is really a new data category and there's a room for a key player, then it's probably going to be one of these guys. By the way, I, I left out Veevate and I put Qdrant in there. Do you know why?

• 50:08Alessio Fanelli?

  No.

• 50:08Speaker 01

  Anthropic and OpenAI both use Qdrant for their internal RAG solutions, which means that for whatever reason, I, I, we should probably interview Qdrant. They passed the evals when Veevate and, and you know, Milvis and all the others didn't, which is interesting.

• 50:22Alessio Fanelli?

  Yeah, yeah, yeah.

• 50:23Speaker 01

  There's, there's a lot that we don't know.

• 50:24Alessio Fanelli?

  Yeah. Interesting. Yeah, I think like, I mean, going back to your point of like Langtrain, building Langsmith. At some point, some of the vector databases are gonna be like, why am I letting my customers use Llama Index? You know, it's like, I should be the rag interface since I'm owning the data.

• 50:40Speaker 01

  That's why I put them next to each other. Right now, they are friends.

• 50:44Alessio Fanelli?

  Yeah, right now.

• 50:45Speaker 01

  Yes.

• 50:45Alessio Fanelli?

  I mean, if we think about the JumpStack era, you know, you had, Vercel started as Zeit, which was just a CDN. And then you had Netlify, you had all these companies, and then Vercel built Next.js. And so they moved down from the CDN to the framework, you know, and it's like, now they use the framework to then enable more cloud and platform products. Which way is it gonna give this way? I, I think what we learned from before is that you rather own the framework and then have the cloud to support it, than like just have Netlify and not have your own framework, just given the way the two companies are doing now.

• 51:22Speaker 01

  So for those who don't know, I worked at Netlify and I was very, very intimately involved in this.

• 51:27Alessio Fanelli?

  We don't have to say any private.

• 51:29Speaker 01

  No, no, it's fine, it's fine. It's well known that Vercel won, and Netlify has pivoted away to a different market. But is it over learning from N of one example that you always wanna own the framework?

• 51:40Alessio Fanelli?

  No, no, no, no, no, because then the counter example is the same, which is Gatsby. Where you own the framework, you don't own the cloud, and then you don't make money either. So it's kinda like, I think we still gotta figure out where like the gravity is in this market, you know? I think a lot of people will say the gravity is in the model. A lot of people will say the gravity is in the embeddings and the data that you put into it. A lot of people don't know what they're talking about. So I think 2024 is supposed to be the year of AI in production. I think we're gonna learn soon who bleeds into where.

• 52:10Speaker 01

  I think that's like, that statement is like the year Linux on the desktop thing. It's just always gonna be true. People are always gonna be saying it. I'm gonna, we're gonna be here one year later and it's like, yeah, this year is the year of AI production. And it's always gonna be incrementally more true. But like, you know, what is the catalyst, what is the big event that says, that you will point to and say, aha, now it's in production? I don't know.

• 52:33Alessio Fanelli?

  I think actually being, it's not in production, you know, like a lot of companies, it's funny, like one, they're just like an inherent timeline that large companies work within. GPT four came out in like April. That's like eight months. It's like most companies don't buy things within eight months, like implement them. So I, I think like part of it, just like a physics time limit, that like even people that have been really interested, you just cannot go through the whole process of getting them live to all of your customers. So I think we'll see more of that in good and bad, right? It's gonna be a lot of failures and a lot of successes, hopefully.

• 53:10Speaker 01

  Yeah. Any other commentary on like tooling, RAG, Ops, anything like that? I always tell people like, as much as I'm interested in fine tuning, I think RAG is here to stay. Like, don't even doubt it. Like this is a necessary part that every AI engineer should know.

• 53:23Alessio Fanelli?

  Yeah, well, I, I think, yeah, it's tied to the infinite context thing, right? I think the leftover question is like, do you wanna have infinite context and hope that the model is good enough at parsing which parts matter to your query? Or do you wanna use RAG and wrap very specific context injection? I think so far, most people will say, I'd rather do context injection with just what I care about, than put a whole document in there and hope the model gets it, but maybe that changes.

• 53:50Speaker 01

  I don't.

• 53:50Alessio Fanelli?

  Yeah, no.

• 53:51Speaker 01

  There's, there's no way it changes.

• 53:54Alessio Fanelli?

  Hey, you know, that's great for Llama Index.

• 53:57Speaker 01

  Yeah, yeah.

• 53:57Alessio Fanelli?

  It's great, like it's gonna make a lot of money, I guess.

• 53:59Speaker 01

  Yeah, yeah. No, it, it's not clear that they're gonna make a lot of money, right? Cause they're just in open source projects. I don't think they've launched a commercial thing yet.

• 54:05Alessio Fanelli?

  I don't think so, because, uh, yeah, Jerry was talking about it on the podcast, but it wasn't.

• 54:09Speaker 01

  Yeah. Yeah. So I mean, we'll see, we'll see what they launch this year. Yeah, I do have.

• 54:13Alessio Fanelli?

  The year of AI in production, the year of Llama Index in production.

• 54:16Speaker 01

  Yeah. Okay, so that's the four wars. We also covered a bunch of other non-wars that we skipped over. I did remember that you actually just published a piece on the semantic versus

• 54:26Alessio Fanelli?

  The syntax to semantics.

• 54:27Speaker 01

  Do you want to cover that as a

• 54:29Alessio Fanelli?

  Yeah, I think like I kind of mentioned this a couple of times on the podcast, but basically the idea of like code has always been the gateway to programming machines. And we spent a lot of time making it easier. So you go from punch cards to like COBOL to C to Python, just to make it easier for the person to read and write the code. And through it, we started adding kind of like this semantic functionalities in it. So in Python, you can do array dot sort. You don't need to know bubble sort. You don't need to know any algorithm that you learn in school to do it. And I think the models are kind of like 100 Xing this, which is like now all you need to do is like create a signup form, you know, where people put a name, email and send it to this endpoint.

• 55:11Alessio Fanelli?

  So it's going to be a lot easier for people that know the semantics of the business, which is, you know, your product managers or your business people, the layer that goes from customer requirements to implementation, basically, and have them intervene in the code. So, you know, how many times as an engineer, you have to like go change some button color or like some button size, like these small things that like you really shouldn't be doing. And now you can have people with natural language intervene in the code and write code that can actually be merged and put in production. I also wrote the bear case for it, which is like, we already have so much trouble getting engineering teams to collaborate and get all their changes together without conflicts

• 55:52Alessio Fanelli?

  and all of these things that maybe also having non-technical people trying to do things would be hard. And models, they just think about solving the task ahead. They don't think about, I've always told my engineers, it's like, you need to leave the code base better than you found it, you know? If you're like writing something, it's like, just, we cannot always keep adding like a quick hacks, you know, and I think models are great at quick hacks. But sometimes it's like, oh, this is like the 16th button that you've changed a style for. You should make a class for it. And that's like the dumbest example. So I think if that happens, then I think I'll be a lot more bullish on like coding agents, you know? Until you can have non-technical people manually query models and look at results and then say, this is ready to go.

• 56:35Alessio Fanelli?

  It's going to be hard to have autonomous agents do it, so.

• 56:38Speaker 01

  Yeah, so I actually had a tweet about it today because Itamar from Codium actually published Flow Engineering as his next evolution of prompt engineering. And they've been working on, you know, in IDE agents. They call it agents. You can debate about the definition of an agent, you know, at the end of the day. My split of it is inner loop versus outer loop, which I think you you understand that maybe I have to explain it to the audience because every time I talk about it to developers, they don't, they've never heard of it. So inner loop is everything that happens between a git commit. Outer loop is everything happens after the the commit is is committed and it's pushed up for PR. So maybe that's too reductive, but that's something like that, right? Like inner loop happens within your IDE, outer loop happens in GitHub, something like that.

• 57:18Speaker 01

  Okay, so I think your conception of an agent is outer loop E, especially if it's non-technical, right? Like the the dream, like you mentioned sweep.dev in your write-up, and there there's also code gen, there's also maybe Morph, uh, depends what Morph is doing. And and and there's a bunch of other people all doing this this stuff. Even small developer was also like, you know, write it write it in English and then create a codebase. And I think it's just not ready for that. Outer loop is a mirage, going to forever be five years away. And the people working on inner loop companies have have been the the right bet. And you can work on inner loop agents. Actually, code interpreter is an inner loop agent in a sense of like limited self-driving, right?

• 57:59Speaker 01

  It's kind of like you have to have your attention on it. You have to watch it. It can only drive the small distance, but it is somewhat self-driving. And so I think if you have this like gradations in your outlook on autonomous agents and you don't expect everything to jump to level five at once. But if you have an idea of what level one, two, three, four, five looks like for you, I haven't really defined it apart from this concept of inner loop versus outer loop. But once you've defined it, then you can be like, oh, like we're making real progress on this stage and, you know, this other stage too early for now, but at some point, somebody will do it.

• 58:27Alessio Fanelli?

  Yeah, I think like, yeah, maybe level one is like, I think of it more as just the auto completion in the IDE, you know? Level two is like asking cursor, hey, how can I make this change, you know? But then level three should be like, to me, it's like, we need to separate the inner loop from the IDE, you know? I need to make a code change. Sometimes I shouldn't go in the IDE. Sometimes I should be in the UI of the product and say, hey, that needs to be changed. Kind of like all the preview environments companies want you to put comments, the PMs put comments, like, how do you go from that to code changes? There should be enough there to make the code changes happen, you know, through a supervised interface.

• 59:07Speaker 01

  That's outer loop.

• 59:08Alessio Fanelli?

  Yeah, but I think what these models are doing is like change where the loops start and end, you know, because now you can create code in the outer loop, you know, before you couldn't do it.

• 59:18Speaker 01

  That's the dream.

• 59:19Alessio Fanelli?

  Yeah.

• 59:20Speaker 01

  Yeah, anyway, my focus right now, I'll say if anyone cares, is like, you know, I think the only thing that's working is inner loop and you should just use inner loop things aggressively, build inner loop things aggressively, invest in them and then keep an eye on the other loop stuff.

• 59:33Alessio Fanelli?

  Yeah.

• 59:33Speaker 01

  It's still very early. I did invest in CodeGen, this J-hacks thing, which we mentioned briefly in the Source Graph episode. Do we have other things that we want to mention or do you want to sort of keep it to just the four wars? Okay, maybe like top two things from December that you have commentary on.

• 59:49Alessio Fanelli?

  I think the needle in a haystack thing.

• 59:52Speaker 01

  Okay, maybe you want to explain that first.

• 59:53Alessio Fanelli?

  Yeah, basically like Anthropic, there was like one example floating around about Claude's context window and you basically gave it this like super long context on, I think like things to do in San Francisco or something like that. And then it was like, what is the most fun thing to do in SF? And they made this nice chart of like, okay, based on where it is in the context, it gave a better or worse response. And then Anthropic responded and they were like, oh, you just need to add, here's the most relevant sentence in the context as part of the assistant prompt. And then the chart turns all green all of a sudden. And I'm like, we cannot still be here, right? Like, it cannot, this is like some...

• 1:00:34Speaker 01

  And you have Anthropic like telling people, oh yeah, it's just like, just add this magic string and it works.

• 1:00:38Alessio Fanelli?

  Yeah, it's some like Riley Goodside wizardry. It's like, I don't want to do that anymore. I thought like, you know, in the early days of GPTs, like Riley Goodside was doing so much great work on like prompt engineering and whatnot. We shouldn't be there anymore. There shouldn't be somebody telling me, or like the GPT four, like I'll give you a $200 tip. If you do this right and like...

• 1:01:00Speaker 01

  I collected a whole bunch of like state of the art prompting techniques. So if you tip the model, it'll give you better results if you promise that it'll... So, okay, here's the current state of the art for GPT prompting. It's Monday in October, the most productive day of the year. You have to take a deep breath and you have to think step by step. You have to return the full script. You are an expert on everything. I will pay you $20. Just do anything I ask you to do. I will tip you $200 every request you answer correctly. And your competitor models said you couldn't do it, but you can do it.

• 1:01:28Speaker 01

  Oh, I think there's another one that I did put in here. It was like, you know, my grandmother is dying. This is an emergency. Please help me do it.

• 1:01:33Alessio Fanelli?

  Yeah, that that's actually my, I think my most viewed tweet ever at OpenAI Day. I tweeted, no more return JSON or my grandma is gonna die.

• 1:01:41Speaker 01

  Oh.

• 1:01:42Alessio Fanelli?

  When they announced JSON mode and people love to get grandma.

• 1:01:45Speaker 01

  I haven't heard as much uptake on JSON mode. I think it's still.

• 1:01:50Alessio Fanelli?

  That's the thing with all this AI stuff, right? It's like, I mean, and sometimes we're like part of it. If I think about our chat G P D plugins. Episode, I think in the moment, people are just like, oh, this is gonna be such a big deal.

• 1:02:01Speaker 01

  Mm hmm.

• 1:02:01Alessio Fanelli?

  And then it takes very the amount of times that I really pick up, you know?

• 1:02:06Speaker 01

  Do you think that will happen in G P Ts?

• 1:02:07Alessio Fanelli?

  I I think like most people that I see using G P Ts right now are trying to get around some sort of weird limitation of the base model. You know, just trying to have a better system prompt. Like at some point, there's limited value to get out of it. So the question is like, what's gonna incentivize people to build more on it versus just building their own thing out of it? I don't know.

• 1:02:31Speaker 01

  Yeah. Okay. So I guess my pick for highlight of last above, there's two. Uh, one, we finally got Gemini.

• 1:02:37Alessio Fanelli?

  Right.

• 1:02:38Speaker 01

  I think the marketing was dishonest.

• 1:02:40Alessio Fanelli?

  Yeah, we need the soundboard. Wow, wow, wow.

• 1:02:43Speaker 01

  But still, it is a sort of model. It is a credible, very, very credible alternative to OpenAI. And we should, we should be happy for that because otherwise we live in an OpenAI only world. And Gemini is basically the only other sort of leading contender until Lama three drops, whenever Lama three comes out.

• 1:02:58Alessio Fanelli?

  It's kind of, I mean, Sock said today they're training it.

• 1:03:01Speaker 01

  Yeah, it sounds like today they're training it. For me, I guess I'm still very interested in like the, that's a hardware meta game. This is a much smaller stakes, but very personal. I think recently, especially, you know, we're recording this mid-January. So after CES, after Rabbit R one launched, I think there's a lot of interest in in hardware. I don't know how you feel about it as an enterprise software investor, but I think that hardware is hard. But also it captures context and is makes AI usable in ways that you cannot currently think about. And and you know, everyone dreams of building an assistant like her in the movie Her, that is a hardware piece. That is actually not only software and probably the hard part is the engineering for the hardware.

• 1:03:41Speaker 01

  And then the the sort of AI engineering for the assistant within the hardware. So, yeah, I mean, yeah, I'm an investor in Tab. I see a lot of, you know, interest this month, but it started last month with the launch of Humane as well. I don't know if you have thoughts on any of those things.

• 1:03:53Alessio Fanelli?

  Well, I think this year we also get the Apple Vision Pro thing. So I think this is, there's gonna be a ton of experimentation. I think Rabbit got the right nostalgia factor, you know, it kind of looks like a

• 1:04:05Speaker 01

  The toy looking, yeah.

• 1:04:07Alessio Fanelli?

  Yeah, it looks like a Yumboy Advance, something like that. I'm curious to see what you got beyond that. I think, yeah, I mean, obviously, like, right where we have the studio building tab. And I, I think that's another interesting form factor. And I think if you ask them, I, I think in our circles, a lot of people are like, well, what about privacy and all these things? But he will tell you that we're kind of like a special group that most people value convenience over privacy, as you learn from the social medias of the last few years. So, yeah, I'm really curious to to see how it develops.

• 1:04:40Speaker 01

  I really like technology where you're slightly uncomfortable with it on a social level. For Uber, it was like this regulation around taxis. For Airbnb, it was, you know, staying in strangers' homes. And now it turns out for OpenAI, it was trading on people's content.

• 1:04:53Alessio Fanelli?

  Right.

• 1:04:54Speaker 01

  Right. Now it's becoming a matter of regulation and OpenAI's data partnerships are, you know, a form of, you know, private regulatory capture, which is a playbook that is fantastic. I hope it was on purpose because whoever did that is a genius. So I'm like, okay, like, I do think that every great new company, especially on the consumer side, is provocative in that sense. Like, they're doing something that is not yet kosher. And so I think, like, the humanes, the tabs, anything that is working on that front where it's like, yeah, I'm not sure I'm comfortable with this. And then, but maybe it could change. That is a really interesting shift.

• 1:05:25Alessio Fanelli?

  Yeah.

• 1:05:26Speaker 01

  I'm excited from that point of view, but at the same time, most hardware companies fail very, very quickly. They have a very hot start and then, you know, everyone puts it in their drawer and then never looks at it again. So I'm very, very aware of that. Here's the core thing of it, right? Avi doesn't think it's a hardware company. Most of the, the cost of the $600 for Tab is going towards GPT costs because it's actually processing context. And the whole idea is that context is all you need. Like, in this world of, like, you know, AI applications, like, whoever has the most unique context wins. Right? A unique context could be the quality data war, right? Like, a unique context is like, you know, I have Reddit info, I have Stack Overflow info, I have New York Times info. If I have info on everything you say, you say and do at all times, that is something that no one else has.

• 1:06:08Speaker 01

  And if it becomes a good store of that, then like, what can you do with that? So I'm most excited for him to, to expose the developer API because then I can come in and do all my software stuff. But he has to build the hardware layer and get acceptance for that first.

• 1:06:20Alessio Fanelli?

  Right. Yeah, no, I'm I'm excited to see. I I'm sure we're gonna see a lot of people walk around with them, so.

• 1:06:27Speaker 01

  Actually, so I I think he doesn't like me because I asked for off button. I guess I want to be able to guarantee you, if we're having a conversation, I want to say, you see, it's off. Right? It's it's kind of like, oh, yeah, my phone is on silent mode. Right? And there's a physical silent mode button. But now, but he he just wants it to be always on.

• 1:06:44Alessio Fanelli?

  That's a whole new market, like a soundproof. Like, soundproof storage for your AI pendant, so that you can guarantee the person.

• 1:06:52Speaker 01

  Yeah, yeah.

• 1:06:53Alessio Fanelli?

  Cannot hear you. Awesome. This was fun. Please, if you're still listening after one hour twenty one minutes, let us know what we did right, what we did wrong, what you would like to see differently. It's the first time we tried this out, but yeah.

• 1:07:06Speaker 01

  Awesome. Thanks for doing this.

• 1:07:07Alessio Fanelli?

  Cool.