2024 in AI Startups [LS Live @ NeurIPS]

Happy holidays! We’ll be sharing snippets from Latent Space LIVE! https://lu.ma/LSLIVE through the break bringing you the best of 2024 from friends of the pod! For NeurIPS last year https://www.latent.space/p/neurips-2023-papers we did our standard conference podcast coverage interviewing selected papers (that we have now also done for ICLR https://www.latent.space/p/iclr-2024-benchmarks-agents?utm_source=publication-search and ICML https://www.latent.space/p/icml-2024-video-robots), however we felt that we could be doing more to help AI Engineers 1) get more industry-relevant content, and 2) recap 2024 year in review from experts. As a result, we organized the first Latent Space LIVE!, our first in person miniconference, at NeurIPS 2024 in Vancouver. For our opening keynote, we could think of no one better to cover 'The State of AI Startups' than our friend Sarah Guo (AI superinvestor https://x.com/EricNewcomer/status/1869492698333528361, founder of Conviction, host of No Priors!) and Pranav Reddy (Conviction partner) to share their takes on how the AI landscape evolved in 2024 examine the evolving AI landscape and what it means for startups, enterprises, and the industry as a whole! They completely understood the assignment https://x.com/saranormous/status/1866933642401886707. Recorded live with 200+ in-person and 2200+ online attendees at NeurIPS 2024, this keynote kicks off our mini-conference series exploring different domains of AI development in 2024. Enjoy! Links Slides: https://x.com/saranormous/status/1866933642401886707 Sarh Guo: https://x.com/saranormous Pranav Reddy: https://x.com/prnvrdy Full Video on YouTube Want more content like this? Like and subscribe to stay updated on our latest talks, interviews, and podcasts. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:15Speaker 00

  Welcome to Latent Space Live, our first mini-conference held at NeurIPS 2024 in Vancouver. This is Charlie, your AI co-host. When we were thinking of ways to add value to our academic conference coverage, we realised that there was a lack of good talks just recapping the best of 2024 going domain by domain. We sent out a survey to the over 900 of you who told us what you wanted, and then invited the best speakers in the latent space network to cover each field. 200 of you joined us in person throughout the day with over 2,200 watching live online. Today, we're kicking it off with a keynote on the state of AI startups.

• 0:58Speaker 00

  Sarah Guo, founder at Conviction and host of No Priors podcast, and Pranav Reddy, partner at Conviction and former engineer at Neva. They'll be unpacking the top five themes of 2024. What ideas are working and what's not? From shifting market opportunities to why the supposed advantages of big tech incumbents might not be as strong as they seem. As always, don't forget to check the show notes for the YouTube link to their talk as well as their slides. Watch out and take care.

• 1:30Speaker 01

  Hi, everyone. My name is Sarah Guo and thanks to Sean and friends here for having me and Pranav. So I'd start by just giving 30 seconds of intro. I promise this isn't an ad. We started a venture fund called Conviction about two years ago. Here is a set of the investments we've made. They range from companies at the infrastructure level in terms of feeding the revolution to foundation model companies, alternative architectures, domain-specific training efforts, and of course, applications. And the premise of the fund, Sean mentioned I worked at Greylock for about a decade before that and came from the product engineering side,

• 2:10Speaker 01

  was that we thought that there was a really interesting technical revolution happening, that it would probably be the biggest change in how people use technology in our lifetimes, and that represented huge economic opportunity. And maybe that there'd be an advantage versus the incumbent venture firms in that when the floor is lava, the dynamics of the markets change, the types of products and founders that you back change. It's a lot for existing firms to ingest, and a lot of their mental models may not apply in the same way. And so there was an opportunity for first principles thinking. And if we were right, we'd do really well and get to work with amazing people. And so we are two years into that journey, and we can share some of the opinions and predictions we have with all of you. I'm proud it's going to start us off.

• 2:55Speaker 02

  So, quick agenda for today. We'll cover some of the model landscapes and themes that we've seen in 2024, what we think is happening in AI startups, and then some of our latent priors on what we think is working in investing. So the, I thought it'd be useful to start from like what was happening at NERIP's last year in December 2023. So in October 2023, OpenAI had just launched the ability to upload images to ChatGPT, which means up until that moment, it's hard to believe that like roughly a year ago, you could only input text and get text out of ChatGPT. The Mistral folks had just launched the Mistral model right before the beginning of NERIP's. Google had just announced Gemini. I very genuinely forgot about the existence of Bard before making these slides. And Europe had just announced that they were doing their first round

• 3:39Speaker 02

  of AI regulation, but not to be their last. And when we were thinking about like what's changed in 2024, there's at least five themes that we could come up with that feel like they were descriptive of what 2024 has meant for AI and for startups. And so we'd start with, first, it's a much closer race on the foundation model side than it was in 2023. So this is LLM Arena. They're, ask users to rate the evaluations from, of generations from specific prompts. So you get two responses from two language models, answer which one of them is better. The way to interpret this is like roughly 100 Elo difference means that you're preferred two thirds of the time. And a year ago, every OpenAI model was like more than 100 points better than anything else. And the view from the ground was roughly like, OpenAI is the IBM, there is no point in competing.

• 4:24Speaker 02

  Everyone should just give up, go work at OpenAI, or attempt to use OpenAI models. And I think the story today is not that. I think it would have been unbelievable a year ago if you told people that A, the best model today on this, at least on this eval, is not OpenAI, and B, that it was Google would have been pretty unimaginable to the majority of researchers. But Actually, there are a variety of proprietary language model options and some set of open source options that are increasingly competitive. And this seems true not just on the eval side, but also in actual spend. So this is ramp data. There's a bunch of colors, but it's actually just OpenAI an anthropic spend. And the OpenAI spend at the beginning, at the end of last year in November of 23, was close to 90% of total volume.

• 5:06Speaker 02

  And today, less than a year later, it's closer to 60% of total volume, which I think is indicative both that language models are pretty easy APIs to switch out and people are trialing a variety of different options to figure out what works best for them. Related, second trend that we've noticed is that open source is increasingly competitive. So this is from the scale leaderboards, which is a set of independent evals that are not contaminated. And on a number of topics that actually the foundation models clearly care a great deal about, open source models are pretty good on math, instruction following, and adversarial robustness. The Llama model is amongst the top three of evaluated models. I included the agentic tool use here just to point out that this isn't true across the board.

• 5:48Speaker 02

  There are clearly some areas where foundation model companies have had more data or more expertise in training against these use cases, but models are surprisingly and increasing, open source models are surprisingly increasingly effective. This feels true across evals. This is the MMLU eval. I want to call out two things here. One is that it's pretty remarkable that the ninth best model and two points behind the best state of the art models is actually a 70 billion perimeter model. I think this would have been surprising to a bunch of people who were, the belief was largely that most intelligence is just an emergent property and there's a limit to how much intelligence you can push into smaller form factors. In fact, a year ago, the best small model or under 10 billion perimeter model would have been Mistral 7B, which on this eval, if memory serves us, is somewhere around a 60.

• 6:32Speaker 02

  And today, that's the Llama 8B model, which is more than 10 points better. The gap between what is state of the art and what you can fit into a fairly small form factor is actually shrinking. And again, related, we think the price of intelligence has come down substantially. This is, this is a graph of flagship OpenAI model costs, where the cost of the API has come down roughly Eighty, eighty-five percent and call it the last year, year and a half, which is pretty remarkable. This isn't just OpenAI, too. This is also like a full set of models. This is from artificial analysis, which tracks cost per token across a variety of different APIs and public inference options. And like, we were doing some math on this. If you wanted to recreate like what a, the kind of data that a text editor had, or that,

• 7:15Speaker 02

  like something like Notion or Coda, that's somewhere in the volume of a couple thousand dollars to create that volume of tokens, that's pretty remarkable and impressive. It's clearly not the same distribution of data, but just as like a sense of scope, there's an enormous volume of data that you can create.

• 7:31Speaker 02

  And then fourth, we think new modalities are beginning to work. Start quickly with biology. We're lucky to work with the folks at Chai Discovery, who just released Chai 1, which is open source model that outperforms AlphaFold 3. It's impressive that this is like roughly a year of work with a pretty specific data set and then pretty specific technical beliefs, but models in domains like biology are beginning to work. We think that's true on the voice side as well. Point out that there were voice models before, things like Eleven Labs have existed for a while, but we think low latency voice is more than just a feature. It's actually a net new experience, interaction. Using voice mode feels very different than the historical transcription first models. Same thing with many of the Cartesian models. And then a new nascent use case is execution.

• 8:13Speaker 02

  So Claude launched computer use, OpenAI launched code execution inside of Canvas yesterday. And then I think Devin just announced that they, you can all try it for $500 a month, which is pretty remarkable. It's a set of capabilities that historically never been available to vast majority of population, and I think we're still in early innings. Cognition, the company was founded under a year ago. First product was roughly nine months ago, which is pretty impressive.

• 8:34Speaker 01

  And if you recall, like a year ago, the point of view on Sweetbench was like, it was impossible to surpass, what, like 13% or so? And I think the whole industry now considers that, if not trivial, accessible.

• 8:51Speaker 02

  Last new modality, what we wanted to call out, although there are many more, is video. I took the, I got early access to Sora and managed to sign up before they cut off accessies. So here is my favorite joke in the form of a video. Hopefully someone here can guess it.

• 9:08Speaker 02

  Yeah, you're telling me a shrimp fried this rice? It's a pretty bad joke, but I really like it. And I think this one, the next video here is one of our portfolio companies, HeyGen, that translated and does the dubbing for, or lip-sync and dubbing for live speeches. So this is Javier Millay, who speaks in Spanish, but here you will hear him in English if this plays. And you can see that you can capture the original tonality of his speech and performance. I think out of here doesn't work, but we'll push something publicly.

• 9:41Speaker 03

  Does it? Did you try it?

• 9:43Speaker 02

  Let's give it a shot. Yeah.

• 9:45Speaker 03

  Let's give it a shot. I think it's on mute.

• 9:47Speaker 02

  Excellent. Yeah, and you can hear that this captures like his original tone and like the emotion in his speech, which is definitely new and pretty impressive from new models. So the last, yeah, that makes sense. The last point that we wanted to call out is the much purported end of scaling. I think there is a great debate happening here later today on the question of this, but we think at minimum, it's hard to deny that there are at least some limits to the clear benefits to increasing scale. But there also seems like there are new scaling paradigms. So the question of test time compute scaling is a pretty interesting one. It seems like OpenAI has cracked a version of this that works, and we think, A, Foundation

• 10:29Speaker 02

  Model Labs will come up with better ways of doing this, and B, so far, it largely works for very verifiable domains, things that look like math and physics and maybe secondarily software engineering, where we can get an objective value function. And I think an open question for the next year is going to be, how do we generate those value functions for spaces that are not as well-constrained or well-defined? And so the question that this leaves us in is like, well, what does that mean for startups? And I think a prevailing view has been that we live in an AI bubble. There's an enormous amount of funding that goes towards AI companies and startups that is largely unjustified based on outcomes and what's actually working on the ground. And startups are largely raising money on hype. And so we pulled some pitch book data, and the 2024 number is like probably incomplete since not all rounds are being reported, and largely suggests like actually there is

• 11:14Speaker 02

  a substantial recovery in funding. And maybe 2025 looks something like 2021. But if you break out the numbers here a bit more, the red is actually just a small number of foundation model labs, like what you would think of as the largest labs raising money, which is upwards of $30 to $40 billion this year. And so the reality of the funding environment actually seems like much more sane and rational. It doesn't look like we're headed to a version of 2021. In fact, the foundation model labs account for an outsized amount of money being raised. But the the set of money going to companies that are working seems much more rational. And we wanted to give you, we can't share numbers for every company, but this is one of our portfolio companies growing really, really quickly. We think zero to 20 and just PLG style spending is pretty impressive.

• 11:59Speaker 02

  If any of you are doing better than that, you should come find us. We'd love to chat.

• 12:04Speaker 02

  And so what we wanted to try and center a discussion on, this is certainly not all of the companies that are making $10 million more or revenue and growing, but we took a selection of them and wanted to give you a couple of ideas of patterns that we've noticed that seem to be working across the board.

• 12:20Speaker 02

  The first one that we've noticed is like first wave service automation. So we think there's a large amount of work that doesn't get done at companies today, either because it is too expensive to hire someone to do it. It's too expensive to provide them context and enable them to be successful at whatever the specific role is, or it's too hard to manage those set of people. So for Scribe, it's too expensive to hire those specific set of people. For Sierra and Decagon, for customer support style companies, it's really useful to do like next level automation. And then there's obviously growth in that. And for Harvey and even at the story is you can do first wave professional services and then grow beyond that. Second trend that we've noticed is better search new friends. So we think that there is a, it's pretty impressive like how effective text modalities have been.

• 13:04Speaker 02

  So Character and Replica have been remarkably as successful companies. And there's a whole host of not safer work chatbots as well that are pretty effective at just text generation. They're pretty compelling mechanisms. And on the productivity side, perplexing and glean have demonstrated this as well. I worked at a search company for a while. I think the changing paradigms of how people capture and learn information is pretty interesting. We think it's likely text isn't the last medium. There are infographics or sets of information that seem more useful or sets of engagement that are more engaging. But this feels like a pretty interesting place to start.

• 13:33Speaker 01

  So one thing that I've worked on investing in in a long time is democratization of different skills, be they creative or technical. This has been an amazing few years for that, across different modalities, audio, video, general image media, text, and now code, and really fully functioning applications. One thing that's really interesting about the growth driver for all of these companies is the end users in large part are not people that we thought of as, we, the venture industry, you know, the royal we, thought of as important markets before. And so a premise we have as a fund is that there's actually much more instinct for creativity,

• 14:16Speaker 01

  visual creativity, audio creativity, technical creativity than, like, there's latent demand for it, and AI applications can really serve that. I think in particular, Midjourney was a company that is in the vanguard here, and nobody understood for a long time, because the perhaps outside view is like, how many people want to generate images that are not easily, you know, they're raster, they're not easily editable, they can't be used in these professional contexts in a complete way, and the answer is like an awful lot, right, for a whole range of use cases, and I think we'll continue to find that, especially as the capabilities improve. And we think the the range of quality and controllability that you can get in these

• 14:57Speaker 01

  different domains is still, it's very deep, and we're still very early. And then I think if we're in the first or second inning of this AI wave, one obvious place to go invest and to go build companies is the enabling layers, right? Shorthand for this is obviously compute and data. I think the needs for data are largely changed now as well. You need more expert data, you need different forms of data, we'll talk about that later in terms of who has, like, let's say, reasoning traces in different domains that are interesting to companies doing their own training. But this is, this is an area that has seen explosive growth, and we continue to invest here. Okay, so maybe time for some opinions.

• 15:45Speaker 01

  There was a prevailing narrative that, you know, some part from companies, some part from investors. It's a fun debate as to where is the value in the ecosystem and can there be opportunities for startups? If you guys remember the phrase GPT wrapper, it was like the dominant phrase in the tech ecosystem for a while. And what it represented was this idea that there was no value at the application layer. You had to do pre-training and then like nobody's going to catch OpenAI in pre-training. And you know, this isn't, this isn't like a knock on OpenAI at all. These labs have done amazing work enabling the ecosystem and we continue to partner with them and, and others. But, but it's simply untrue as a narrative, right?

• 16:30Speaker 01

  The odds are clearly in favor of a very, very rich ecosystem of innovation. You have a bunch of choices of models that are good at different things. You have price competition, you have open source. I think an underappreciated impact of test time scaling is you're going to better match user value with your spend on compute. And so if you are a new company that can figure out how to make these models useful to somebody, the customer can pay for the compute. Instead of you taking as a, as a startup, the CapEx for pre-training or, or RL upfront. And as Pranav mentioned, you know, small models, especially if you know the domain can be unreasonably effective.

• 17:11Speaker 01

  And the product layer has, if we look at the sort of cluster of companies that we described, shown that it is creating and capturing value and that it's actually a pretty hard thing to build great products that leverage AI. So broadly, like we have a point of view that I think is actually shared by many of the labs. That the world is full of problems and the last mile to go take even AGI into all of those use cases is quite long. Okay, another prevailing belief is that, or you know, another great debate that Sean could host is like, does the value go to startups or incumbents? We must admit some bias here, even though we have, you know, friends and portfolio, former portfolio companies that would be considered incumbents now.

• 17:54Speaker 01

  But, uh, oh, sorry. Swap, swap, swap views. Sorry. You know, there are, there are markets in venture that have been considered traditionally like too hard, right? Like just bad markets for the, the venture capital spec, which is capital efficient, rapid growth. That's a venture backable company where the end output is a, you know, a tens of billions of dollars of enterprise value company. And, and these included areas like legal, healthcare, defense, pharma, education, you know, any traditional venture firm would say like, bad market, nobody makes money there. It's really hard to sell. There's no budget, et cetera. And, and one of the things that's interesting is if you look at the cluster of companies

• 18:37Speaker 01

  that has actually been effective over the past year, some of them are in these markets that were traditionally not obvious, right? And so perhaps one of our more optimistic views is that AI is really useful. And if you make a capability that is novel, that is several magnitudes, orders of magnitude cheaper, then actually you can change the buying pattern and the structure of these markets. And maybe the legal industry didn't buy anything because it wasn't anything worth buying for a really long time, is one example. We, we also think that, like, what was the last great consumer company? Maybe it was Discord or Roblox in terms of things that started that have just like really enormous user basis and engagement, until, you know, we had these consumer chatbots

• 19:21Speaker 01

  of different kinds and, and like the next, perhaps the next generation of search. As Pranav mentioned, we think that the

• 19:29Speaker 01

  Opportunity for social and media generation and games is large and new in a totally different way. And finally, in terms of the markets that we look at, I think there's broad recognition now that you can sell against outcomes and services rather than software spend with AI because you're doing work versus just giving people the ability to do a workflow. But if you take that one step further, we think there's elastic demand for many services. Right? Our classic example is there's on order of 20 to 25 million professional software developers in the world. You know, I imagine much of this audience is technical.

• 20:11Speaker 01

  Demand for software is not being met. Right? If we take the cost of software and high quality software down two orders of magnitude, we're just going to end up with more software in the world. We're not going to end up with fewer people doing development. At least that's what we would argue. And then finally, on the incumbent versus startup question, the prevailing narrative is incumbents have the distribution, the product surfaces and the data. Don't bother competing with them. They're going to create and capture the value and share some of it back with their customers. I think this is only partially true. They incumbents have the distribution. They have always had the distribution. Like the point of the startups, you have to go fight with a better product or a more clever product and maybe a different business model to go get new distribution.

• 20:56Speaker 01

  But the specifics around the product surface and the data, I think, are actually worth understanding. There's a really strong innovator's dilemma. If you look at the SaaS companies that are dominant, they sell by seat. And if I'm doing the work for you, I don't necessarily want to sell you seats. I might actually decrease the number of seats. The tens, the decades of years, the millions of man and woman hours of code that have been written to

• 21:24Speaker 01

  Enable a particular workflow in CRM, for example, may not matter if I don't want people to do that workflow of filling out the database every Friday anymore. And so I do think that this sunk cost or the incumbent advantage gets highly challenged by new UX and code generation as well. And then one disappointing learning that we found in our own portfolio is no one has the data we want in many cases, right? So imagine you are trying to automate a specific type of knowledge work. And what you want is the reasoning trace, all of the inputs and the output decision. Like, that sounds like a very useful set of data.

• 22:07Speaker 01

  And the incumbent companies in any given domain, they never save that data, right? Like, they have a database with the outputs some of the time. And so I would say one of the things that is worth thinking through as a startup is when an incumbent says they have the data, like, what is the data you actually need to make your product higher quality? Okay, so in summary, you know, our shorthand for the set of changes that are happening is software 3.0. We think it is a full-stack rethinking and it enables a new generation of companies to have a huge advantage. The speed of change favors startups. If the floor is lava, it's really hard to turn a really big ship. I think that some of the CEOs of large companies now are incredibly capable,

• 22:51Speaker 01

  but they're still trying to make 100,000 people move very quickly in a new paradigm. The market opportunities are different, right? These markets that we think are interesting and very large, like represent a trillion dollars of value, are not just the replacement software markets of the last two decades. It's not clear what the business model for many of these companies should be. Sierra just started talking about charging for outcomes. Outcomes-based pricing has been this holy grail idea in software, and it's been very hard, but now we do more work.

• 23:22Speaker 01

  There are other business model challenges. And so, you know, our companies, they spend a lot more on compute than they have in the past. They spend a lot with the foundation model providers. They think about gross margin. They think about where to get the data. It's a time where you need to be really creative about product versus just replace the workflows of the past. And it might require ripping out those workflows entirely. It's a different development cycle. I bet most of the people in this room have written evals and compared to the academic benchmark to a real world eval and said like, you know, that's not it. And how do I make a user understand the non deterministic

• 24:04Speaker 01

  nature of these outputs or gracefully fail? I think that's like a different way to think about product than in the past. And we need to think about infrastructure again, right? There was this middle period where the cloud providers, the hyperscalers took this problem away from software developers. And it was all just going to be like, I don't know, front end people at some point. And it's like, we are not there anymore. We're back in the hardware era where people are acquiring and managing and optimizing compute. And I think that will really matter in terms of capability in companies. So I guess we'll end with a call to action here and encourage all of you to seize the opportunity. It is the greatest technical and economic opportunity that

• 24:44Speaker 01

  we've ever seen. Like, we made a decade plus career type bet on it. And

• 24:52Speaker 01

  we do a lot of work with the foundation model companies. We think they are doing amazing work and they're great partners and even co investors in some of our efforts. But I think all of the focus on their interesting missions around AGI and safety do not mean that there are not opportunities in other parts of the economy. The world is very large and we think much of the value will be distributed in the world through an unbundling and eventually a rebundling, as often happens in technology cycles. So we think this is a market that is structurally supportive of startups. We're really excited to try to work with the more ambitious ones. And the theme of 2024 to us has been like, well, thank goodness, this is a this is an ecosystem that is much

• 25:36Speaker 01

  friendlier to startups than 2023. It's what we hoped. And so, you know, please ask those questions and take advantage of the opportunity.

• 25:51Speaker 03

  I think we have some questions. We also think there's questions online.

• 25:54Speaker 04

  Okay, so if some of these companies can go from, you know, one to 20 in such a short amount of time, do you think that they can also disappear in a short amount of time?

• 26:05Speaker 01

  I can, I can take this one. I mean, I think you've seen companies go from zero to 80 million and stall out pretty badly, actually. So, your data's correct. There's gonna be, there's a set of challenges that are just the challenges of scale, right? Like, I think sometimes the revenue numbers in these companies can overstate the maturity of the businesses themselves, right? They need to figure out how to serve customers. They need to scale their leadership. They need to prepare to service these customers with the right quality level. And, you know, like the company that we showed that went zero to 20, that company has 20 people, right? And they have, you know, X hundred thousand users is, yeah, is very challenging.

• 26:51Speaker 01

  And so I think there, there's a set of good, hard problems that these companies will have. I think part of the, like, most catchphrases or memes, they don't catch on unless there's some seed of truth. And so there was a set of companies that were described by this term GPT wrapper that were not more than a somewhat trivial set of prompts and SEO pages that directed people to a particular use case. And I think that's not, that's like, likely not a durable position as a technology company. And so it's not a very clean answer for you. It's a nuanced one. But some of the value that is represented by this, I'm gonna scroll back to it.

• 27:35Speaker 01

  Some of this value that is represented by this cluster is durable, and that's the thing that we are interested in.

• 27:43Speaker 01

  The, the zero to 20, and the zero to 80, and then collapse, it's actually valuable. It's just not durable, right? Users are voting for it, and other people can compete. And so, you know, we kind of separate these two questions of like, you know, which of these companies is defensible? And where is the revenue or the usage, not a novelty, but something that's really important to like, work or play or communication?

• 28:10Speaker 01

  Sean, do you want me to take questions, or do you want to do it?

• 28:15Speaker 01

  Hi.

• 28:17Speaker 05

  Hi, I don't think my mic, oh, here it goes. So if all of these companies need a lot more money, and this is the greatest economic opportunity ever, don't we need much bigger venture funds? Like orders of magnitude bigger? And won't the economics of those funds be really broken if they're still raising $40 million, like, could have invested a bunch of seed company funds?

• 28:39Speaker 01

  Okay, this is a bit of a triggering question for me, because I take a particular point of view on it. Hopefully, without arrogance, we've chosen to raise funds that are relatively small as early-stage investors. And part of it is the view of, like, this company that, you know, this company,

• 29:01Speaker 01

  I think they've spent, like, maybe $7 million to date. Right? And so the view that all AI product companies, or all AI companies in general, are very expensive is not true, objectively. We have several companies that are expensive in the traditional sense of SaaS. Like, we've got to go hire a lot of go-to-market people, and we have to pay them, and there's a J-curve of that investment before it comes back in repeatable SaaS revenue.

• 29:28Speaker 01

  And, you know, I think inference revenue. We have companies that are profitable or break even and have been incredibly efficient, and we have companies that spend a lot upfront. And so I think there's an entire range. Our view as a firm is that, you know, very early on,

• 29:49Speaker 01

  my friend Alad has a funny phrase here, which is "no GPU before product-market fit." I think that is not always true. We have given people GPUs before anything, right? But there is a shred of truth in this, which is you can experiment. Like, thank you to the OpenAI and Anthropic and other companies of the world that allow great product people to experiment at very low cost, very incrementally. And so I think much of our portfolio looks like those companies, where you're going to see what kind of value you can bring to users without spending a ton upfront. As one example, like, we just saw New fine tuning interfaces for O1 come out.

• 30:32Speaker 01

  The amount of data that you need to, in theory, improve those models for a particular domain is very small. If that pans out, like, that's incredibly encouraging as well. So I would say, like, our goal is to work with the most important companies in AI with a relatively small fund. And I think that most companies don't actually, they don't benefit from a huge amount of capital upfront.

• 31:01Speaker 02

  The only thing I would add to that is, I think an interesting trend is that we work with a number of second-time founders whose point of view this time around is like, we're never going to make the company that big again. I think it's not a surprise. Actually, I was doing the math in my head, and this rough ratio of $1 million of revenue for, per employee of early-stage company holds true for, like, a remarkable number of our companies. Like, a number of our companies have more millions in revenue than they do employees. And the point of view of a bunch of these CEOs, like, we're going to keep it that way. Like, we're not going to grow into a giant team. AI will make us much more efficient, and if you believe in the grand vision of much of the intellectual labor that we do should actually just be captured by some number of models. And we can build much more long-term, efficient businesses than we have been able to historically.

• 31:44Speaker 01

  I do think it's an interesting question, because if we think there is this much opportunity, like, your opportunity doesn't come evenly, right? And so I'd say our investment pacing is higher than, I guess, mine has been traditionally. And another part of our view is like, okay, well, we want to offer, and we want to offer founders a certain service level. And, you know, founders can decide if they want that or not, but it is, it's very time-expensive to us. We can only work with that many companies. We think many more are really interesting, and that is one of the reasons that Pranav and I did this program for the ecosystem called Embed, where we can work with a larger set of companies. We own less, but we give them, you know, a network and some guidance.

• 32:27Speaker 01

  And it is genuinely because there are more interesting things that we think are going to work than we can work on in a traditional, like, artisanal venture sense.

• 32:35Speaker 02

  And shameless plug, applications will open in January.

• 32:40Speaker 03

  Do you have like press a button to request?

• 32:43Speaker 01

  So fancy.

• 32:44Speaker 06

  Cool. Thanks for the talk, it was awesome. So I work for a series C enterprise focused company called Rider. And one of the interesting things about the multimodality thing that we're seeing in the enterprises, beyond vision, we're not actually seeing a lot of like demand for multimodality. Like we'll get asked about audio and video stuff, but then when we ask like sort of what's the use case? It's sort of like, uh, no. And so I'm curious if you and your portfolio companies are seeing that in the enterprise space? And if so, like what use cases? It seems very focused, like the multimodality stuff seems great for the consumer level. Curious if you're seeing anything on the enterprise side?

• 33:24Speaker 01

  I think it's a good call out. Enterprises, the data they have is mostly like it's text, it's like structured data and some sequel data. Like it's, uh, I don't think your average enterprise has that much vision, video, audio data that is that interesting. But I think that will change. Like, maybe it's because I'm like lazy and disorganized, but humans are very unstructured. Like they don't want, they don't necessarily think in terms of like relational database schema and like hierarchical management of their own information. And I think there's a future where we take that away from people. And the capture of information that you're going to use for different enterprise workflows

• 34:09Speaker 01

  enables more multimodal use, if that makes sense. And so like the, the sort of obvious example would be there are companies from like perhaps a half generation ago, like the gongs of the world that captured video and found some keywords and initial insights for sales reps. But the communications within an organization, the decisions made, the things that people create. I think there will be much more capture, especially of video, but.

• 34:41Speaker 01

  Making use of it requires companies to do that capture. So we kind of require this intermediate step, I think. But there's a company in our, and this is still a prosumer company today as well, to your point of like, you know, the consumer, prosumer side is ahead of the enterprise. But there's a company in our last embed batch called Highlight that kind of has this premise that like, okay, well, you know, we're going to use the multimodal modality by using on-screen capture. That's what this little like bubble is. On-screen and audio capture. And I think that, I think it's a powerful idea.

• 35:15Speaker 01

  Also, cut us off when you're.

• 35:16Speaker 07

  Yeah.

• 35:17Speaker 03

  Oh, yeah, yeah. Oh, by the way, just a quick check. Peter, Isaac, are you here? They're walking. You're walking.

• 35:26Speaker 07

  I think, yeah, there's sort of like a meme going around that the price of intelligence is going to go to zero. And you can kind of see this with GPT-4O. And with Gemini Flash, you can get a million tokens a day, which is probably enough for a small company, right? Like, so I'm curious how, as these large companies lose tons of money for market share, like, how are startups going to respond to this? Like, how does that change the market?

• 35:54Speaker 01

  Okay. I think it is impossible for anything to be too cheap. So I'll start with that. I would also say this company with this like awesome revenue chart, like, I'm pretty sure we paid like five to seven million dollars to a foundation model provider in this period of time. Right? And so, um, uh, demand is like, if there was like a secondary theme to this talk, demand is elastic in so many ways, especially for technology. And when you make things cheaper, we want things to be more intelligent. Right? And so if you make hundreds of calls in order to deliver an output, then suddenly like the fact that the cost of a call has come down 85% doesn't do you enough.

• 36:36Speaker 01

  And so, yes, it's like an incredibly compelling idea of like having intelligence too cheap to meter. I'm like, maybe this is a really old school of me, but for the last two decades, like, the internet and compute and software and data pipeline, like, they, it still hasn't been cheap enough, actually. We would do more if it was free. So, uh, the other like, uh, physical barrier that we've run into is, um,

• 37:02Speaker 01

  When models are really large, if you're not going to like quantize and distill and do domain-specific things, like it's hard to run, you need a lot of compute, just to state the very basics. And even with the foundation model providers, we are seeing people run into inference capacity issues. And so I do not know if this is true, but like one way to read anthropic pricing change is there's not enough capacity. Right? And so I think like incredible kudos to the open source ecosystem, incredible kudos to OpenAI for like staying on this drumbeat of offering cheaper and cheaper intelligence in every generation.

• 37:39Speaker 01

  But like we have companies that are spending a lot of money on, you know, let's say, search and validation systems with many calls, and we think that will continue.

• 37:51Speaker 02

  I think you can see that as well in like the the price charts that we had before. The like O1 pricing is still absurd.

• 37:58Speaker 01

  With love.

• 38:00Speaker 02

  Yeah. Yeah, but I mean, like, volume of tokens.

• 38:06Speaker 01

  Absurd. How could they?

• 38:08Speaker 02

  I think like it is really interesting that if you believe like the, I mean, the other part of this is like, if you look at the test time compute scaling, this is, it's a log scale. Like, it's easy to forget that, like, that's a lot of, like, historically, like, as a result of overtraining, a small set of companies took on the majority of financial burden for generating high-quality models, which is you just overtrain the shit out of your model and then it's useful for everyone else. If the customer has to pay this, like, that's a lot of money. If you want high-quality generation and that means that I pay on the order of like thousands of attempts, that's that's ends up being pretty expensive.

• 38:45Speaker 03

  Question from YouTube.

• 38:48Speaker 01

  Hi, YouTube.

• 38:50Speaker 03

  So, you know, you're talking about price, right? Price going down. There's also the other dimension of capabilities going up, and people always get steamrolled by OpenAI. So the question is, what are some specific ways that you've seen companies build to prepare for better models like GPT-5 or O2? Like, how do you future-proof that?

• 39:11Speaker 02

  So I think the most common refrain from at least OpenAI, but I think the model companies is you should build a company where you're excited when you hear that a new model is coming out, not anxious. I would have like one edit to this, which is like in the limit, it seems like the majority of things that are worth building today are actually, I don't know, should you hire a sales team at all if you think that models will be perfectly capable? Like one framing that I've thought about on this is you should decide like how much you believe foundation models will improve on like some core learning or intelligence capability and then build your company imagining that on that prediction. So the like an example here would be like if you take like, I think there's a generation

• 39:56Speaker 02

  of these like copyrighting companies that were largely subsumed by ChatGPT. And the story for many of them was the original usage was they understood better than other people how to get the model to like learn what my intent was in generating some piece of content, some piece of SEO content, or they understood how to ingest information about my business. And it's not hard to imagine like the next generation of models are just natively better at this, like the context length gets longer, you can stuff more into the context length, you can crawl and like learn more about external websites, like all that is like relatively cheap. And so if the core thesis of the company looks like we don't think models will be capable of doing that, that feels likely short sighted. On the other hand, like there are a number of delivery mechanisms that are like far out of range of what what models will do.

• 40:41Speaker 02

  Sarah had a good example of this, which is like there are some businesses where the limiting factor is like not actually intelligence, like the the limiting factor for a number of businesses is like access to a specific set of people or like, I don't know, we work with a pharmacy services company where like a core question is like, long term, can you negotiate pricing contracts? The core issue there is on intelligence, you need some amount of scale and then the ability to negotiate contracts. So I think I think many businesses are not exactly just a function of your ability to efficiently compute some small set of things.

• 41:10Speaker 01

  I give this presentation with Pranav and I'm like, oh, I'm so biased. It just sounds like startups are going to win everything. We still, I like to play this game, which is what investment decision do you regret from the past year? It's a really fun game. So fun. Yes. But one of the decisions that I regretted was actually a company that operates in a space that feels very core to perhaps foundation model companies and to hyperscale software players, where there's tons of ecosystem risk around the company. And by the way, the people are amazing, the metrics were amazing. We're just like, oh, they're going to get crushed.

• 41:50Speaker 01

  And so with everything I said, I still like overestimated the incumbents' ability to compete and make aggressive strategic decisions. And so I think it's like really hard to overstate how important it is to understand. Somebody can steamroll you if they focused all of their effort and all their best people on a particular area. Are they going to? Right? The copywriting example is illustrative because it's just not hard to see that understanding the context of a business from its website and from a couple of documents and by making prompting a little bit easier and adding like some buttons that replace some prompts

• 42:34Speaker 01

  or doing suggested queries, like it's just not a lot of work. Right? But there are things that are a lot of work, like having taste in developer products and distributing something amazing. And so I actually think that it's, if you ask me, like, we have to make predictions in this business. I worry more about underprojecting capability than I worry about overprojecting, at least in the short term. And then I worry more about expecting too much from the incumbents and being too afraid of them than being not afraid enough. Maybe it's just one investment regret.

• 43:13Speaker 03

  That's right. Either one of you. Yeah.

• 43:17Speaker 08

  We have one more from online. Oh, okay, you can do it. Uh, has to do with hardware. I'll just shout. Uh, how do you see AI changing hardware or in what ways? And for example, do you see a new Apple coming out and transforming hardware to that level? Not specifically the human situation, if they're trying to pass through general, how AI?

• 43:42Speaker 01

  Uh, okay, I'd approach this from two dimensions.

• 43:51Speaker 01

  Everybody, every investor wants like a new consumer hardware platform to exist because it's so valuable. And the question is like, why, why should it? I can think of two very good reasons. One is that the usage pattern that you can imagine for AI applications actually requires you to, like, the specs you'd want are different, right? Like, what if I want to capture image or video 100% of the time? And that's like a determinant of my battery life, of my sensors, of how I manage my network, et cetera. What if I want to run local models all the time? Like, maybe like most of the phones should be a GPU, right?

• 44:31Speaker 01

  I don't, I, I think that the usage patterns are perhaps very different for the next generation of, you know, the, the intelligence in your hand. I think it's a hard thing to pull off. Another reason that you could believe in a new hardware device is that the advantages of the existing consumer platforms go away, right? And so at the extreme, like,

• 44:56Speaker 01

  should you have individual applications that track a single habit? Like, drink water today, Sarah. Like, I don't know, like, I can generate that pretty easily now. And like, maybe the single function applications that live in the mobile phone ecosystems are part of a more general intelligence and they, like, that ecosystem is less important. And, and so I, I think there are different arguments for this. And like, we continually look for opportunities to invest here. I don't think this is exactly what you asked, but I also think the, like, there are.

• 45:37Speaker 01

  We invested in a company this past year that is doing robotics. I, for many years at Greylock, my prior firm, like, thought of robotics as an easy way to lose a lot of money over a long period of time. And, and like, I think that is true when you look at the outcome set for classical robotics, even for the companies that got to scale of distribution for an industrial robot or a single-use consumer robot. But, like, it's really cool that algorithms and generalization from the broader machine learning field seem to apply here as well. And so I think being imaginative about what physical intelligence looks like is also something we're excited about.

• 46:21Speaker 04

  So, related to agents, I think everyone has been chatting about agents, you're seeing more agent usefulness in production, but I'm more curious, like, at the infrastructure layer, what agent, what infrastructure primitives do you think are required for agents to actually work and continue to work in production?

• 46:42Speaker 02

  Okay, I, we've talked about this a little bit, I'm not sure if our points of view on this are the same. I think it is, I think it's really hard to tell. My suspicion is that, like, if you look at the number of, like, true agents that work, like, the number roughly rounds to zero, maybe it's, like, low single digits or low double digits now.

• 47:02Speaker 01

  Double.

• 47:03Speaker 02

  Doubles.

• 47:04Speaker 01

  Yeah.

• 47:04Speaker 02

  And, like, they're, they're all, like, relatively recent. I would say, like, beginning of this year, we saw, like, a bunch of agent framework companies and, like, I, like, I empathize with, like, the root of the question, which is, it's just really hard to tell what any of these companies need, especially when, like, the set of companies that works really well is unclear. And I, I think there's a lot of valid anxiety on what foundation model companies want the interface to be. Like, the computer's interface is a pretty low-level one. Like, the Anthropic version is, like, actually just makes specific clicks and, you know, like, rumors of other interfaces are, like, much more general, like, they're take actions on a specific webpage, or, like, entire browser environments. And so, like, at a high level, like, I imagine that there are sets of, like, there's, like, the full scope of tools, which is, like, I worked on a search engine for a while, like,

• 47:47Speaker 02

  crawl seems pretty useful, live data seems pretty useful, like, an API that looks something like... Here's a URL, give me the set of data that's available. Or here's a URL and a user login. Let me take some action on this page seems pretty useful. And then I don't know what the right place to operationalize this and commercially develop a product are. If I had like, if I was building a company here, like one thing that, I think it's useful to just remain agile. Like the core set of infrastructure is consistently useful. Like a crawler is consistently useful. And then one day you can figure out how to expose this better. But I, I like empathize with the difficulty of like, it's really hard to know what works for a bunch of agent companies. And my suspicion is like, the most successful agent frameworks will come from the most successful of these agent companies

• 48:32Speaker 02

  that solve these problems in-house for themselves and then operationalize this externally. Like it's some version of like, React is really useful because React was like well-adopted at Facebook for a while.

• 48:45Speaker 01

  I think we can say that there are like missing components in the ecosystem where that if there was a default, lots of agent developers would use it, right? And so like, identity and access management is a big problem. Like, if you could make agent development feel more like traditional software development, I think a lot of people would use that and be like, oh, like, you know, it magically retrys until it gets something and then it gives me like data back about how well it's working. Like things that, that like, I think it's pretty easy to actually imagine the utilities and the abstract that would be useful to the ecosystem. And then the entire environment is fluid, right?

• 49:25Speaker 01

  And so, do you need, like, if you think about other things in infrastructure, like, will more workloads need vector indices? Yes. Like, what is the shape of company that gets to be durable here? Like, we don't know yet. And we'll, we'll keep looking at it. But as Pranav said, I think we look to the handful of companies in our portfolio that are agents working at some scale and, and like look for the patterns there versus try to intuit it right now.

• 49:53Speaker 02

  My cash hit was wrong. I should have updated. It's a, it's a dozen, not a small number.

• 50:00Speaker 01

  It's been a long six months, guys.

• 50:03Speaker 03

  I think one last question, then there's a whole bunch of online stuff we won't get to, but.

• 50:08Speaker 01

  Mark.

• 50:12Speaker 05

  It seems like there should be more consumer companies. Like, why, why aren't there? Or is it just a matter of time?

• 50:20Speaker 01

  I think simply a matter of time. Like we keep bringing people into embed. We keep looking. I think I genuinely, this is not a knock on the research community or the really young set of founders that like, I think focused on AI companies first. But the diffusion of innovation curve that applies to customers, I think also applies to entrepreneurs. Researchers saw the capability first and they're like, like we should do something with this. This is going to be amazing. And it's like that will continue to happen. Like our portfolio is heavily overly represented with, with people from the research community, pushing the, pushing the state of the art with creative technical ideas.

• 51:04Speaker 01

  I think young, very young people also were quite early to AI because they're like, oh, of course. Like this makes sense. I've never seen other technology like ChatGPT all the way. And their opportunity cost is lower than like, you're the best product person at an amazing product organization. Like you have to leave your job to start a new company. And it's been a really long two years. Like, I feel like that's just started to happen where some of the talent that has the, and you know, maybe, maybe it's just like the next Zuck. You know, there's some dropout that figures out like the pattern of social interaction and is like really AI native about this stuff. I also think there's a chance that some of the people who have built intuition

• 51:49Speaker 01

  for consumer adoption and consumer interfaces, they're just taking a little bit to also build intuition for AI products. And now they're showing up and starting companies and experimenting. And so we have a lot of confidence, like it is going to happen over the next few years. It's a matter of time. Okay. I think we're, I think we're out of time. Just trying to defer to Sean here, but thank you so much. You know, please call us.

• 52:14Speaker 03

  Thank you. That was an easy one.