Ep 85: Has AI Infra Stabilized, FM Vibe Shift, & What's Next for Coding Agents

This episode is a wide-ranging conversation between Jacob and Swyx (Shawn Wang), an AI engineer, podcaster, and now operator at Cognition, who sits at a uniquely informed intersection of builder, investor, and community organizer in the AI world. The two cover the current state of the AI engineering zeitgeist: from the stabilization of agent infrastructure and the surprising stickiness of Claude Code, to the competitive dynamics of the AI coding wars, the rise of open models, the threat to traditional SaaS, and the frontier questions around world models, memory, and what it actually means for AI to "understand" something. The episode is grounded in practitioner-level candor, with Swyx offering real takes from running AIE conferences, working inside Cognition, and thinking deeply about what the next wave of AI-native software development looks like.

• 0:00Speaker 00

  Isn't that crazy? That number is just mind boggling.

• 0:03Speaker 01

  What is the state of the AI coding wars today?

• 0:05Speaker 00

  We're in a phase of sort of like capability exploration. The general thesis that I have been pursuing now is that the same way that 2025 was a year of coding agents, 2026 is coding agents breaking containment to do everything else.

• 0:16Speaker 01

  Do you worry about the foundation models just eating into a bunch of these startup categories?

• 0:21Speaker 00

  Midsize startups, yes.

• 0:23Speaker 01

  What do you think the end state of this market is?

• 0:25Speaker 00

  For the market structure to significantly change, there would be.

• 0:28Speaker 01

  Today on Unsupervised Learning, we had a fun episode in what's really become an annual tradition, a crossover episode with our friends at Laten Space. Swix and I sat down and we talked about everything happening in the AI ecosystem today, what we thought of the various changes at the model layer, what's happening in the infra world, the coding wars and a bunch of other things. It's a ton of fun to do this with someone I really respect and another great podcaster in the game. But without further ado, here's our episode.

• 0:55Speaker 01

  Well, Swix, this is a super fun to be back with another Unsupervised Learning Laten Space crossover episode.

• 1:02Speaker 00

  Yeah.

• 1:02Speaker 01

  I feel like a lot of places we could start, but you know, one thing I always find fascinating about the way you spend your time is you obviously are like at the epicenter of this engineering movement and community. And you run these events and conferences and put on these awesome talks and I think just have a great pulse on the zeitgeist of what's going on.

• 1:16Speaker 00

  Yeah.

• 1:17Speaker 01

  Maybe to start, just what are the biggest topics people are thinking about right now?

• 1:21Speaker 00

  Yeah, so I just came back from London where we did AI Europe and we're doing roughly one per quarter now.

• 1:26Speaker 01

  Yeah, you're really up the pace.

• 1:29Speaker 00

  It's trying, we're trying to match AI speed.

• 1:30Speaker 01

  Yeah, exactly. The topic will be completely different, I imagine.

• 1:33Speaker 00

  I definitely curate the tracks. Like you can see what I think when you see the track lists and the speakers that I invite. Obviously, OpenClaw is like the story of the last four or five months. And then just below that, I would consider harness engineering, context engineering to be two related topics in agents and RAG. And then there's a long tail of evergreen stuff like evals, observability, GPUs,

• 1:59Speaker 00

  L M infra and just in general. We also have other updates on like multimodality and generative media, let's call it. But I definitely. The first three that I mentioned are top of mind.

• 2:12Speaker 01

  Yeah. I think harness is particularly like so interesting. There was this tweet from Harrison Chase, the Lane Chain CEO, that caught my eye recently where he said, you know, it finally feels like we have stability around the infrastructure for, you know, around AI. And I think what he basically was implying is like, look, over the past two, three years as a company at the epicenter of AI infrastructure, it was a bit like playing whack a mole, right? You were constantly moving around with however the building patterns were evolving.

• 2:36Speaker 00

  For Harrison, for sure, right? Like, he's basically had to reinvent the company every year since he started LangChain, right? It was LangChain, LangGraph and all deep agents. And like, I think he's like one of the most nimble, adept, sharp people about this.

• 2:48Speaker 01

  Yeah. But he's saying now, now is finally the time for stability.

• 2:51Speaker 00

  Yeah, yeah.

• 2:52Speaker 01

  Do you buy that or what have you kind of make of that take?

• 2:55Speaker 00

  Hmm. I think that it's very expensive to say this time is different sometimes. But when you're just writing code, like, it's actually okay to just like try to make a call. And I think it may not even matter if this call is right or not. Like, I just don't even care that much because you can be right on the thesis, but if you don't figure out how to monetize the thesis, then who cares if you said something first? That said, it does feel like, for example, we went through a lot of different ways of packaging integrations up with agents. And it feels like we've landed at skills, which is like the minimal viable format,

• 3:38Speaker 00

  which is just a markdown file with some scripts attached to it. And I don't see how it can be more simple than that. And so there is some justification for the stability around harnesses. I feel like there may be more adaptation with regards to maybe like the real time elements or subagents or memory or any of those like agent disciplines, as it's called in agent engineering. But if the thesis is that, okay, you just want, agents are LLMs with tools in a loop, with a file system, what they can do, retrieval with skills and all these like standard tooling that now seems to be relatively consensus, then probably that makes sense. I just think like.

• 4:23Speaker 00

  There's no point trying to stake your reputation on this thesis that we're there. Because if it changes again, just change with it. It's fine.

• 4:33Speaker 01

  Yeah, I've always been struck by how that is much more challenging for infrastructure companies and application companies. Obviously, I think on the application side, you've seen Brett Taylor from Sierra, Max Unstrum from LaGora. They're like, look, we build what's ahead of the models and we're willing to throw everything out every three months as the models get better and better. But the thing you at least have there is you have an end customer, right? That's decently sticky. They will mostly stick, they'll give you a shot at least of building these things. What I've always found more challenging, they kind of reinvent yourself every three months at the infrastructure layer. It's like developers are definitely a pickier audience maybe than an accounting firm or a bank. And so it's definitely a more challenging position to be in to have to constantly reinvent yourself.

• 5:17Speaker 00

  Yeah, and when they churn, it's very complete. They'll leave to the hot new thing because there's no defensibility, I guess. Even if you are a database, people can migrate workloads off databases. It's a known thing. So I think basically what we're talking about is the vertical versus horizontal debate in AI startups. And the way I think about it also is just that when you're LaGora, when you're a bridge, you are the outsourced AI team, right? Your job is to apply whatever state-of-the-art AI methods.

• 5:55Speaker 01

  Yeah, there's translation layer between model capabilities and your end customers.

• 5:58Speaker 00

  To the end customers and like, well, if they didn't have you, they would have to hire in-house. And they're not going to hire in-house, so they have you. And I think that's a reasonable, very robust to any whatever trends and discoveries that people make in the engineering layer. I do think there is sort of useful horizontal companies being built, but they're all very much like the reinventions of

• 6:25Speaker 00

  Classic cloud in the AI era, and the primary one being sandboxes.

• 6:29Speaker 01

  Yeah.

• 6:31Speaker 00

  Which, like, it's another form of compute, guys. Like, let's not get too excited about it. But I mean, the workloads are enormous.

• 6:38Speaker 01

  Right.

• 6:38Speaker 00

  Yeah.

• 6:39Speaker 01

  It's interesting, and I feel like as part of this, you know, the questions that folks are asking around infrastructure, there's a lot around, you know, the extent to which companies should have their own AI teams and what they should be doing in-house. And, you know, I think there's questions around should people be training their own models? Should people be doing, you know, RL in-house based on the data they have? I feel like, you know, one has to evolve their takes on this every, every three months with Paces, but where are you at on this today?

• 6:59Speaker 00

  I think, actually, all models have gone up. And obviously, I'm involved in cognition and also curseurs, doing, doing a lot of own model training. And I think that that is some part of the, what I've been calling the agent lab playbook, where you start off with the state-of-the-art models from, from the big labs, and you specialize for your domain. But once you have enough workload and enough high quality data from your users, then you can obviously train your own models and like save a lot on cost and latency and all that, all that good stuff. You also get like a marketing bonus of like calling it some fancy name and putting out some research.

• 7:38Speaker 01

  From my seat, I can't tell how much of it is like actual, you know, value that's provided to the end user and how much of it is that marketing bonus, right? It seems some combination of the...

• 7:45Speaker 00

  I think it's both.

• 7:46Speaker 01

  Yeah.

• 7:47Speaker 00

  No, no, there actually is real value. And you know that for a number of reasons. Like, one, even when it's not subsidized, people do choose it as like one of the top four or five. This is both composer two and sweet 1.6. I want a top five models, like, you know, in a fair market, in a free market.

• 8:05Speaker 01

  Yeah.

• 8:05Speaker 00

  In a, in a model switcher, people do choose it. And like, it's not subsidized. So like, so that's as good as it gets. But beyond that, like, domain-specific models, for example, for search with, with both, which both companies have, absolutely makes, makes a ton of sense. Everyone says like, yeah, we should always, always do this. And honestly, like, I think the infrastructure for that is becoming easier with, like, Thinking Machines Tinker thing, as well as Prime Intellect's lab stuff. Yeah, I mean, like, this is one of those like reversal of the, the bitter lesson where you first bootstrap on the large models and the general purpose models to get big. And as you get very well-defined workloads that are just high-quality but not high-variants, then you just distill down to a smaller model and run that on your own.

• 8:49Speaker 01

  Right.

• 8:49Speaker 00

  Which like totally makes sense.

• 8:50Speaker 01

  What I'm less clear on is the kind of DIY RL use case, which I think is really mostly around, you know, improved quality for different things. Obviously, there's probably like more efficient ways to, you know, get a smaller model that's faster and cheaper. And it'll be interesting to see whether, you know, obviously, you had, you know, two, three years ago, this whole case of companies that were, you know, pre-training and claiming better outcomes in their domains than getting kind of cooked as each model iteration improved. You know, I wonder whether that's a similar story plays out in the RL space.

• 9:20Speaker 00

  Yeah.

• 9:21Speaker 01

  For the focus on pure outcomes and quality, not the cost side, which clearly your own models for cost at scale makes a ton of sense.

• 9:28Speaker 00

  I think there are two sides of the same coin. Like, you basically always want to hold quality constant or trade off a little bit of quality for a drastic decreasing cost. And that's true for everyone. One element I wanted to bring out, which is very much in favor of open models, is custom chips. So this would be Cerebras, but also Talus. Then there's a huge range of stuff in between. This has been a huge story this past year on just like everything non-Nvidia is getting bid up, including like freaking MathX is working, which is very rewarding for me. But I think one of those things where like, oh, like the suddenly because the number of alternative hardware is increasing and

• 10:11Speaker 00

  the inference that you can get is insanely high, like we're talking thousands of tokens per second instead of less than 100. So the trade-off for quality doesn't hold as much anymore because the speed is so high.

• 10:24Speaker 01

  Have you seen a lot of companies go all-in on the alternative chips?

• 10:26Speaker 00

  So Cognition has on Cerebras, and so has OpenAI.

• 10:31Speaker 00

  And so, no, I don't think so beyond that.

• 10:35Speaker 01

  But you think that's like a foreshadowing of what's to come?

• 10:38Speaker 00

  Yeah, I used to be kind of a skeptic in terms of like, okay, so what if I get my inference at 100 tokens per second sped up to 200 tokens per second? It's only 2X faster, it's not that big a deal. But when you, I think every 10X does unlock a different usage pattern. And we have proof in Talos and some of the others that you can actually drastically improve inference speed. And what happens from there, I don't even really know. Like, it's so hard to predict when entire applications just appear at once.

• 11:09Speaker 01

  Yeah.

• 11:10Speaker 00

  And it also isn't that expensive, right? So like, this is one of those things where like, I think that the investment cycle is gonna be multi-year. And I would caution people to not dismiss it too quickly.

• 11:25Speaker 01

  Yeah, I mean, one other like infra question I was curious to get your thoughts on is, obviously, it seems increasingly a lot of the cutting edge infra companies are building for agents as the buyers of their product or users of their product, right?

• 11:35Speaker 00

  Ooh.

• 11:36Speaker 01

  And...

• 11:37Speaker 00

  Another huge theme, yeah.

• 11:38Speaker 01

  And I'm trying to figure out like, what, what do you have to do differently about selling into agents? Are they just the ultimate rational developers, or is there?

• 11:46Speaker 00

  No, absolutely not. I think they are easily prompts-injected and very tuned towards like, basically compounding existing winners.

• 11:57Speaker 01

  Yeah.

• 11:57Speaker 00

  So like, if, like, congrats if you won the lottery for getting into the training data

• 12:02Speaker 01

  Right.

• 12:02Speaker 00

  before 2023, because now you're like installed in there for the foreseeable future. But yeah, you know, one stat that Versale CTO Malta Ubel dropped at my conference was that there are now 60% of traffic to Versale's like app, like admin app architecture for like configuring Versale applications is bots, it's not, it's not human. So like, your primary customer is agents now. And it's mostly like, mostly coding agents, mostly people using CLI, MZPO, whatever. But yeah, I mean, I think, I think step one, if it doesn't exist as an API that agents can use, it doesn't exist.

• 12:40Speaker 01

  Right.

• 12:40Speaker 00

  Right, which I think is like... It's a good hygiene thing anyway to make everything API available. But now it's like an extra push on like products, people to not only work on the UI. You should probably work on the on the CLI stuff. Beyond that, I think honestly, there is like, so I come from the sensibility of, I think, everything that you are trying to do for agent experience now, which is the term that Matt Billman at Nellefy is trying to coin, is the same thing that you should have been doing for developer experience, that you should have had good docs, you should have had a consistent API that is mostly stateless. You should have, I guess, discoverable or progressive disclosure or like search or like whatever.

• 13:22Speaker 00

  And so now that people have energy in like finding these customers to do that, that's great. Do I believe in extending beyond that into something like AEO for gaming the chatbots? Not necessarily, but obviously there's going to be huge advantages from people to figure out the short term wins and short term wins can compound.

• 13:42Speaker 01

  Do you think these compounding advantages to like the the pre-training data cutoff company is like, you know, obviously, over some period of time, I imagine that doesn't persist. And so as you think about like, I don't know, three, four years from now, what the selection criteria end up being, do you think it still mirrors exactly what you were saying before, like it's exactly what you should have been doing all along to sell a good product to developers?

• 14:01Speaker 00

  It could be, except that I think in three, four years, we'll probably have much better memory and personalization. So then general AEO or GEO doesn't really matter as much. So I think whatever memory or personalization system we end up with will probably determine what you end up choosing much more than than what is currently the case, which is just frequency of mentions, let's call it.

• 14:25Speaker 01

  Yeah.

• 14:26Speaker 00

  So you just spam quantity. And I think that's, I mean, that's something I'm looking forward to. I do think like, you know, I think that the fundamental exercise to work through for yourself is if you start a new sort of disruptor company now, there's a there's a big incumbent that everyone knows, like like Superbase, Superbase is like kind of like the Postgres like database incumbent. If you want to start like new Superbase, how would you compete with them? And I don't necessarily have the answer, but I do think like people like resend, like relatively new, I think they were starting like 2023. And it's still, there was there was a recent survey where like people. Checked what Claude recommends by default. If you just don't prompt it with anything, just say, give me an email provider. And says resend as in like seventy, seventy percent of cases.

• 15:09Speaker 00

  Like the fact that you can get in there with like such a relatively short existence, I think is encouraging.

• 15:14Speaker 01

  Yeah.

• 15:14Speaker 00

  I do think like you do want to do whatever it is to, to, like, to, to get in. That very short mentions this because it's not going to be twenty of them. It's going to be like three.

• 15:26Speaker 01

  No, definitely. It feels like, uh, you know, probably more, more consolidation than ever, uh, or, or kind of like, you know, uh, a winner take most market than maybe the, the, the physics of go to market in the past might have enabled.

• 15:38Speaker 00

  The other thing also is like semantic association is going to be very important. In a sense that like you want to do like the combo articles where you're like, use my thing with for sale with blah, blah, blah. And like that all gets picked up in a, in a corpus. And so that's probably one thing that you, you want to do well. I don't know what else. It's, it's, it's, it's one of those things where like, I think, I feel, I feel I'm behind. Uh, I don't know how you feel about this, but like.

• 16:04Speaker 01

  I think AI is just everyone constantly feeling like they're behind some, uh.

• 16:08Speaker 00

  But yeah.

• 16:09Speaker 01

  I want to meet the person that doesn't feel behind.

• 16:11Speaker 00

  But like with, with AX, right? Like, so, so like my, my stance was exactly what I said before, like. Everything that you, that you should do for agents is something that you should have done for humans anyway.

• 16:20Speaker 01

  Yeah.

• 16:20Speaker 00

  And so to the extent that you're just getting it more energy to, to do things for agents, great. But like, uh, it's hard to articulate what new thing apart from just like more spam, um, that you should be doing anyway. That would be my take right now. Um, I, I, I do think like there, there will be more turns at this. I think the personalization turn that is coming. Will be big and I don't know what that looks like because like basically we're kind of, we, we feel kind of tapped out on the memory side of things.

• 16:49Speaker 01

  Yeah. I guess since we last chatted, you know, you, you took this role of our cognition. And you've obviously have a, have a front row seat to the AI coding space today. You know, I feel like coding in many ways, you know, people view it as this like, I mean, besides being like the, the mother of all markets and this massive opportunity. I think it's kind of a preview of like what's to come for many other spaces, both.

• 17:08Speaker 00

  Yeah.

• 17:08Speaker 01

  You know, I feel like agents are most advanced in coding. I also feel like the, you know, competition between foundation models and application companies, you know, and, uh, mirrors what we may see in other spaces. And so maybe for our listeners. Can you just lay out like, what is the state of the AI coding wars today?

• 17:25Speaker 00

  It is massive, right? Like, and I don't think necessarily last time we talked about this, we appreciated the size of what.

• 17:31Speaker 01

  I wish we did.

• 17:34Speaker 00

  The state of AI coding wars today. Both OpenAI and Anthropic have made it their P-cero to competing coding. And Anthropic is at like 2.5 billion in ARR just from Claude Code. The way they recognize ARR is up for debate. OpenAI, I don't think that a public number is known, but let's call it 2 billion as well. And then Cursor is like rumored to be 2 billion, you know, and those are like the public numbers that are known. So like huge markets that have just been created in the past one year. Like, like Anthropic just, like Claude Code just recently celebrated their one year anniversary, which is.

• 18:17Speaker 00

  So and then I think like the other thing that I see is there's there's some other people who are like, oh, here's like the sort of relative penetration of Claude use cases, right? Like, and it's like coding 50% and then legal, whatever, health, it's like the remaining ones. And there was a very popular tweet that was like, okay, look at the the empty space and all these other use cases. If you are a new founder today, you should be betting on the other stuff because on a sort of catch up theory. And my my pushback is the same pushback that I had on Apple versus Google, which is like, well, why is this time different? Like, why if it went from, let's say, 10 to 50% in the past year, why can't it keep going?

• 18:56Speaker 00

  And like getting that wrong is actually a very painful one because you could have just did the momentum bet instead of the mean reversion bet. So I think that that is the state of things now that people are very much into psychosis. They are getting rewarded for spending more rather than spending less. And I think we're not in that phase of efficiency. We're in a phase of sort of like capability exploration. So I think people who are more crazy, who are more creative, get rewarded comparatively.

• 19:27Speaker 01

  Well, it's interesting. I mean, it feels like behind these like token maxing leaderboards and whatnot is this. It's like the first phase of this transition from a workforce perspective is you just got to show your employer like, hey, I use these tools.

• 19:37Speaker 00

  Here's my number of tokens I cost. And that's it. They don't care about the quality right now. It is uh maybe distasteful to someone who cares about the craft and and all that. Um but directionally, everyone just wants you to go up regardless. And so um there it's not very discerning. It's and it's probably very sloppy, but I think it's net fine because we're still probably underusing AI just in generally.

• 20:00Speaker 01

  Yeah.

• 20:00Speaker 00

  Um. And so I think that's like very interesting. Like we had on the podcast, uh Ryan Lepoplo from OBI who spends a billion tokens a day.

• 20:07Speaker 01

  Yeah.

• 20:08Speaker 00

  Um. And that's for those counting home. It's like something like 10,000 worth. $10,000 worth a day of API tokens if they did market rates. Um. And like most of us can't afford that. But like a- and and probably a lot of what he does is slop.

• 20:25Speaker 01

  Right.

• 20:25Speaker 00

  But like he's going to this. He's like, if there were a new capability, he would discover it first before you. Because he was he was trying and you were not trying.

• 20:33Speaker 01

  Right.

• 20:33Speaker 00

  And like you only do things that work like, well, good for you, but like the the people who are going to discover the next hot thing are living at the edge.

• 20:41Speaker 01

  Right. And increasingly, living at the edge is just having the compute budget to like run these experiments. I mean, kind of similar to what living at the edge on the research side has always been, you know, it was constrained in many ways by the amount of compute you had to run these experiments. It feels similarly on the on on almost on the builder or like actualizing these tools now.

• 20:56Speaker 00

  The other thing that's, I mean, uh, very obvious is Anthropic is kind of like the high-priced premium player. Um, that where, you know, restricting limits or restricting model releases even is like the name of the game, whereas Codex is like, come on in, guys, use our SDK, use our login. We don't care. We're going to reset limits, whatever. You do want to try to exploit the subsidies where you can get it. And definitely Codex is super subsidized right now. Gemini also very subsidized. Um. And comparatively, like, I think you should make hay, I guess, while while that's going on. It's not that bad to be a capabilities explorer on just the $200 a month plan from Claude Code. Or from OpenAI. And my sense is that people aren't even there yet.

• 21:41Speaker 01

  How do you think this like market ultimately plays? I mean, it's obviously such a big market that, you know, any slice of that market is interesting for for anyone going after it. But I think what makes people so interesting in the coding market particularly is it feels like it's kind of this foreshadowing of what will happen in other, you know, any other kind of application market that the foundation models eventually turn to and are all their models against and gather data around. And so how do you think, you know, like, does there end up being room for lots of different kinds of players or like, what do you think the end state of this market is? And is that, do you think that's applicable to other markets?

• 22:10Speaker 00

  I feel like there will be, I mean, status quo is probably the most likely outcome, which is there are two big players and there's a small range of longer tail people that fit other use cases that the two big players don't. That feels right to me. I think that for it, for the market structure to significantly change, there would be, there needs to be significant change in like the economics or like the the brand building or like the the value propositions of the of the companies involved. And I haven't seen any in the last six months that that have really changed the stories materially. So I feel like they will just keep going until something something else happens.

• 22:52Speaker 00

  Something else happens, meaning like Microsoft wakes up and like goes like, guys, we have GitHub, we have, you know, we'll do something much bigger here than other than just Copilot. And that would be a big change. MSL has put out a model now, and I was in a breakfast with Alex Wang where they were like, yeah, like, we really, really want to go after the coding use case. They haven't done anything yet, but like, don't underestimate them, right? And similarly for the Chinese labs, like, I think they're trying to go after it, like ZAI is doing stuff, GLM, yeah, GLM is the same thing. And so it's like, everyone's trying to get a piece of that pie.

• 23:34Speaker 00

  I feel like the status quo has been pretty stable for the past, like, almost a year, I'll say.

• 23:39Speaker 01

  Yeah. And is there room for the not, like, you know, for for the application companies more on like the enterprise side or like where do the, where do the like, what surface area do the model companies leave for application companies?

• 23:50Speaker 00

  Yeah, that's a good one. It's very much evolving. I will say because OpenAI did not have this level of attention on coding a year ago, we just don't have that much history, right? And it seems like, for example, so the big push at OpenAI now is the super app. Is that a consumer thing? Is that like a products like portfolio rationalization thing? How much is that going to take away attention from coding at the time when they actually do want to put more coding? I think it's very unclear. So I do think like there's all these like, in both big labs, there's, sorry, at both OpenAI and Anthropic and DeepMind and XAI are separate cases.

• 24:32Speaker 00

  They are trying to see the other TAM expansion areas, so cloud code for finance,

• 24:39Speaker 00

  cloud cowork, all those things. Whereas I think curse and inclination are like comparatively just focused on coding. And so I do think they leave space. And I do think for the other verticals, that also means the same thing, right? That they're not going to be that intensely focused on that domain. Except for, I think, I would mark out finance and healthcare as like the next ones that they're clearly going after. I would say comparatively, healthcare seems more thorny. There have been some announcements about it, but like I would respect the finance work a lot more just because like the path to money is a lot clearer.

• 25:12Speaker 01

  Yeah. No, I mean, obviously, like I think, you know, maybe similar to the space that's being left in these other domains. You know, there's obviously a lot that's required to actually implement these tools in enterprises versus, you know, maybe just giving them, giving model access to folks out of the box.

• 25:27Speaker 00

  Yeah, yeah, yeah. So the agent lab thing is like, we'll do the last mile for you. Whereas I think the model labs tend to just trust the model and be minimalist about it. Both of them work. I don't necessarily think one beats the other for every use case. All I do know is that it does seem like. The large enterprises do want a dedicated partner that isn't just the model labs, which is kind of interesting.

• 25:55Speaker 01

  We've been in this phase of pure capability exploration. And so I think nothing has been better for the large labs. I mean, they're always going to be at the frontier of capability exploration. And so I think have a very good relationship with a lot of these enterprises. But ultimately, over time, like the incentive structure of these labs is always going to be maximal token consumption for the end customers they work with. And there's just, I think, so few companies that have actually gotten to massive scale. Maybe coding, again, is the most interesting. It's the first space that really is just completely gone. You know, yeah, you must live it every day, like absolutely insane. And I think when you get

• 26:32Speaker 00

  Even, okay, I mean, like, I think we say good things about curse of condition, but the sheer liftoff of like both Anthropic and OpenAI, because they have independent valuations. I mean, let's throw in XAI in there.

• 26:44Speaker 01

  Yeah.

• 26:44Speaker 00

  Now, IPO-ing at 1.2 trillion. That number is just mind boggling. Like, I feel like in normal investing or normal startups, there's kind of like a ceiling market cap or valuation that like you reach and you're going like, all right, it's going to be chiller from now on. And these guys are not slowing down.

• 27:01Speaker 01

  No. Well, I also think the dynamic that's fascinating about some of these later stage companies is, you know, in the past, I feel like in venture world, if you got to a certain level of scale, the question around you was really more a valuation question. This is like why there was different phases, like, you know, types of venture people did. And like the late stage growth people were just incredible at like, you know, a little bit of what's the ultimate market opportunity of this company, but also what's the right way to evaluate it. Like, we know it's in some bands of an outcome that is like, sure, there's some variance to it, but it's like relatively understood what that bands is. And then maybe you get over time, surprise to the upside. Whereas any kind of like, even the labs themselves, any later stage company, the bands of which that company might be worth right now, even in a year or two years, are so massive because of how fast the ecosystem changes, that it's like, even for later stage companies, every

• 27:45Speaker 01

  three months could be an existential level event to the upside, to the downside. And I think that like, you're obviously seeing it in the positive with code, which, you know, if you think about a company like Anthropic, you know, For a while, it was unclear if they were going to have access to enough capital to really stay in the in the race, right? And then coding hit at the exact right time. They had the perfect model for it. They executed brilliantly. And now we're one of the most viable companies in the world.

• 28:13Speaker 00

  At the same time, I don't find I have zero sympathy for OpenAI because they're crushing it and they're all rich. You know, this is like a high class champagne problem to have to be number two at coding or whatever. Like, who cares? Like, you're doing great.

• 28:27Speaker 01

  Yeah, it's funny though. I can't even, I mean, you would be closer to this, you know, even though you're in the coding space. But it's like, a lot of people I talk to think Codex is just as good, if not better than Claude Code, right? I think one thing that I've been really surprised by, and maybe Claude Code is a better product in some ways. I'm curious your thoughts, is just in consumer AI with ChatGPT. You saw this big first mover advantage, right? Where admittedly today, like, I don't know, Claude Gemini, great products. Not sure, not abundantly clear ChatGPT is any better, but like, people stick with ChatGPT. It's the first thing to introduce them.

• 28:56Speaker 00

  They stay, but they're not growing anymore. I don't know if you've seen.

• 28:59Speaker 01

  Right, but that to me is more of like a product problem than it is, they're not, it's not like they've like lost share to someone else. My understanding is the overall problem with consumer AI today is much more of a, how do you take this tool? And, you know, for folks like us, like knowledge workers, it's like this incredible magic tool. But it's not necessarily a daily active use tool for a lot of people around the world today. And what are the like products? It's kind of a category wide problem. Like in coding, for example, like the entire space has gone parabolic. There may be some relative growth in in other consumer AI players. But it's not like consumer AI as a category is like going parabolic and they're not capturing most of that thing. I think it's actually, the larger problem is much more, hey, the category has kind of hit a bit of a plateau. People haven't figured out how to bring, you know, tons more users on board or increase the frequency of those users.

• 29:43Speaker 01

  And so it seems more of a category wide problem than it is, you know, a massive market share change. I was going to draw the comparison to the coding space where Claude Code was the first product, obviously, to introduce people to this magical experience. You know, by all accounts, Codex is pretty damn close to as good, if not better. But like, still, that first product, you would have thought that would not be a super sticky, you know, product surface area. And it actually has, it turns out, it feels like the first lab to introduce you to an experience really does keep a lot of the, a lot of the focus.

• 30:12Speaker 00

  I think. Maybe it's like still early days, you know, ChatGPT is like three plus years old and

• 30:18Speaker 01

  Yeah.

• 30:19Speaker 00

  Claude Code is only one.

• 30:20Speaker 01

  Just turned a year, yeah.

• 30:20Speaker 00

  So just give it time.

• 30:21Speaker 01

  Yeah.

• 30:22Speaker 00

  Like, yeah, I mean, definitely, a lot of people have switched from to Codex. Maybe that will keep going. It's like really hard to tell. Yeah, I do. I do think that because we are in this like high volatility, high temperature phase, the loyalty and stickiness to first movers and category creators, I don't think is as high as it might be in some other areas in our careers that we've looked at.

• 30:47Speaker 01

  Yeah, though I mean, I've been surprised by the Claude Code thing. I would have thought that like, in many ways, I always worried about the

• 30:52Speaker 00

  You think it would have been gone by now?

• 30:53Speaker 01

  Not gone, but I would have, I always worried that the, that the consumer business of these companies would be quite sticky and then the enterprise API business was actually like, you know, in some ways like your least loyal buyers, like they would, they would move to

• 31:05Speaker 00

  Right, right, but they worked out that wasn't the enterprise API, it was enterprise product.

• 31:09Speaker 01

  Totally. And maybe that was the, that was the secret that like, but the amount of lock-in or just default behavior that has happened in that space is, is more than I might have imagined with two products that by all accounts are pretty damn similar.

• 31:21Speaker 00

  Yeah, no fight there. I will say, I do think that Codex is still in like a catch-up, like in terms of personal experience. The only thing I like out of, out of Codex is like Spark and like, I feel like the skills integration is a little bit better. I feel like the speed is a bit better, maybe because it's written in Rust or whatever. Very minor things that you like, almost like telling yourself, rather than like objectively assessing between two of them. I, I do think like vibes-wise, I think that's going on. The, you know, I feel like the, the missing questions in, in this whole debate is like, why is it so concentrated in only two names, right? Like, where is the Gemini, you know, presence? Where is the

• 32:06Speaker 00

  XAI presence? And like, they are trying, it's just, they haven't made that much progress yet.

• 32:12Speaker 01

  But what the Claude Code moment does show, it actually in some ways makes you a little more bullish on the potential for someone else to catch up because it does feel like if you're the first person to introduce some magical net new product experience, that that actually might be stickier than one might have imagined.

• 32:27Speaker 00

  Right, right, right. Okay, yeah.

• 32:28Speaker 01

  And so everyone can believe they have a shot at that.

• 32:30Speaker 00

  What do you think that new product experience might be? Like, it's, it's like, and this is a failure of imagination on my part. Like, I always wonder, like, people always say this, like, well, the thing that will save us is like being first to the next new thing. Like, what is it?

• 32:41Speaker 01

  Yeah.

• 32:44Speaker 01

  I don't know, something around like a consumer agent, computer use, like hybrid. I think obviously, I think we're like scratching the surface on the consumer side.

• 32:53Speaker 00

  So my current theory is like the open claw is like a vision of things to come.

• 32:58Speaker 01

  Totally.

• 32:58Speaker 00

  And it's gonna good that opening I has like the association with open claw. But by no means that they have the rights to win it. The general thesis that I have been pursuing now is that the year, the same way that 2025 was the year of coding agents, 2026 is coding agents breaking containment to do everything else. And so coding agents continue to still win, but because they generate software and software eats the world. So like, it's kind of like the trans associated property of like software eats the world, coding agents eat software, therefore coding agents eat the world, which is like an interesting.

• 33:30Speaker 01

  Yeah, and breaking containment always an easier phase phrase in the consumer context than the enterprise one. You've seen people run these really cool experiments in their own personal lives. I think like

• 33:37Speaker 00

  Yes.

• 33:38Speaker 01

  Figuring out, you know, how you, obviously, everyone's focused, you know, on the enterprise side now around how you create these experiences. I feel like the vibes, you know, people love to have these narratives of like, everything is completely shifted. It's like, I actually, you know, open AI organizationally, you know, volatility aside is, you know, great products, great team, great models. Like, everyone else in the world is incentivized for there to be two, three, more, everyone would love more, like, great model companies. And so I feel like the, the natural forces of the world revolt when any one company, you know, is too much the star of the show, right? There's so many people in the ecosystem that are incentivized for that not to happen. And so I think I'd be shocked if we don't have a reversion of vibes, not maybe completely the other way, but at least a little bit more equal at some point over the next six,

• 34:23Speaker 01

  12 months.

• 34:24Speaker 00

  I think there's just a kind of different stages. When you talk about the world wants wanting more model companies, I think about like the Neolabs.

• 34:30Speaker 01

  Yeah.

• 34:31Speaker 00

  And I mean, I don't know, is it fair to say none of them have really broken through in the past year?

• 34:35Speaker 01

  I think that's totally fair.

• 34:37Speaker 00

  Which is rough.

• 34:40Speaker 00

  And well, how are we gonna, how are we gonna grow that diversity in choice? Like, that's, this is it.

• 34:46Speaker 01

  Yeah, it'll be really interesting to see what, what ends up happening with that. And you've seen, you know, folks like Nvidia, you know, very incentivized to make sure there's, there's a broader platform of other model providers.

• 34:57Speaker 00

  I think, I don't know, people say this, but I don't think they try that hard.

• 35:03Speaker 00

  Nvidia tries harder to build NeoClouds.

• 35:05Speaker 01

  Yeah.

• 35:06Speaker 00

  Than NeoLabs.

• 35:07Speaker 01

  Well, they try pretty damn hard to build NeoClouds, so that's.

• 35:10Speaker 00

  But like, you know, let's call it like the core weaves of the world's much happier place in, you know, than any NeoLab built on top of them.

• 35:18Speaker 01

  Yeah, though one might argue it's easier to enable a NeoCloud to be successful than it is. You can't will a NeoLab into existence the same way you can with a NeoCloud.

• 35:25Speaker 00

  More direct control over it, for sure.

• 35:27Speaker 01

  What else is kind of catching your eye today on the startup side? And you worry, there's obviously this whole narrative of like, you know, the foundation models, you know, they announced a product and every stock goes down 15% like.

• 35:36Speaker 00

  Yeah.

• 35:37Speaker 01

  Do you worry about the foundation models just kind of eating into a bunch of these startup categories?

• 35:43Speaker 00

  Not really. I think actually like as this, okay, there's the point of view of like being an investor in startups. And there's a point of view of like, do you want to start something? And I think honestly, like the downside for all of these is so minimal in a sense of like, the worst you do is you just get hired into one of these labs anyway. So I think the market for people who just do things and try things and try to execute in like a competent way, even if like it doesn't work out commercially, even if it just wasn't that great anyway. Like, but like, that's your job interview to go into one of these things anyway. So I don't feel that from a very, very small startups perspective. Midsize startups, yes.

• 36:26Speaker 00

  I would say there's been a lot of dead LM infra consolidation, like the Lang fuses at the world getting absorbed into the ClickHouse. And I think like people have maybe worked out the domain specific playbook. And like, I think that's okay. And yeah, I'm not that worried about. Okay, so I would say I'd be more worried about traditional SaaS, like low NPS SaaS. This is the whole AI versus SaaS debate that has been going on. And like literally, I'm going through that exact thing in my company. Where so I'm like kind of thinking through this on a very visceral level, right? On one hand, you have the people who say, you vibe coders don't appreciate the amount of work that goes into a CRM.

• 37:10Speaker 00

  And like, yeah, you think you can rip out Salesforce, so did the 30 entrepreneurs before you, right? Like, like, you know, you classically underestimate the things that you don't. Deeply know and talking to the audience is not you. At the same time, like, we have never been able to build software so easily and customize software so easily. And like, yeah, you're not gonna use ninety percent of the things it just was. So like, what's the? What have you done internally? So we have the main SaaS that we do for event management and sponsor management. And we pay two hundred K a year for that. Not huge, but like chunky for my scale. And like, yeah, I could probably spend two thousand and build like a custom version of that. The trick has been dealing with the rest of my team and getting them on board.

• 37:53Speaker 00

  Because I'm the most cynical person on my team, but like, I can't make that decision myself, right? Like, I think in the same way, I've been telling with other CEOs, team leaders as well. It's like, well, you can be super cloud-pilled, you can be super LM psychosis, and you think that's okay. But you have to bring your team with you. And I think like, the sort of widening disparity in LM psychosis in companies is causing real rifts. Because on one hand, the people who are less AI-native are not getting with the picture. They're not, they're actually like behind, they're actually not waking up to the fact that like, everything you think is necessary is not actually that necessary. And in fact, it would be better of you if you just like held your nose and went in.

• 38:38Speaker 00

  And when came out the other side, only talking to agents in natural language. And like, your life would actually be better and you're just like closed-minded. There's that perspective. The other perspective is, oh, you vibe coder, you did this in a weekend and you got the eighty percent solution. And now the rest of your employees have to pick up the rest of your shit. Right.

• 38:59Speaker 00

  That you thought you were such hot, amazing at, but like, actually you didn't figure it out. And like, actually LM's are still useless at this and blah, blah, blah. So like, I think there's this huge debate going on in every company right now. And like, you know, I have a small microcosm of it, but like, yeah, it's making me hesitate to pull the trigger. But like, I will at some point. It's like, maybe I've put it off for one year, but not like five.

• 39:24Speaker 00

  But like so like SaaS is definitely getting squeezed.

• 39:29Speaker 00

  It does make me wonder, like, I I do think that there is an opportunity for a more AI-native system of record thing that is not just Postgres or not just MongoDB, although both are very good. Maybe it's like a convex or like people bring a convex a lot. I don't know, like, like, I I just feel like the sort of quote-unquote "firebase" of of AI apps isn't really a thing yet beyond what we have, which which is fine, it's it's it's just, we could probably start in a more sort of rapid iteration cycle first before scaling up to like a Postgres or MongoDB, which are more sort of old tech. I was at a dinner with Mike Krieger, the CPO of Entropic, and and we were just kind

• 40:13Speaker 00

  of going around the room, going like, "What are people most worried about?" And for me, I instead of security, I brought up bio safety.

• 40:21Speaker 01

  Yeah, classic.

• 40:23Speaker 00

  Actually, like, I said it was cliche and classic, and the rest of the table were like, "What do you mean, someone sitting at home can manufacture a virus that wipes out half of humanity?"

• 40:32Speaker 01

  It looks like the OG Jeffrey Hinton, like, "This is why you should be scared."

• 40:35Speaker 00

  I'm like, "Yeah, like, read the, you know, risk reports, like, this is like the thing." I think, and Mike was just sitting there, knowing he was sitting on Mythos and going like, "Actually, it's security."

• 40:48Speaker 00

  And I think like, I think that there's there's part of it is very good marketing, like, too good. Like, I would actually advise Entropic to tune down the marketing, because also, it's it's just a very good model, and you don't have to make so many marketing claims around it. At the same time, it is not really a private model if you give it to forty companies, each of whom have like ten thousand employees or whatever, right? It's not it's not private. It's like, there's bad actors in there.

• 41:18Speaker 01

  Yeah, hopefully hopefully not as as bad as releasing it widely. But no, I mean, it's an interesting, you know, it's an interesting case study for how, I mean, many model releases might, I mean, you know, this might be the first model release that looks like the rest of them from from now on, right?

• 41:31Speaker 00

  It's there's an overall product strategy of for Anthropic of like bundle, you know, restricted access, bundle, product with model, maybe, whereas OpenAI has definitely been a lot more sort of philosophically aligned on like, we will just enable access everywhere and we don't know what you what will come out of it, right?

• 41:51Speaker 01

  Though, I mean, this current moment, obviously, the cynical take is also just ties to the amount of compute that both companies.

• 41:56Speaker 00

  Yeah, right, right. Yeah, I think I think that's true. I do think like the this is the the scale, the dawn of like larger than 10 trillion parameter models is very interesting. I don't think it, I think it's a temporary phenomenon because we have much larger compute clusters coming online for everyone over the next like three, five years. And this is like already written in the cards.

• 42:18Speaker 01

  Yeah.

• 42:19Speaker 00

  So to the extent that like, you know, will we have rationing of models above 10 trillion in like two years? I don't think so. I think everyone will have.

• 42:29Speaker 01

  We'll just have rationing of the next phase.

• 42:30Speaker 00

  Right, right. But like, that's as it should be almost like my my classic example, which I, this is just me theorizing, not anything confirmed by Google. When Google announced Gemini, they actually announced three sizes, which was Flash, Pro, Ultra. They never released Ultra. They only have Pro and Flash. So my theory is they have Ultra sitting in a basement and they just could distilling from it for for Flash and Pro. Which like, yeah, I mean, I actually think that's as it should be for any lab that they do that.

• 43:02Speaker 01

  Yeah, just because those are the models that people actually want to end up using and it's just like cost.

• 43:06Speaker 00

  It is more, yeah, it's cost. It's not the want, it's just just the cost. I do think like it is interesting that for a while I was I was considering the theory that models capped out at two trillion. And I think that's proving to be wrong. And Well, then, if I'm wrong, how wrong, how wrong am I? Do we do 200 trillion? Do we do 2 quadrillion, whatever?

• 43:32Speaker 00

  And I don't think we have the straight answer to that. But like, it's interesting that we are continuing to scale number per AMs. When everyone kind of like can see that we're not going to get like the next thousand or one million X from this paradigm. So like the others, like the Elys of the world are working on other model architecture improvements. We need a different scaling law, I guess, because like, I feel like people already feel like we're taut on this. Like, the end state of this is we turn most of the world into data centers.

• 44:05Speaker 00

  And like, I don't know, I don't know if we want that.

• 44:08Speaker 01

  Yeah, I mean, if the return of intelligence were there, maybe, maybe not so bad.

• 44:13Speaker 00

  I, I, I think there's just a sheer amount of like, like unscalability that like is wrangling people's sensibilities right now. Especially in terms of like context lengths. My classic quote is that context length is like the slowest scaling factor in LMS.

• 44:30Speaker 01

  Yeah.

• 44:31Speaker 00

  We, like, we took maybe three years to go from like 4,000 context length to a million, and that's about it. Like Gemini has had a million token context length for two years now. And no one's using it. Like, so, like, yeah, it's, memory is probably going to be the biggest limiting constraint on all these things.

• 44:50Speaker 01

  Yeah, certainly seems that way. I guess I'm curious over the last year since we recorded last, like, what's one thing you've changed your mind on?

• 44:57Speaker 00

  I feel like I was kind of bearish on open models. Like, last year. In a sense of like, I had just done the podcast with Ankur Goyal.

• 45:07Speaker 01

  Yeah.

• 45:08Speaker 00

  Of Brain Trust, where he, I mean, you know, he has a good cross section of all the top AI companies. And he says, market share of open source is 5% and going down. I think that's changed. I think it's going up. And even if

• 45:22Speaker 01

  Even though the capability gap does seem to be increasing.

• 45:25Speaker 00

  Depending on the It's hard to, yeah, it's really hard to tell. Because like, okay, for listeners, capability gap increasing is like on public benchmarks. And let's say you're comparing Methos versus like, I don't know, GPT OSS or like GLM 5.1. And it's really hard to tell because even if they were closing, you will also not believe that they were closing that much. Because it's very easy to game the benchmarks. So you just don't really, really know. All you know is like, there's somewhat objective open router stats on like what people choose in the free market. And people do choose some of these open models in significant volume. Except that a lot of them are heavily discounted. So you need to kind of like price adjust these things.

• 46:06Speaker 00

  So even if, even if that were true, which I'm not sure, like I feel like the number is just up now instead of down. I think the separation between what the top tier agent labs are doing versus the average startup in AI or the average GPT wrapper is significant enough that you should not worry about the sort of mean industry number. And you should, you should cohort things into like, here's the median, here's like the bottom 80% and here's the top 20%. And top 20% acts very differently than the bottom 80%. And so top 20% is, which is what I, all I care about is definitely going towards more open models. The fireworks and the together is a crushing.

• 46:48Speaker 00

  And, and so all the fine tuners, right? So like, I think maybe last time we even said things like, fine tuning is a service doesn't work. Well, now it's going to work. It's a derivative of the open market, open models market.

• 47:01Speaker 01

  Well, and also in the workload scaling to the point where people care about cost and speed, you know, more and more. And that like that, you know, moving from just pure use case discovery of like, what can these models do to, okay, we know what they can do at scale. Now let's do them cheaper and faster.

• 47:14Speaker 00

  Yeah. So, so like that change, I think is probably the most significant in, in my mind. And like, I always like to do the mental math of like, this is what I think about scheduling a learning rate. Like when you've been wrong once, what else were you wrong on? And I'm kind of working through it. I, to me, the, the other thing was the coding one. Which obviously I have now come full 360 on. But I think like people are not appreciating dark factories enough, which I don't know if you've discussed on the pod yet.

• 47:44Speaker 01

  No.

• 47:46Speaker 00

  And so this is a kind of a strong DM slash Simon Willis in term. The the general idea is, OK, there's different levels of AI coding psychosis you can have. The the very first level, which I I, by the way, I encountered first in cognition five months ago, was zero human written code.

• 48:05Speaker 01

  Yeah.

• 48:05Speaker 00

  Right, which like seems like a reasonable thing now was less reasonable five months ago. The next frontier that sounds as crazy today as it as as zero coding was in in the past is zero human review.

• 48:17Speaker 01

  Yeah.

• 48:18Speaker 00

  Like you just just check it in without even reviewing it. And very few people are doing that, but OpenAI is is exploring this. And I feel like it's it's definitely the only scalable way to do this, which it just means like you have to just kind of like flip the S D L C or change large amounts of what what you normally do.

• 48:38Speaker 00

  Which is probably things you should have done anyway, more testing, you know, more automated verification or whatever. But like, that is a frontier at which like when you have unlocked that in your companies, you are just going to produce much more quantity of software than than you've ever had. It's going to be like so much so disposable, so cheap, that you can probably innovate in quality a lot as well. Like that that quantity helps you get to quality.

• 49:00Speaker 01

  Yeah.

• 49:01Speaker 00

  Which I think people are very uncomfortable with because like people associate more quantity with slop.

• 49:07Speaker 01

  Right, now it's back to exactly the discussion we're having on like the reaction to these token maxing scoreboards and the and the idea that like today, maybe that's not the most, the the the best sign of of a product of any efficiency but going forward.

• 49:18Speaker 00

  Yeah, you but you still get rewarded for it, so like, fuck it, whatever. But like, I I I think like the the the people who are who are doing well, who do well, who do most well in twenty twenty six are not the cynics who go like, oh, that's just slop, I'm not going to participate in that. They're like, OK, like this is happening with or without me, let's bend this the right way.

• 49:36Speaker 01

  Yeah, no, I love that. I think for for me, like a kind of related thing on the open source model side is for so long, I really didn't think it made any sense to do any sort of RL, post-training, pre-training, anything you could do to like improve kind of overall quality, certainly for like latency and cost, it always made sense to me. But for overall quality, like, God, you just get that for free in the models like three, six months later. I think what I'm starting to change my tune on a little bit is, you know, hearing all these app companies talk about like, you know, we build stuff and then we throw it out three months later as like the models improve. You're like, okay, well then what you're doing for capability improvement is just another version of that, right? Like, I still don't think that like your RL or like post-train is going to make you have a better model for like

• 50:15Speaker 01

  years and years to come. But maybe, I think you still have to be pretty rigorous on like, is that the single best thing you can do to solve a customer problem? And like, you know, oftentimes like it's literally just like, now like add more data and like feed more data even via connectors to these models or like, I don't know, do some clever engineering on the backend or whatever it is. But if the single best thing you can do for that three month time period to improve your customers' outcomes is, you know, post-training in some way that like really improves the output of a model, even if you throw it out three months later because the general models get up there, it still might have been worth doing. And so I think I'm like more open to.

• 50:45Speaker 00

  You throw out the results, but you don't throw out the raw data.

• 50:47Speaker 01

  Totally.

• 50:48Speaker 00

  And like so.

• 50:48Speaker 01

  Right, then you just run it again. And so basically there's some, obviously at the level of cost of like $10 million, maybe that's too much, but there's some level of cost where.

• 50:55Speaker 00

  No, it's not even 10 million.

• 50:56Speaker 01

  No, of course it's not. You know, there's obviously some level of investment at which it's the equivalent of just like staffing four engineers to go build something for three months.

• 51:04Speaker 00

  Yeah. And so the other thing I really, for listeners, I'm just going to leave some droplets of info. Looking like the long trajectory, the synthetic Rubik's work that people are doing is very important, including something that's called Dr. GRPO. I'll just, I'll just leave those key search terms in there. I think what it means is that RL is going much more multi-turn than people think. And that means that you can customize the models in way more specific dimensions than traditional, let's call it SFT or

• 51:37Speaker 00

  you know, like a sort of shallow RL that was done in a year ago. So like hundreds of turns.

• 51:44Speaker 01

  Yeah.

• 51:45Speaker 00

  And, and, and I think that that leads you down a path of like complete domain specificity.

• 51:50Speaker 01

  What else like are you, you know, of these like unanswered questions in AI today, are you like looking for, you know, in the next year, are you, you, you know, paying close attention to?

• 51:58Speaker 00

  I have a few theses for like what is the sort of next frontier. One is memory, which memory and personalization we talked about. The other is really world models, which we've done a small little series on from Fei Fei Lee

• 52:10Speaker 01

  Yeah, of course.

• 52:10Speaker 00

  to even Moon Lake and General Intuition. And there's a lot of debate as to like the relative importance of this. I think a lot of it manifests as like 3D static walls that you kind of inhabit for a little bit and you walk around. And they're like, cool, but like, how does this help me with my B two SaaS?

• 52:29Speaker 01

  And it's like all the hype now is robotics, right?

• 52:31Speaker 00

  Yeah, and there's obviously a correlation between world models and embodied vision and experiences, which leads to robotics. But I think world models is very interesting in just in improving intelligence itself from the next from the next token prediction paradigm. And so I think people are kind of testing their edges around that. One of our top articles this year so far has been on adversarial world models. I do think like, if you don't do anything else, just read Fei Fei Lee's essay on spatial intelligence, on why L L M's don't need, don't have it. And she is, she may, she may not have the solution yet, but she has the right problem statement.

• 53:15Speaker 00

  And so everyone else is trying to solve that problem statement in their own way. And let's see who wins, but like, I don't think it does you any favor to equate world models to robotics or world models to gaming or some kind of like, or like the current manifestations. Because what is at stake is a much more important conception of intelligence than just answering questions. It is, does, does, does, does the AI understand what a table is, like what, what matter is, what physics is? It's almost like for, for those who are movie fans, it's like Good Will Hunting, where Matt Damon like knows everything because he read it in a book, but he's never.

• 53:53Speaker 01

  Great, great scene with Robin Williams.

• 53:55Speaker 00

  With Robin Williams. And I, I look at that scene and I go like, that's exactly the, the difference between like a very intelligent L L M who knows everything, but hasn't experienced anything.

• 54:04Speaker 01

  Wow. That's an awesome note to end on. That's a great. Have you used that? That was great.

• 54:08Speaker 00

  Yeah. So one thing I've done with the end space is I moved to like adding daily write-ups.

• 54:13Speaker 01

  Yeah.

• 54:13Speaker 00

  And so one of the times I was doing this daily write-up, I wrote that.

• 54:16Speaker 01

  That's a great one. I love that. Well, so it's been a ton of fun. Thanks so much for coming.

• 54:20Speaker 00

  It's good to hatch up, man.

• 54:21Speaker 01

  I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.