AI Vibe Check: Chinese Open Models, Distillation & The Hugging Face Breach

In this installment of their recurring roundtable, Jacob, Ari, and Rob dig into the accelerating Chinese open-source model race, debating whether Kimi K3 has actually closed the gap with the US frontier or just looks like it has, and whether distillation fully explains China's progress. That opens into the messier fight over open-weight models generally, with the group dissecting the backlash against Anthropic's stance and asking whether powerful open models are inherently dangerous or actually a necessary part of collective defense. They connect this directly to the Fable ban episode as a possible preview of a coming frontier-model licensing regime, then turn to the OpenAI and Hugging Face security incident as a landmark, publicly legible moment of autonomous AI causing real-world harm. From there the conversation shifts to business strategy: how much value are companies actually handing over to frontier labs by building on their APIs, and what does the Grok and Cursor data advantage really tell us about the future value of real-world usage data versus purchased training data. They close with a rapid-fire tour of the news cycle, covering SSI's mysterious $5B raise, the rumored Stripe and OpenRouter acquisition, Google's continued underperformance despite structural advantages, and OpenAI's leadership speculation, plus a reflection on venture's swing back toward deep tech and physical infrastructure.

• 0:00Speaker 00

  My mom like knows nothing about AI. She texted me last week, Rob, what do you think about OpenAI's agents hacking into Hugging Face?

• 0:06Speaker 01

  What do you two make of it?

• 0:07Speaker 02

  I think this is actually a great example of why open models of that capability are important.

• 0:12Speaker 01

  How much do you think distillation is the explanation for what's happened here?

• 0:16Speaker 02

  I don't think it explains nearly as much of it as many would like to claim.

• 0:19Speaker 01

  What specifically do you see as the risk of like dependency on Chinese open source models?

• 0:23Speaker 00

  I do think there are downsides to having the AI substrate of the world come out of China.

• 0:28Speaker 01

  I'm Jacob Effron and this is Unsupervised Learning, a podcast where we probe the sharpest minds in AI on what's happening in the space today and what it means for the future. We've had a lot of new subscribers in recent weeks. As part of that, we have a great recurring segment with Ari Marcos from Datology and Rob Taves from Radical Ventures. The three of us chop it up on everything happening in the AI ecosystem. This one was particularly eventful and a ton of fun. There's been a lot going on. We talked about the OpenAI Hugging Face incident, how big a deal it actually is and what it means for the future of frontier AI research. We talked about Chinese open source models, including how good Kimi K3 actually is. And Ari shared some really interesting insights into what the real long-term risks are of Chinese open source models and how we know if those risks existed.

• 1:09Speaker 01

  We talked about the labs and how they're competing increasingly up the application stack and what it means for their customers and the development of models going forward. And we hit on what we think government policy will look like for frontier model releases in the future. I love talking to these guys about everything in AI. We do it a lot off camera and it's particularly fun to get to do it on camera and share with you all. We're constantly iterating on the format, so any suggestions, things you want us to talk about, please let us know. But if not, without further ado, here's Ari and Rob.

• 1:38Speaker 01

  Ari, Rob, it's good to see you both again.

• 1:41Speaker 02

  Great to be back.

• 1:41Speaker 00

  It's great to be back. Thanks for having us.

• 1:43Speaker 01

  Yeah, we've got some new backgrounds, some new craziness happening in the AI world. It's all, it's all changing.

• 1:49Speaker 00

  Ari's background got better, Jacob's background got worse. It's a, it's a dynamic space right now.

• 1:54Speaker 01

  It is, it is. It's constantly ebbing and flowing who's on top. So, you know, no, no surprises there.

• 2:01Speaker 01

  But I've been excited. I feel like, you know, we had all these spicy predictions last episode and there's like a follow-up on each of them, which is, I guess, what happens in the rapidly changing AI world. I feel like there's a ton of stuff we're going to talk about today. I'm excited to hit on open source, you know, government getting involved in a bunch of these models. You know, what's been happening at the labs, the hacking over at Hugging Face, as well as this new letter we have out today from a, from a bunch of researchers trying to slow things down. But I figured, you know, no better place to start than with open source models. It's kind of been one of the, the main topics of the past weeks. We've had some incredible Chinese model releases. We've had a lot of intrigue around whether the government's going to allow said Chinese models to continue to be used here stateside.

• 2:41Speaker 01

  And then we've had letters signed by many people to keep open source models and backlash against Anthropic for, for, for maybe going against them. A lot happening. So maybe to start it off, Ari, I feel like you've been vindicated. You, you, earlier this year, you kind of said, I think there's going to be a Chinese open source model. It's going to be the world's best at something. You're looking pretty smart with how things have shaped out. What's your read on how this has all kind of happened?

• 3:06Speaker 02

  Yeah, well, I mean, pleased with that prediction, obviously. But, but I think this trend is, is going to continue. I don't know if they're going to continue to be fully open. So for example, if you look at K3, they adjusted the license to have a revenue gate on it now. So, you know, there's a conservative revenue gate for model as a service and then, you know, less conservative one for otherwise. But if you make more than, you know, 20 million a year, 200 million a year, depending on the threshold or 20 million a month, I'm sorry, depending on the threshold, you can't use without a commercial license at that point. I have a feeling we're going to see a lot more of those starting to come out across the board. But I do think we're going to continue to see extremely strong models coming out from China. And then also, you know, now we have Inkling from Thinki and a couple others from US players.

• 3:51Speaker 02

  That will hopefully continue to, to fill out that space.

• 3:55Speaker 01

  Yeah.

• 3:56Speaker 02

  But I think this trend is only going to continue with seeing powerful models that are on the open and that are outside of the frontier labs.

• 4:02Speaker 01

  Yeah, I mean, I think, Rob, you had said last time you were a little skeptical of the, of the, of the open source business models. And so maybe we're starting to see the, the, the outlines of how that might come forward. Like, do you kind of, you know, is this inevitable that like this, this kind of new model is required? And is that, is that 20 a month going to come down pretty, pretty soon for, to be a little more all encompassing?

• 4:22Speaker 00

  Well, so first on this topic, I would say while there is no question that Ari is very wise, I, I would, I, I would actually posit that. I think the Kimi K3 launch and, and other open models, like, I think they've been, I think it's been oversold a little bit, how big of a milestone it is. Because like, I don't, I don't think K3 is actually state of the art in any real sense. Like, I know there are certain, there are certain categories that it outperformed, but the, like, it's important to keep in mind that Kimi K3 is being compared to Fable, which is itself a like neutered version of Mythos, which itself was finished training in like February, March.

• 5:07Speaker 00

  And so like, take, all of that taken together, like, I think it's fair to say the, the open frontier coming out of China is still several months at least behind the actual frontier in the US. So I do think that's one, like, piece of the narrative that has gotten kind of lost because it's a much more compelling narrative to say like, oh, China, open source caught up to the US frontier. And I don't think it actually has. I mean, it's certainly like a very, it's the best open source model. Like, in that, in that sense, it's state of the art. It's the best open way model.

• 5:32Speaker 02

  To push on that for a second, like barring a full hard takeoff sort of scenario, does that actually matter? Like, does open source being three to four months behind closed source, is that actually a meaningful difference?

• 5:44Speaker 00

  It depends on the use case. Like, certainly there are a lot of use cases for which it doesn't matter. And I do think there's this broader trend of companies, enterprises rationalizing their spend on AI and realizing like, we shouldn't be using the, the cutting edge frontier model for everything. And in fact, it's like totally nonsensical from a cost perspective to do so. But I do think it's like symbolically and strategically very important, like where the frontier is and how far behind the frontier other players are. Including if, if you, yeah, if you take seriously the, the possibility of recursive self-improvement or so forth, like that advantage can compound to your, to your point about a hard takeoff.

• 6:23Speaker 01

  If there is a frontier, there are certainly many customers out there that are buying AI products that want to feel like they are getting the, the best out there again. Like, I don't know if anyone wants to feel like they're going to a, a health system or a law firm that's not using like the most cutting edge thing, even if it ends up being, you know, like slightly better or, or worse in some areas.

• 6:39Speaker 02

  I would guess consumers don't, don't know the difference. I would bet that most people.

• 6:42Speaker 01

  Consumers know the difference for like years?

• 6:44Speaker 02

  No, I don't think any, like, I think certainly one thing that I think is interesting is like, you get this really bimodal distribution, especially if you go to like Blue Sky, where there are like all these people who like absolutely hate and despise AI and have clearly not used anything except for like GPT 3.5. Like, the general public is, is actually misled substantially by the free tier of these models. Because I think many of them think that's the best. But I would, I would guess that actually like, most customers are going to have no idea about this. And like, if anything, I think we've started to see this backlash to the token maxing idea in some ways, right? It's all gone to routing companies and how there's so much more you can do there. So I think like the right size model for the task is, is something that everybody is increasingly moving towards more and more.

• 7:29Speaker 02

  And I think that reduces the benefit of being at the frontier because it's now only as relevant for whatever subset of your problem space actually matters, where you need that marginal increase in intelligence.

• 7:41Speaker 01

  Yeah, exactly. You better hope that the, the frontier keeps improving capabilities so that that set of tasks keeps increasing. Because it's certainly things like the ones that, that anyone that's there a year ago is now perfectly addressed by, by these open source models. You know, Ari, one thing I think you've been consistent on since the start of, of us recording these is like, that maybe in the, in the narrative, in the Western ecosystem, we always like to say, you know, it seems that this broader narrative emerged for these Kimi models, like, oh, it must be distillation like that has driven this. And obviously there's been incredible research, you know, out of, out of DeepSeek now, out of, you know, incredible models out of Kimi. Maybe contextualize for our listeners, like, how much do you think distillation is the, the explanation for what's happened here?

• 8:21Speaker 02

  I think it explains some of it, but I don't think it explains nearly as much of it as, as many would like to claim. It's definitely not a zero advantage. There is a real advantage from distilling, especially distilling the reasoning traces. Those are obfuscated in many ways, but you know, there have apparently been some ways to get around that and so on that, that have led to some of that. And there is real value there that is non-zero. That said, I think trying to take that to the claim that like, the only reason that the Chinese models are even remotely competitive is because of distillation. And they're just piggybacking off the hard work of the American companies. I think that that stretches credulity a little bit. It's somewhere in between.

• 9:03Speaker 02

  They're meaningful, but it's not the it's not the end all be all. I also think there's something to the effect of, you know, obviously, many of the American frontier models, at least initially, were trained on a lot of data that wasn't permissively licensed. And you could argue that they were distilling from everyone. So then they turning around and saying, hey, no one distill from us. There's some amount of hypocrisy to it, even if it's understandable from a business incentive perspective.

• 9:28Speaker 01

  Yeah. Rob, where do you think this conversation on distillation like ends up? You know, is it something that can be policed? Is it something that like actually we want to be policed?

• 9:37Speaker 00

  Ari probably has a more a more concrete sense from a technical perspective. But my general sense is like, I think it's very hard to prevent distillation as long as there are models that are publicly accessible via API. Like, I think the only the way to shut it down would be for cut for frontier labs as Ari like intimated in the last conversation to say, you know, we're actually going to like shut down our APIs and not make our models publicly available and and use them to monetize in proprietary ways, you know, whether that's trading on on stock markets or, you know, building your own products end to end. So it feels like distillation is kind of an inevitability.

• 10:19Speaker 00

  I think it's something that like US labs are doing and Chinese labs are doing, like Meta is doing it and, you know, other non frontier lab companies are also doing it. I think in this open source debate, there like, there's been a lot of, there's obviously been a ton of discourse over the past few days on Twitter. And like, frankly, it kind of feels like Anthropic has kind of been framed as the bad guy in a lot of ways, or like they like, certainly the vibes are moving against them in the sense that they're the sole company that didn't sign this open letter. And they're kind of being cast as like, you know, the company that's trying to shut down open weight models for competitive reasons and regulatory capture. I like my view on it is I do think there's like a very different.

• 11:04Speaker 00

  I think a couple of different conversations are being muddied here. I think that there's it's a very different discussion about open weight models versus open weight models from China. And I think that there's not really good argument as to why open weight models in general should be banned or heavily regulated. And I don't think Anthropic is advocating for that. In fact, Dario published a letter yesterday where he said, like, to be clear, I'm not like this is this has never been the stance of Anthropic. I do think there are like actual legitimate geopolitical reasons to be very cautious about how reliant we want US enterprises and like the global, you know, global users more broadly to be on Chinese open models and to have a Chinese tech stack,

• 11:47Speaker 00

  the the foundation of AI going forward. So I do, I actually think that like, I think that Anthropic's position is more defensible than like is is popular to to believe right now.

• 11:58Speaker 02

  Yes, I think that makes sense to a point. But, you know, Anthropic's position isn't so much open weight models are fine and they're not against it. It's more that they they don't care whether they come from China or the US. But if they're too powerful, they don't think open weight models should should exist. And I really question, you know, that argument. I think we'll talk about the opening eye, hugging face incident in a bit. But there's a very obvious asymmetry here. If, you know, only a small set of people have access to the attacking model and the defense model is then very limited. These are going to proliferate. There's this notion of, OK, we want to pause AI for safety reasons and so on and so forth. And like, look, I think a lot of AI researchers would actually support

• 12:43Speaker 02

  a pause if there was some mechanism to globally enforce it.

• 12:47Speaker 01

  1,122 and counting right now that have that have signed this letter as the that came out the day we were recording here.

• 12:54Speaker 02

  The thing is, though, it only makes sense if you can actually globally enforce it. And like, while I would be, for example, you know, in favor of some sort of pause for safety reasons, if it could globally enforce, it absolutely cannot be globally enforced. That's just not a realistic thing in any sense of the word. Right? So you have to assume that these models are going to continue to get better. And that's going to mean that you're going to end up with the bad actors having access to the more powerful models and good actors not necessarily having those that access. So I think the argument that like open weight powerful models are inherently dangerous and therefore a very limited set of players should have access to them is both not an effective strategy and not realistic. I think to some extent Anthropic was walking that back.

• 13:39Speaker 02

  But but I also that, you know, I think it's hard to not see their business interests in that, particularly given the extent to which, you know, they have been particularly framing this as a war between America and China.

• 13:53Speaker 01

  Just wanted to take a quick break to say that if you're enjoying the conversation, the single best way to support the show is hitting follow or subscribe wherever you're listening. It makes a real difference in helping us get the best guests and helping others find the show. Now, back to Ari and Rob. I mean, Rob, what specifically do you see as the risk of like dependency on Chinese open source models? I mean, is it literally that like, if, you know, there's like a the ability to cut off access and like the impact that would have like throughout the economy? Or like, what kind of play it forward, I guess, of what you what you'd be worried about?

• 14:23Speaker 00

  Speaking from a pragmatic perspective, like, I do think that we are engaged in a high stakes, global power struggle between the US and China. And the countries represent different value systems. And I do think it's clear that AI is becoming one of the most important axes on which these two country, two countries and two kind of world spheres are competing. And so in theory, it would be great if, you know, it's it's a positive some world and everything is great and China can do great and the US can do great. But I think in in reality, like, there are, there will be a lot of kind of political stakes and policy decisions that are made globally

• 15:08Speaker 00

  and and global influence that has real implications for the way people live around the world. And I just, I think it's, I think we're in a better place if the technical foundation on which all these applications are built globally, not just in the US and in China, but in Europe and in Latin America and in Africa. You know, I think it's a real power disadvantage for the US if like the foundation of all that technology is built on Chinese models in terms of the, you know, the training data that went into them, the values that are embedded in them. I do think it's like a, you know, it's a it's kind of a soft power thing, but I do think it's meaningful.

• 15:50Speaker 00

  And, you know, it's similar, in a way, it's similar to how like, I think China has done a really strategically foresighted thing in investing billion, many, many billions of dollars in developing the infrastructure in much of the developing world, in Southeast Asia and in Africa. And that gives them a lot of influence over those countries and their governments and their policies. And and, you know, I think a similar thing can and will happen in the digital world. So I think like, if you're, you know, if you believe that a more free and democratic and open society is better globally, like, I do think there are downsides to having the like the AI substrate of the world come out of China.

• 16:30Speaker 02

  I'm actually inclined to agree with part of that, for sure. Like, I think between ignoring the problem with open models generally, which that I am very skeptical of. But specifically about Chinese versus non Chinese open models. One kind of truism of model customization and model building is that the kind of easier it is to put in, the easier it is to take out. And by extension, the harder it is to put in, the harder it is to take out. So what do I mean by that concretely? If you can post train a behavior into a model with order 1,000 samples, you can likely remove whatever capability change you've injected with order 1,000 examples.

• 17:11Speaker 02

  If in order to induce a new capability, it requires trillions of tokens of data, it will likely require trillions of tokens of data to remove that or very well might to fully remove it. So I think that the situation, which I do think is actually realistic, and I'm not sure if they're doing this right now, but I can easily imagine this being something that happens in the future, is effectively building in certain patterns of behavior to the models from the earliest stages of pre-training that are very, very difficult to remove through post-training and are very difficult to test whether they exist. Just to kind of use a hypothetical, somewhat scary framing of this. Think about Stuxnet, right?

• 17:54Speaker 02

  For those that aren't familiar, this was a virus, a computer virus that was purportedly developed by the Israeli and US militaries that was designed to interfere with Iran's nuclear development. And this virus transmitted through USB primarily. And I think at the when it was reported, I think it was on something like 60 or 70% of like all devices in the world, period. It proliferated everywhere, but it was a no op almost everywhere. And then in the one circumstance where it was loaded on to this particular model of centrifuge that was used for enriching uranium that Iran was using, it would then activate and do the secret thing where basically

• 18:37Speaker 02

  it would make the centrifuge run at a higher speed than it was supposed to, such that the failure rate of the centrifuge would be much higher than expected. A really, really interesting story if you're if you're not familiar with it and excited about it. But like, the the attribute of that that is very scary, right? Is that it's prevalent literally everywhere, but it only takes the malicious action in a very restricted set of circumstances. It is not hard to imagine China doing something like that, where, you know, if the model is deployed in a particular circumstance, it then starts to biases bias its results, right? And you can imagine this from like really big things, like maybe this is deployed in actually some sort of mission critical operation where it's going to give the wrong answer at the at a critical time.

• 19:21Speaker 02

  Or you could potentially imagine this even in a more subtle way where, hey, say that, you know, this ends up being used for ranking as part of a social media, you know, platform or something like that. Say Snap were to start using that and then the model would have biased more towards, you know, authoritarian values or or things like that. So I think there is some real risk there. Now, I'll note, I don't think we have any evidence this is happening at the moment, but I don't think there is any technological reason why this couldn't happen. This is, I think, a very, very realistic thing. And I think that the problem with this is that if you don't know how the model was baked, it's very hard to, I can't see how you would say, I guarantee that this model doesn't have any of those problems.

• 20:03Speaker 02

  So I think that is one real risk, and I don't think it's hard to see that the incentives for the CCP are certainly pointed in the direction of doing something like that, right? We already have evidence that this was the case in TikTok's ranking algorithms, right? And things like that. So that does seem like a real risk to me. And while I don't think that matters for 99% of deployment cases, there are some where you can imagine that being impactful.

• 20:28Speaker 01

  It's so interesting, because obviously then, you know, there's been this government backlash of like, hey, well, you know, should we do something about these models? Or is it really fair that like, you know, they're basically, you know, the Chinese models are distilling a ton of American models, and then in turn, the American open source players go and try and distill the Chinese models. And it's like this, this round tripping around. And so I wonder where, where we'll end up from a government policy perspective. You know, it seems like people have been flirting with the idea of a ban, and that's what like got, you know, the whole folks galvanized against that. And I don't know, do you either, you have a gut instinct on like where this all lands?

• 20:58Speaker 02

  Hard to predict whether the government will make a wise decision. I think it would be a very bad thing for us to cut off access to open source models. I think it would be a pretty big gift to the frontier model companies, frankly. And I think is, you know, would be a good example of why, you know, Jensen, for example, has been so adamant about trying to build a non-Chinese open model ecosystem, you know? And I think if we don't see that change, like, I think that that could lead to, you know, more challenges for folks. That said, I will also note that, you know, especially for niche use cases, it is increasingly viable to build from scratch with the right data. You know, it's still challenging.

• 21:36Speaker 01

  And the right partner to do that with.

• 21:38Speaker 02

  And the right partner to do that with, such as Datology. But, you know, there is, there is still a real path towards that, and I do expect that's where people will go long term, because they'll start to want to cut in for its costs. But at the moment, I think where most enterprises are now really focused on how can they own their intelligence and are focusing on doing that through customization of existing models, I think it would be a really harmful thing to the ecosystem and AI development overall if we were to do something like that.

• 22:05Speaker 01

  I mean, to this point, like, a lot of people seem to be thinking now about, like, you know, the long-term implications of like, being tied to a frontier model or being tied to a Chinese open source model. And I think, you know, Rob, you mentioned earlier, and it's felt very true the last few months since we recorded, like, there's been this overall backlash against, you know, Anthropic and the, like, large labs. And I think maybe it was best encapsulated by the Palantir CEO, Alex Karp, who went on this, like, rant on CNBC that went pretty viral. And I think, you know, there were many points he was making, but one of the things he was saying, which I think, you know, increasingly, a lot of folks have felt as the frontier labs move more and more into different application areas, is like, these labs, they're partnering with you, they're kind of like getting their models into your ecosystem, learning your business, and then they're like, learn your business, and then suddenly, like, you can't compete in all these different ways.

• 22:50Speaker 01

  And I'm curious for both of you, like, how fair a critique is that? Like, what are the model company, what are and aren't the model companies, like, actually learning through these deployments? So clearly, like, Palantir has their own incentives to, like, pitch a different story where, like, you need to own all this stuff internally, because why would you give any sort of, like, valuable insight into the labs themselves? I'm sure, as always, the truth is somewhere in the middle of that spectrum. So maybe we'll start with you, Ari. Like, how do you articulate, like, what is a business actually giving up to the labs if they, you know, deploy Anthropic models all over and put them in Slack and all that good stuff?

• 23:23Speaker 02

  Yeah, I mean, I think the real challenge comes when you start handing over data and expertise, right? And I think the reality is that both of the major frontier labs have been, have said the quiet part out loud pretty reliably, that they intend to, you know, obviate many of their customers' business and take it over, right? So it's not even a case where they're pretending not to. And the primary challenge to get those specialized capabilities is going to be the domain expertise that exists in many of these legacy companies and the data that is representative that exists in many of these companies, right? So if you hand those two things over, then I expect what you will see rolled out is that eventually, you know, their unwrapped version by default is, you know, comparably good to your wrapped version.

• 24:05Speaker 01

  But is your average business giving away, like, domain expertise and data to the labs? Like, is that really how these contracts are set up?

• 24:12Speaker 02

  If you're working with them to customize models for you, then you are starting to hand over the data, I think. And then I think more generally, if you do build on top of them and are completely dependent upon them, the risk you run is that, you know, as they improve, eventually, your differentiation becomes smaller and smaller, and you have to pay a much higher margin. So their business just has simple, better unit economics. I mean, this is the story we saw with Cursor and, you know, now with Harvey and with many others, where kind of a lot of these companies that are built on top of this are realizing that it's a really huge risk for them if they are entirely built on one of the frontier models, because, you know, they're competing with their hands tied behind their back, just in the same way that, you know, Spotify competing against Apple Music is going to be fundamentally limited, because they, you know, Apple doesn't have to pay the Apple tax and Spotify does.

• 24:57Speaker 01

  What does Anthropic really see if you're, if it's deployed at JP Morgan of like, you know, how people are using the model or things that actually make it that much better? And, and I think sometimes, I don't know, I feel like there's been this narrative of, like, every business is so unique and has such special data that, like, no one could possibly replicate. I'm like, some of that is true, but also, you know, it turns out, like, if you go hire a bunch of ex-bankers, like, you will figure out how to do basic banking tasks. Or like, the, like, data providers are, like, willingly able to provide a lot of what exists in a, you know, in a work environment to the labs directly. And so, I don't know, I felt like there's been such a blowback against the labs themselves and like, this idea that, like, every time you, you know, you use the Claude API, you're like, just bleeding your business value into them directly, I felt like was maybe a little overstated.

• 25:43Speaker 00

  I am inclined to agree, and I do think, like, the overriding motivation for creating this counter narrative is, like, ulterior motives from, from other groups.

• 25:55Speaker 01

  I mean, all this stuff, everyone, the letters, all this stuff, even if you agree with it, it's like, everybody clearly has their incentive.

• 26:00Speaker 00

  Yeah, exactly, yeah. To your point, obviously, Palantir is in a position where, like, they, it benefits them tremendously to have a world where enterprises are not going to open AI in Anthropic to get their enterprise AI solutions, but are instead relying on Palantir. And even Nvidia, like, Jensen's, Jensen is hugely strategically focused on promoting an open weight environment, and it's because, I think he's done the calculus, that that is much better for Nvidia in the long run. And I think it's a similar, similar logic to.

• 26:31Speaker 01

  Commoditize your compliments, right? Like, just have a bunch of, nothing will be better for him.

• 26:35Speaker 00

  Exactly, yeah. And it's very similar dynamic to how Nvidia has propped up all these Neo clouds and therefore, like, decentralized the power and kind of reduced the influence that AWS and Azure and GCP have. And I mean, I think it makes a ton of business sense, and then it's easy to come up with the kind of, like, ethical or philosophical justification for it after the fact.

• 26:56Speaker 01

  I think it makes it hard to actually disentangle, like, what is true? Because there's, there's kernels of truth in, like, all of these things.

• 27:01Speaker 00

  Yeah.

• 27:01Speaker 01

  But they're obviously taken to, to an extreme.

• 27:04Speaker 02

  I think people definitely, you know, end up being hyperbolic with this. Like, I think the, the version that you portrayed, Jacob, of just, like, every time you use the API, you're giving away business information.

• 27:12Speaker 01

  It's like kind of what he was saying.

• 27:14Speaker 02

  That's, I think, going a bit too far. That said, I think, like, there are folks that are talking about partnering much more deeply with OpenAI and Anthropic to do this. I think that is much more of a risk when you go and do that. I think the other thing that's, that's, you know, just realistic is, there's that feedback loop of, hey, the model is poor at this set of queries, where, you know, by do it, by using the API and then working closely with them, what you are effectively doing is giving them distribution to the, to your exact set of customers.

• 27:42Speaker 01

  The way you're giving them is like, is evals, right? Like.

• 27:45Speaker 02

  You're giving them evals that help them to improve it. Exactly. And those are hard to access. So what you can imagine is that by using the API reliably and by giving feedback reliably to Anthropic about where the model's working and where it's not working, or to OpenAI or whatever, you are effectively hastening their development of their competitive product to you. That, I think, is a very realistic one.

• 28:05Speaker 01

  Yeah. And it's interesting, because in the early days of application development, I felt like every cutting-edge app was like, just working so closely with the labs of, like, here's all our evals that don't work, and like, please make them better. And it feels now more and more like, you'd kind of expect them to keep it a little internal and like, maybe fine-tune the open source model and address, address that eval that way.

• 28:22Speaker 02

  I think there's also just like a broader point of, like, businesses don't like extreme vendor dependence, right? Like, that's a risk that every business has faced forever and like. It's not good for the world if everyone is dependent upon this small set of companies. I think fundamentally, at least that's that's more starts getting more to a moral belief to some extent. But it is one that I believe strongly that this is too powerful a technology to have it be concentrated in a small set of actors. And I think realistically, a small set of actors that have shown that they are generally putting their interests first. Which is fine. That's what I would expect of most anybody. Everybody acts to their incentives.

• 29:06Speaker 02

  But that means that you want to incentivize the ecosystem to have much more control, to have much more diversity of players, to not have this centralization, especially for a technology that is, you know, even a tenth as impactful as AI.

• 29:22Speaker 01

  So I've obviously been, you know, hyper focused on us not becoming a cliche of a podcast. And like, one of the most important things to avoid that is to avoid talking about like, politics and at all. That being said, like, it does feel like a lot of government stuff has really dominated the headlines in the last few months. Like, I actually found, we recorded our last podcast and we were talking all about Fable and like them, you know, banning it for different uses. And then I think we released it 48 hours later and like, five hours after we released, the government just like banned Fable. And so, you know, I hope we get a mulligan of like, talking about politics, but it's pretty relevant to just AI world. We won't go into like the 28 election, but like, maybe just like on the Fable side. You know, what did you make of, that was like quite the, you know, quite a process.

• 30:05Speaker 01

  You know, they obviously banned or tried to ban like, you know, all foreign nationals from using it and anthropically cut off access. And then, you know, we pretty soon after had open source models that were, you know, pretty close to parity. And, you know, it all kind of feels like a footnote in how it went, but maybe a dress rehearsal for more of this stuff down the line. What do you two make of it?

• 30:23Speaker 00

  So Jacob, are you sure you don't want us to talk about JD Vance versus Marco Rubio for the Republican nomination?

• 30:29Speaker 01

  I think we should do a separate episode, just like the three of us, as little qualified, as little pre-research as possible, just opining very confidently about, you know, that matchup.

• 30:39Speaker 00

  Yeah, you're right. It is. It is a real virtue of this podcast that we have. We have not yet talked about politics at all. But you're right. I mean, it is, I think it's becoming like impossible to ignore if you're serious about the world of AI, because it is such an important dimension to it. I happen to share a few thoughts on the Fable thing. I mean, part of it is like, I do think you can't ignore the dimension which is like idiosyncratic to it, which is that the Trump administration really doesn't like Anthropic. And Anthropic has done itself no favors in terms of how it's positioned itself with the Trump administration and with the public. They've been essentially like begging to be regulated, you know, and kind of like put they're at the forefront of making this argument that AI is so powerful, the government needs to be involved.

• 31:24Speaker 00

  And then the way they handled, like, when there was this security breach or concern raised by Amazon, the way that they handled it, it sounds like was very dismissive and flippant, like, oh, no, that's not a big deal. Don't worry about it.

• 31:40Speaker 00

  And so they just, I just think they haven't navigated, like, they haven't taken seriously enough their relationship with what is a very serious partner to be thinking about, which is the administration, the White House and the Trump administration. So I do think that was part of it. It also, like, it always felt like the kind of thing that was going to be temporary and unwound, because it's just not. It's untenable and a little scary for the US government to be that heavy handed in terms of how, you know, arbitrarily deciding what models can and can't be released in an ad hoc basis. So it always felt like something that was going to be lifted and it was lifted.

• 32:21Speaker 00

  But I do think it, as, you know, to use the phrase you used, you said as a dress rehearsal, Jacob, like, I do think,

• 32:29Speaker 00

  you know, as these models get more and more powerful, to this point around like, what happens with open weight models? I think I could envision a world where, in a more systematic way, there is some, some version of a licensing regime, and this is like, started to be discussed. This has started to bubble up more and more as a policy concept, but like, for true frontier class models, like, do you need to jump through certain hoops and get certified by the US government prior to being able to make the model publicly available? That like, obviously will get the backs up of many libertarian minded folks. But like, I do think that's one possible policy direction we may be headed in.

• 33:09Speaker 00

  And if so, like, you know, what played out with Fable could be sort of like a harbinger of that.

• 33:16Speaker 01

  I mean, if open source is truly three to four months behind and continues to be like, how in the world do you do that? Like, how would you have a process that was fast enough?

• 33:23Speaker 00

  Yeah. So I think that, like, I think if we go down that path, and I'm like, I'm not saying that's necessarily the most likely path, but that is one path people are talking about. I do think it handicaps open weight models. Like, I think there will still be open weight models, but they will be more lacking edge. Or they will be from unauthorized parties like out of China. And then, you know, there may be all sorts of other complications with using them. But I think it would be hard for like a US based company that is in good standing with the US government to, to release like true frontier open weight models. And so like, that sort of policy would really favor opening on Anthropic, for sure.

• 34:03Speaker 02

  I actually think that that policy would also really favor non-America.

• 34:06Speaker 01

  Yeah, 100%.

• 34:07Speaker 02

  Then what you're going to have happen, right, is that China's not going to have any restriction. And then a bunch of non-US players are going to use more powerful models that US players do not have access to. And it's going to actually explicitly put America as at a meaningful disadvantage. I'm very concerned about such an outcome, because I do think it is probably the modal outcome at this point, given what everyone is saying, and it seems like a pretty bad outcome. I think fundamentally, there's a certain amount of like, you can't close Pandora's box that's happening here, right? Like, this is not nearly as hard as people make it out to be. It's not easy to build one of these models. Don't get me wrong. Like, I wouldn't say it's fully commoditized.

• 34:46Speaker 02

  But if you get enough smart people in the room with, you know, access to the literature and with access to these models, which can also help you, of course, build better models. And then like, the compute constraints aren't going to be massive, you know, especially now that China's building their own chips. And you can see chemi starting to support more and more of that, you know, to start building on Huawei. I don't see how you put it, put it back in the box. And I think trying to limit by regulating or delaying this, it just, it only seems like it can ever do anything for a month or two. And then fundamentally, after that capability, after that point, you're now going to have a plethora of models with whatever that capability is.

• 35:27Speaker 02

  So like, each time we have a new capability leap, there's going to be some challenge to catch up. But then that just becomes a capability where, yeah, models can do that in the world now, and that's realistic. And like, I think trying to operate in a world where you say, okay, cybersecurity is a huge problem. So we want to limit access to hacking models, just feels very short sighted versus what we know is the right solution, which is you give everybody access to those models so that they can defend against them as well.

• 35:54Speaker 02

  I think that's just like the only practical solution. Everything else kind of feels like wishful thinking to me.

• 36:00Speaker 01

  So would you do anything on the policy side?

• 36:02Speaker 02

  I think it's a challenge of what you do here. I think you might want to say, okay, if there's a certain capability leap, there needs to be a measured rollout or something like that, which is actually kind of what Anthropic did with Mythos, which I think was pretty responsible with respect to that. So I think that could make sense, but I think it's fundamentally a pretty hard problem to solve beyond. I would, I mean, honestly, the best thing to do here, I think, would be to spend a tremendous amount of government funding to have, you know, open model efforts that are, that are going to be guaranteed so that we, you know, you try to fight this, fight fire with fire to some extent by accelerating as much as we can to catch up to that, so that you don't have these discrepancies, which are very concerning.

• 36:44Speaker 00

  Maybe just to play devil's advocate here to tease this out a bit, and Jacob, I'm curious how you think about this as well. Are there, are there any capabilities, or do you think future frontier models could have capabilities that are, that would be problematic to be widely available to anyone who can download the weights, such that there could actually be logic to trying to limit the number of parties that have the capabilities? And I know it's like a controversial and much maligned analogy to nuclear weapons versus AI. But like, you know, I don't like, I think most people would say, it wouldn't be, it's not better for the world if, like, everyone, anyone could have access to the recipe and uranium to create a nuclear weapon.

• 37:28Speaker 00

  And it's better that it's limited. Is the difference with AI just that it's software and software can endlessly replicate? And therefore, there's no possibility of holding it in. And so let's just like, adapt to the world where it exists more broadly.

• 37:41Speaker 02

  Yeah, it's a good question. I think yes is my thing. Like, I think to your question of what's different about nuclear, the recipe for building a nuclear bomb is super, it's straightforward, like, it's available, right? Readily available. The challenge is getting access, and the challenge isn't even necessarily getting access to the raw materials. In this case, the uranium, right? It's getting access to the particular machines you need in order to, you know, get the uranium to a certain level where, you know, it is usable for fission.

• 38:10Speaker 02

  Like, you can restrict it on that capacity. And I think that's why we first started going to chips. Because we said, okay, chips are the equivalent for AI models. But like, realistically, they're not, especially there are so many chips that already exist that you can use more, you can use those, right? So I don't know if there is a realistic way to stop it. I mean, to your question from the beginning, like, what is a capability? Like, I think zero day vulnerabilities is a pretty good one. Like, you have something that can find zero days in everything, you know, particularly in encryption strategies. Anthropic released that in a weaker form of encryption, right? They were able to, to crack it certainly. Like, that would be massively disruptive to the entire world if that were to happen.

• 38:49Speaker 02

  I just don't know how you actually prevent it. It seems like the best case scenario is you delay it by a couple of months. But then the capability comes out and like all you're gonna do is you're gonna make it more likely that that capability is first gonna come out from a Chinese lab or from a place where, you know, it would go in the hands of bad actors. So I just don't actually see a way to contain it realistically. Like, look, it's already super hard to contain nuclear work, right? Like, we've tried quite hard. Like, the entire Western world, you know, tried to prevent North Korea from getting access to the bomb, and they got it. Like, how do you actually prevent this in the AI space? I just don't see a practical path.

• 39:31Speaker 02

  In which case, I think you kind of have to go the other way, because otherwise, this in-between strategy just ends up disadvantaging America.

• 39:38Speaker 01

  Yeah, I mean, certainly needs some sort of like global governance to enforce any of this stuff, right? And enforce some standard set of like guardrails that were put in the models or testing that were done on the models around access to things. Or maybe you just mitigate, you know, I mean, obviously, I guess cyber of course fully in the digital world, a lot of things people are concerned about has ties to the physical world. And so maybe you like really start regulating, you know, the ordering of chemicals or things like that, that, you know, that, you know, because, you know, not all things that could be harmful have as high barriers to building as nuclear does, certainly.

• 40:09Speaker 02

  Right, like, look at, like, trying to buy Sudafed in the US, right? Like, because it's the most readily available precursor for meth, like, you now have to show your ID every time you buy Sudafed. Maybe that's the route we go down, but like, obviously, that hasn't stopped meth production.

• 40:23Speaker 01

  Fair. Hopefully, GPT-7 can just give us a good list of all the things we should, we should claim that on.

• 40:28Speaker 02

  But I think we have to prepare for the fact that we are going to live in a world very soon, if we don't already, where this incredible capability is accessible to anybody. And we need to figure out how do we, how do we live in that world and how do we secure that world, rather than kind of putting our head in the sand and hoping we can prevent that world from, from coming to be.

• 40:50Speaker 01

  Yeah. Well, that's a good transition to, you know, the, obviously, we had the OpenAI Hugging Face incident and like, you know, I'm sure our listeners are familiar, but for quick context, OpenAI had a combination of its latest model and an unreleased one, escape a sandbox evaluation environment it was in, reach the open internet, use some stolen credentials, find this zero-day vulnerability, and get into Hugging Face's systems. And, you know, it's obviously been something that people have been talking about a lot. I'm curious for both of you, like, as we, you know, look back on the history of this era, you know, a few years from now, is this like gonna be, it's obviously feels like a big deal now, is it, do you think this will like persist as being seen as a big deal, or is it just like yet another one of these crazy capabilities AI has and will kind of blend into the, to the background?

• 41:35Speaker 00

  I think it's a genuinely big deal. Like, I think it is, I think it is a landmark event that we will remember for a couple reasons. One is like, it may not have been the first time this has happened, but it is the first time that this story was publicized in a very easy to understand crystallized form for the public. And so I do think it was like a wake-up call for the broader population. My mom actually texted me, I'm gonna, I'm gonna read her text. I hope you're, you're, you're famous now, you're on Unsupervised Learning.

• 42:06Speaker 02

  She's a, she's a regular listener, I assume, right?

• 42:08Speaker 00

  My mom like knows nothing about AI, knows nothing like, is not plugged into the tech world. She texted me last week, "Rob, what do you think about OpenAI's agents hacking into Hugging Face?" And I was like, "I can't believe you know what any of those words mean." I think it did, it like struck a chord with the broader population. And in, I think relatedly, it was like, one of the first examples of AI, like, AI rather than AI being a tool that human bad actors can use to wreak havoc in the cybersecurity world, it was an example of like an autonomous AI being the problem. So I do think, yeah, I think it was like, I think it will trigger a lot of downstream policy actions and commercial actions.

• 42:52Speaker 00

  And I will also say, like, I think OpenAI was very lucky that the victim, if you will, of the attack was Hugging Face, which is an organization that's very like AI-forward, and, you know, was willing to collaborate with them in a very open way. I also think OpenAI handled it very well, like, they were admirably transparent about it. But you can imagine if it happened to be like, I don't know, like, General Electric or Visa, or like some non-AI-native company, that it could have been much worse for OpenAI. So, yeah, it was a very interesting incident.

• 43:23Speaker 01

  I like this eval, my dad emailed me too about it, and I feel like the eval of like, what, what, you know, meets the bar of a parent who's not super deep in the space, sending a contact to you in some way, is a good eval for what will stand the test of time.

• 43:36Speaker 00

  Yeah.

• 43:37Speaker 02

  I mean, it definitely is one of many things that feels like sci-fi come to life, right? I think there's a real thing here.

• 43:45Speaker 02

  I do think this is a really big deal, whether or not, I mean, look, like, OpenAI, the model, they were testing models with fewer safeguards and like, like a less secure sandbox than they would in other contexts. There's a question of how wise that is to do with unreleased, more powerful models. And I think that's kind of odd that they did that, but I would definitely say that I think their transparency throughout this was, was quite good.

• 44:11Speaker 02

  But, you know, I think this is exactly what you expect paperclip maximizers to do. And I think it's particularly funny that in this case, the model was hacking into evals so it could train on the test set effectively. Like, it went to Hugging Face to go and try to find the evals so it could cheat. I think that is both quite hilarious and extremely close to the paperclip maximizing story that goes to show how, you know, models will find a way to minimize their objective, however that may be. I am kind of surprised that there wasn't some like very clear instruction repeated many times in the prompt not to try to hack the test set, because models do that quickly. In fact, post-train bench, you know, which has been one of these benchmarks that people

• 44:56Speaker 02

  are using to automate post-training, just had an update that came out today around specifically preventing models from doing that. So I'm a little surprised that that wasn't in the prompt, or maybe it ignored it. But I do think it's a real thing. I agree with Rob very much that they're lucky that it had Hugging Face and not some other more critical piece of infrastructure. But going back to the open model, you know, point, Hugging Face was only able to detect and kind of manage it as quickly as they could because they had access to GLM-5.2. And, you know, Clem even pointed out explicitly that the asymmetry between the attacker and the defender was a very meaningful gap, especially given that, you know, GLM-5.2 is

• 45:40Speaker 02

  not as good as 5.6 Sol and whatever the unreleased model they were using in concert with it is. So I think this is actually a great example of why open models of that capability are important. It was a critical part of the defense toolkit, and I don't think it's reasonable to expect humans to be able to respond to frontier models attacks. You're gonna need to have defender models there as well. And if you do this kind of tiered release where, okay, you need to be of a certain level of accomplishment before you get access to those unreleased models, of course, like the bigger you make that group, the harder it is to actually keep it limited.

• 46:25Speaker 02

  And overall, I think actually it goes to show really how important it is to have extremely capable, you know, open models that can be used for defense.

• 46:33Speaker 01

  Which is kind of crazy to think that, you know, for the conversation we had earlier, it's like, you know, we'll have open source models with these capabilities in three, four months, right? And like, you know, obviously, there's only, yeah, there's only so much testing you can do around this stuff, right, before, before it's like widely, widely available and accessible.

• 46:49Speaker 02

  I think people also like don't understand the distinction between the model and the harness nearly enough. Like, a lot of the improvements that we're seeing are actually harness improvements more than model improvements. And in fact, even at the time that Mythos was released, or was announced, not released, right, like in February, March timeframe, there were already a bunch of people that just put better harnesses around open models and were able to discover many of the same vulnerabilities that Mythos had discovered, right? So it's also this, it's not just the model, it's the harness that's improving in a lot of these, in a lot of these cases. And certainly, we don't have nearly as much attention going to harnesses and models. But like, no matter what, we're gonna be in a, by the end of the year, there are gonna

• 47:31Speaker 02

  be many open models that have these capabilities, that everybody in the world, including all of the black hats, will have access to. Pretending that we can prevent that world is not realistic and is going to lead to very stupid decision making. Like, we need to like take, bite the bullet and say, okay, this is the world now. These models exist. How do we live and operate in a secure way in that world with these vulnerabilities?

• 47:59Speaker 01

  It feels like kind of crazy to transition from, you know, like the, these big existential questions about like the future of the world with AI to like the much more mundane business questions of like who's going to capture value and how the labs are all competing. But, you know, we must give the people what they want and it can't purely be that. And so I guess I'll move us on to just like, you know, the other developments in the broader AI space since we last talked a few months ago. One of the most interesting ones, I think for that love both your take on is Grok. I feel like the experiment that's happening right now with like Grok and Cursor is so fascinating for the implications that it has for like what's actually valuable and important going forward for, for models. Obviously, the two came together. We had Grok 4.5 released. Honestly does pretty well on like a lot of,

• 48:41Speaker 01

  a lot of benchmarks and, and folks have had a lot of good things to say about it. You know, I think a big part of the excitement around these future Grok models is, you know, the training on Cursor's data, right? And the, you know, just the massive amounts of traces they have and then like, you know, real world experience of people using their models. I think this question is going to be like, how valuable is this? And like, how important is this for, in a world where lots of people can train similar models? Ari, kind of care to start with you. Like, what's your, what's your gut instinct on like, how much is this going to improve the, the Grok models and where it takes them?

• 49:13Speaker 02

  I think it's very meaningful. It's, it's a, it's a very real advantage, right? If you think about just like the, you know, edit by edit information you get from the model put an output, how did the developer edit that output? How did, you know, how, how many times did they reject its output and ask for modifications before getting the eventual thing that they liked? That sort of information is just incredibly valuable and can absolutely be used to really improve the model. And what's great about it, right? It's on policy information. It's information about how the model's currently performing. So it's not just an offline data set that you get and then you apply, right? Like preference pairs, you might go collect from Merkur or something like that. It's actually from

• 49:58Speaker 02

  people using your model and getting the data on that model. That's incredibly valuable data and is very hard to get. And I think it makes a ton of sense why Cursor, you know, would want to, to use that and why, you know, SpaceX decided to acquire them for that.

• 50:17Speaker 01

  So does it become much more valuable once it's actually the Grok model within Cursor? Because obviously, a lot of the data they had previously was other people's models. There weren't a ton of people using Grok within, within Cursor. But now-

• 50:26Speaker 02

  The more on policy it is, the better. Right? But like, just to be clear, there's a tremendous amount of value from off-policy traces as well. They're just less valuable than the on-policy traces.

• 50:37Speaker 01

  Yeah. And then how do we think about like the quantum of data? Because obviously, it seems like, you know, one, one big question that I think will determine a lot is, hey, do you have to be like the number one, you know, external coding application like Cursor to get enough data that matters? Or you just have to have like some amount of traffic? In which case, like, most major model providers can probably find a way to get enough traffic to like improve their underlying models. So I guess it's like order of magnitude more of data that Cursor generates. Do you have a gut instinct like how much more helpful that is than like, you know, I don't know, the number seven most popular coding tool?

• 51:09Speaker 02

  So here's what I would say. At a fixed quality, more data is always better. At a fixed quality, that is a very important caveat, right? Getting more data and having cheap access to compute makes it harder to mess up the model because you are definitely going to show some of the really good data that's in there, right? I would expect that Cursor's, the, the data they're collecting has the same kind of power law, long tail effect that we see in all other data sources where there's a comparatively small fraction of the data that actually drives most of the improvement.

• 51:51Speaker 01

  For a listener, what makes a piece of coding data like particularly good?

• 51:55Speaker 02

  It'll depend fundamentally on where the model is, but like you want to find data that is going to teach the model something it doesn't know, right? So like, the technical term we always care about is we want to maximize the marginal information gain. So how much extra information does a model learn from each successive data point?

• 52:10Speaker 01

  The vast majority of Cursor data is like going to be stuff the model already knows. And then they'll-

• 52:13Speaker 02

  Exactly.

• 52:14Speaker 01

  Yeah, I guess you could think about in the self-driving context where it's like most of driving is like pretty obvious to, to, to the models now. And then there's going to be some, some subset of data that like really, really matters.

• 52:23Speaker 02

  Yeah, I think the self-driving context actually is a really easy like analogy for it. Like, consider that like the data set that Tesla would be collecting, right? Almost all of that is highways. Like, self-driving cars are good at highways. They've been good at highways for a decade, right? The hard part is construction zones and, you know, going through SF when, you know, there are like several lanes are closed and there's a person, you know, with a baby carrier that's looking at their phone and may or may not walk into the street, right? It's that long tail of data. So the more data you collect, the more of that long-tail data you have access to and then the more of it you can use. So there is a really meaningful advantage. At the same time, I think we have consistently seen, and I would argue that this is kind of our whole mission at Dataology certainly,

• 53:05Speaker 02

  is that if you are clever about these things, you can do a tremendous amount more with less. So I think that, you know, all things being equal, if Cursor were to, were to apply the same amount of effort to kind of really enriching the high-quality data as a smaller player would, they will always have a fixed advantage because they will have more data at any given quality level than a smaller player. That said, I think what we have seen consistently is that bigger players end up using compute as something of a, of a coverall to some of these issues. And they then invest less in, in these areas because they're less critical and it's easier to just say, okay, we'll just like throw more compute at it. And that leads to this. So, so if you're the number seven player, I think there's

• 53:44Speaker 02

  still absolutely room to catch up. But the, the barrier to entry from a kind of technical standpoint is now higher. You have to be more clever to, to make up for that. And like circling back to the Chinese labs, I think what we've consistently seen is that in many ways, they have been extremely clever. And that's how they've been able to overcome the disadvantages that they have. So I don't think it completely runs it away, but this is definitely a rich get richer sort of process where you would expect that having more data makes it more likely that you're going to have the best coding application, which makes it more likely that users use you and you get more data and so on and so forth.

• 54:19Speaker 01

  Yeah. I mean, interesting is like how much better it makes the models, I think, has like massive implications for like the general application space and how the model companies, you know, play in it where, you know, today, like most of the model improvements are probably, or a lot of them are driven by like just buying, you know, Merkur or Scale or Surge data. And the relative value of that versus the, the data you get out in the real world from folks using the product, if it turns out like, God, that other data, there's only so much you can do on like predisposing what people might do or like getting, buying a bunch of RRL environments or labels. And it turns out, it's way more valuable to have that data in the real world. You know, you may see labs get more aggressive, even more aggressive on offering their models directly to customers and/or like acquiring companies and spaces similar to, you know, what was done with Cursor to like, you know, to, to improve those models further. And so I feel like this experiment of these Grok models

• 55:03Speaker 01

  will have a lot of implications for, for how the broader space unfolds.

• 55:06Speaker 02

  Yeah. I mean, going back to kind of my, my prediction from before about the APIs potentially, you know, being deprecated or I don't think long-term deprecated, but being deprioritized relative to first-party offerings. I think part of that is because the first-party offerings have higher margin. But I think a bigger part of that is that the first-party offerings have all the contextual information about the way the models are being used that you don't get in the same way if you're just access, being accessed through the API, right? If, if, if Cursor accesses Claude through the API, Claude, Anthropic doesn't get all the context of what the user was doing and then, and exactly how it was editing things and so on and so forth. It just gets the queries every time Cursor decides to call it versus getting all of that information fully featured in the context of Claude code

• 55:51Speaker 02

  or Codex or something.

• 55:52Speaker 01

  Rob, where do you think Grok's going to end up on coding after this experiment? Is, has this been played out a little further?

• 55:56Speaker 00

  It's hard to say. I, I mean, I think Ari makes a lot of very interesting points around the value of the data. I still am, I remain fundamentally skeptical that Grok will be a, will be a model that stays on the frontier in coding or in other use cases. Just because I think it's, it's so hard to keep up with the frontier. And I, I think this, like the signals like SpaceX renting out, you know, much of their compute to Anthropic and Google, to me is a pretty definitive signal that like they're choosing to monetize that infrastructure in other ways. So I, I still don't view, I don't view SpaceX AI

• 56:39Speaker 00

  as a company that will stay on the frontier long-term. It'll be interesting to see if that, if that proves to be wrong. But Cursor as an application, I think, can continue to be a super, super powerful and lucrative property for the known.

• 56:51Speaker 01

  Well, we've got to hit a, a variety of other headlines before we wrap up here. And so I'll rapid fire through them. The first is, is a pod favorite topic of SSI. Finally, we had, we had a big announcement, right? Like it's time, time to scale. You know, what, $5 billion from, from Nvidia, a lot of compute. You know, still no, no exact inklings of what, what the, the breakthrough is, but you know, certainly some, some excitement there. You know, either of you have, have, have some thoughts here?

• 57:20Speaker 00

  Big announcement and yet nothing was actually said. Surprisingly, I mean, yeah, if we if we really wanted to make this podcast go viral, we should we should have a big reveal about what SSI is working on because I honestly still have no idea and I haven't talked to anyone that has any any real idea. It'll be interesting to see. What I'm curious about is like, how much did they pull back the kimono to Nvidia and Jensen? And my guess would be not that much. So

• 57:50Speaker 01

  Yeah, I mean, if you're Jensen, don't you just blindly give

• 57:52Speaker 00

  Yeah.

• 57:52Speaker 01

  that amount of compute?

• 57:53Speaker 00

  Really no downside. Yeah, like if he wants to, I mean, it's just he's just gonna go spend it on GPUs. So yeah, it'll be interesting. And I'm sure like, folks have heard the rampant rumors, which seem to be fairly substantiated that SSI is trading in public markets. And that's kind of like the use case that they're targeting. And it makes sense that it's, you know, it's an example of being a very lucrative use case for frontier models where you don't need to expose what you're building to the outside world. So

• 58:20Speaker 02

  Although that aside, why take the why take the investment if if if they actually have a trading platform that's working so well? Unless if it's about actually just getting access to the compute.

• 58:29Speaker 01

  Now, it has to scale. They've seen early signs that it can work, but they'll have to they'll have to scale to really to really solve. But yes, maybe if with this $5 billion in compute is the last round, they have to raise from from from the outside.

• 58:42Speaker 02

  Yeah, all I'll say is SSI is the most secretive out of everybody. I certainly don't have any any special insight into it. So who knows? But, you know, Elia is a pretty smart guy, and has clearly been right many times in the past, and has been right when many others told him he was wrong many times in the past. So

• 59:01Speaker 01

  It's truly so impressive. And like the incestuous gossip circle that is Silicon Valley, that they have managed to keep this as under wraps as they have.

• 59:09Speaker 00

  And I completely agree with that. And just to add to that, not only have they managed to keep it under wraps in this incestuous gossip circle, but they kept it under wraps even though their CEO left the company and went to Meta, went to another hyperscaler. And somehow it still has a it really is incredible.

• 59:26Speaker 02

  That makes me think that whatever the key breakthrough came came happened after Daniel Gross left. I have no idea. But that that that would be the most parsimonious explanation of those events, I think. Because it's also kind of was surprising, like, why would Daniel, why would the CEO leave? You know, unless if there was some conflict or something like that, which who knows? But just purely on the face of it, it would make more sense if kind of whatever the key breakthrough was happened afterwards.

• 59:52Speaker 01

  You know, in other in other headlines, we have OpenRudder, which is rumored to be getting acquired by Stripe for $10 billion. Obviously, at the peak of folks talking about, you know, open source models and, you know, routing, you know, tasks to the to the right cost and capabilities trade off. What do you guys think of the acquisition? What do you have done if you're Stripe? Rob?

• 1:00:13Speaker 00

  I kind of like the strategic rationale and thinking for Stripe. I mean, you know, to this point, Stripe as a business has been so successful because they basically take a cut of all transactions on the Internet. They sort of serve as this toll booth. And an open router is a different form of that business model, right? Like, they're taking a cut of all model calls and model usage across different models. And if you if you think of tokens as a like, you know, a future form of currency, essentially, like, I think it's a it's a conceptual leap. But I think it I think it makes sense for Stripe to move in this direction. And so, yeah, I think it's I think it's creative. I like I'll be curious to see if the if the transaction actually closes. But I think it's, yeah, I think it makes sense for Stripe.

• 1:00:59Speaker 01

  It'll be interesting to see whether like, there ends up being any sort of like differentiation or brand power to this space, right? It feels like everybody in the last week has announced they're building a model router. I'm not sure totally the amount of technology that goes in under the hood to make these things work.

• 1:01:12Speaker 02

  Yeah, it's hard for me to see the the long term moat that's there. I mean, like, presumably, it's the routing function. Like, that is going to be the most critical thing there. That's going to change constantly. So it's kind of hard to imagine a fully durable moat there. The one question I would have about the Stripe, I like the way you framed it, Rob, of kind of like, they were taking a percentage of the Internet, and now they took a percentage of all tokens. The thing I don't quite understand that I would usually look for in an acquisition, you know, is what's the nonlinear additive effect? You know, what's the synergy necessarily between Stripe's business and OpenRudder's business, where combining them together makes something, you know, more powerful than either independently?

• 1:01:54Speaker 02

  I don't see that here. It feels like more it's just like a fully kind of independent business that you can make a nice analogy, the way you drew for it. But I don't see how Stripe's existing product offering or infrastructure or talent or core competency makes OpenRudder more successful. That, I think, would be like, that, I think, is the gap. You clearly going back to the conversation of XAI and and Cursor, like, you can clearly see, at least for XAI, why buying Cursor helps them to make their core offering much more effective. I don't see that for Stripe and OpenRudder. But like, revenue diversification could be

• 1:02:35Speaker 01

  Getting in the AI game, right?

• 1:02:37Speaker 02

  And getting in the AI game. It's a foothold that it enables you to kind of have a fairly low risk, you know, possibly high upside, subject to the commoditization risk that you just raised, Jacob, that there that that makes a reasonable amount of sense.

• 1:02:52Speaker 01

  And hey, if the ultimate eval for a business is how much money they make doing something, maybe maybe Stripe pulls the keys to the to the evals that actually matter for for routing these models. I guess, you know, I do want to hit before the quickfire round, just like the the broader venture world and what's happening, given that we have two thirds of this podcast in VCs. Rob, you know, I know you're always a big BCI guy, big robotics guy. Where are you spending time these days? And what do you make of all of us now becoming, you know, genius NeoCloud public market investors and, you know, exclusively hardware focused?

• 1:03:20Speaker 00

  As you said, the prevailing sentiment in VC world right now certainly is to be leaned in on deep tech. And as a as a reaction to the fear that the frontier labs are eating the world of software and Anthropic and OpenAI are moving up the stack, as we've talked about, and, you know, no, no app category is safe. So deep tech is certainly fashionable. Definitely spending a lot of time in robotics. And I do think like, we could have a whole other discussion on this topic. But I think, you know, both of us, Jacob, are invested in foundation model robotics companies. And I do think that in just the past few months, these models have really started to work in a powerful way. And so I think, like, we're

• 1:04:03Speaker 00

  on the cusp of something really exciting happening in robotics, I think, which is which is going to have years and years to run. So that's exciting. I'm spending more and more time kind of on the physical infrastructure underlying AI, data centers, energy grid, etc. And there, you know, at the end of the day, I think there's like, there has literally never been a larger capital investment supercycle in the history of civilization than what's happening right now in the physical AI infrastructure build out. You know, it dwarfs the space race in the 20th century, dwarfs the Manhattan Project, it dwarfs the railroad build out.

• 1:04:38Speaker 01

  Obviously, back in the day, the venture was all about like, you know, networking and a bunch of this kind of hardware. And then, you know, we had like, you know, a good good SaaS run for many years. And now what's what's old is new again. And I think all the all the founders of Redpoint think it's hilarious that we're spending all this time like now back in in areas where we honestly need to call them with decent frequency. And, you know, it has come come full circle with what's given the value its name.

• 1:05:04Speaker 00

  Yeah, I think it's great, honestly. I mean, I think it's it is venture capital at its most pure is like, real technology risk and real technology differentiation. And so, yeah, I think I think it's a really exciting

• 1:05:16Speaker 01

  I think venture capital's most pure was a company going one to five, you know, with like, some good net revenue retention and just like thinking they could hit 15 next year. Yeah, that's pure. That's pure. Well, you know, I want to end our episode with just like checking in on some some predictions we had previously. And so, Rob, I want to put you on the hot seat, because Ari's honestly on kind of on a hot streak. I feel like he's, you know, his open source models take looks pretty good. You know, his take on on the models, limiting access to their APIs, we'll see. But like, definitely seems like it could happen.

• 1:05:47Speaker 02

  I think my timing's off on that. But I'm pretty confident in that directionally. I think it'll happen in 27, though, probably not 26. But

• 1:05:54Speaker 01

  Rob, I'll put you a little more on the hot seat. Like, you've been very consistent with your Google take. And it just feels like things keep getting worse and worse for these guys. Are you still you're still holding holding out hope and being a bull?

• 1:06:08Speaker 00

  Obviously, I gotta I gotta get Ari to start writing my my Forbes prediction column for me. Google, yeah, it's a

• 1:06:17Speaker 00

  It's like, I would say long term, yes, I continue to be bullish. I just think that they have so many advantages.

• 1:06:24Speaker 01

  Pushing out long term, right?

• 1:06:27Speaker 00

  Yeah, at some point after we're all dead. No, I mean, we've, we've talked about all the structural advantages they have. But they, I agree with you. Like, I, I think like the week after we released the last podcast where I like, again, gave it impassioned defense of Google. Both John Jumper and Noam Chizzi are left and like, you know, they have such a deep bench of talents that like, that's not fatal for them, but people matter so much. People are the most valuable asset and, you know, Noam went to OpenAI and, and John Jumper went to Anthropic and that does hurt. And so, you know, do I think that they will be successful? Do I think that they will continue to be one of the very small handful of

• 1:07:05Speaker 00

  labs at the frontier? Yes. But they have a lot of challenges related to being a big company. Like, I think it's fundamentally related to just being such a massive organization. There's so much bureaucracy and slow movingness and process and so forth. So they definitely have, have their issues.

• 1:07:20Speaker 01

  When, when are they getting back to the frontier?

• 1:07:22Speaker 00

  I don't know. I, I don't know. I think, I think they are basically at the frontier. Like, I, I don't think of that as, I think the, I think Anthropic, OpenAI and Google are basically at the frontier together. Depending on what benchmarks you use. Um, I don't know.

• 1:07:36Speaker 02

  There's a meaningful gap in coding.

• 1:07:37Speaker 00

  You're right, on coding. But I, yeah, I think of that more as like a product prioritization and focus thing than like the core intelligence lagging.

• 1:07:45Speaker 02

  So the thing that's confusing to me is, you know, because I've also been fairly bullish on Gemini and on Google, is that it felt like they had this initial really kind of mess up with Bard prior to kind of rebranding to Gemini. They tried to rush something out and then they spun it up and then it seemed like they had the ship pointed in the right direction. And I, you know, I think it was like Gemini 2.5, that was the first one that started to look like pretty good. And like, you followed the trajectory and it seemed like three was going to, you know, make a big impact and so on. And, and then they haven't, they, they've underperformed that trajectory pretty meaningfully over the last six months.

• 1:08:21Speaker 02

  And I think that's been especially noticeable, given the extent to which, you know, OpenAI and Anthropic have accelerated, you know, dramatically over that time period. So I'm really, I'm really puzzled. Like, it, it seems like the best answer is just like organizational inertia and politics and all the standard big company stuff. Like, I'm very curious to read the book whenever it comes out in two years of like the inside story of Gemini and, and what, what happened there.

• 1:08:47Speaker 01

  Because we're holding two truths at once. Like, you know, I think what you say about, hey, you know, with the right, you know, with the right data and like the recipes are kind of known and people can get, you know, pretty near the frontier and lots of folks are able to do that. Holding that. And then at the same time, it's like, Google has every advantage that they should have. I think coding is a top priority for them. I mean, I think Sergey himself, right, is, is on the, on the team, uh, working on it night and day. And yet they still can't seem to get anywhere close to where OpenAI and Anthropic are.

• 1:09:12Speaker 02

  I think there just has to be that they're getting in their own way in various ways, you know? I, I just think that's the, that's the only explanation. One thing I will say though here is that I think that like people tend to over focus on like, not talent as a general concept, but as on particular famous people. Like Noam Chizzi, John Jumper, of course, incredible scientist, you know, AlphaFold would not exist without John. Um, and like that entire line of research, um, you know, there. But I think we fall way too prey as a society to the great man theory of science. Um, and it's something that I just, you know, firmly believe does not exist. There are, you know, thousands, at this point, thousands of extremely talented researchers who can do an awful lot.

• 1:09:54Speaker 02

  Um, and obviously, you know, there are these exceptional folks, um, but there are a lot of really good people at Google as well. I think the thing that, that those people are, I think, are also leaving because they're frustrated by the, the lack of pace. And like, they don't have the famous names, but you lose enough of those people and that starts to make a really measurable impact and really starts to dole forward looking prospects.

• 1:10:17Speaker 01

  And then Rob, but you know, we got to do an, an opening eye check in your, your, your, honestly, I, I feel like have definitely that, that take seems, it was very spicy when first introduced that, that Sam would be replaced as CEO. You still feeling good about the, uh, the Brett Taylor theory?

• 1:10:30Speaker 00

  We'll, we'll see, we'll see. Um, I, I see Fiji officially stepped down, which I think had been coming for a while, given her health issues and we'll see. I mean, like Sam Altman has an unparalleled ability to hold on to power. And so I don't think it will be easy to unseat him. I think it would be in OpenAI's best interest, frankly, like, and we talked about this last time, like, I do think that having someone like Brett Taylor running the organization would be just such a massive vibe shift in terms of the company's trustworthiness and reliability and, and, you know, overall reputation. And I, you know, they're, they're obviously, Brett is the, is the chairman of the board, like, I've, you know, how much would OpenAI have to pay to acquire CRF?

• 1:11:15Speaker 00

  Who knows? But like, I, I, I rank it, I would still rank it as non-zero. It'll be a very interesting, uh, next several months.

• 1:11:22Speaker 01

  Well, all right, Rob, this has been a, a ton of fun as always. I appreciate you letting me, you know, constantly put you on the hot seat with, uh, with prediction check-ins and, uh, probably, I hope it doesn't disincentivize you from juicy ones at the, uh, at the, at the end of this coming year, knowing that you have to talk about it every two months. But, uh, it has truly been a pleasure and, and appreciate you guys taking the time.

• 1:11:40Speaker 00

  Thanks for having us.

• 1:11:41Speaker 02

  Great to be here as always, and always a fun conversation.

• 1:11:43Speaker 01

  I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.