AI Vibe Check: The Actual Bottleneck In Research, SSI’s Mystique, & Spicy 2026 Predictions

Ari Morcos and Rob Toews return for their spiciest conversation yet. Fresh from NeurIPS, they debate whether models are truly plateauing or if we're just myopically focused on LLMs while breakthroughs happen in other modalities. They reveal why infinite capital at labs may actually constrain innovation, explain the narrow "Goldilocks zone" where RL actually works, and argue why U. S. chip restrictions may have backfired catastrophically—accelerating China's path to self-sufficiency by a decade. The conversation covers OpenAI's code red moment and structural vulnerabilities, the mystique surrounding SSI and Ilya's "two words," and why the real bottleneck in AI research is compute, not ideas. The episode closes with bold 2026 predictions: Rob forecasts Sam Altman won't be OpenAI's CEO by year-end, while Ari gives 50%+ odds a Chinese open-source model will be the world's best at least once next year.

• 0:00Speaker 00

  Can one of you articulate the bull case for one of these Neolabs? Let's say in three to five years, this is the paradigm that matters. It feels like the Neolab that gets there first has a decent chance of being in the oligopoly.

• 0:12Speaker 01

  It feels like it is just a matter of time until a researcher leaves. The one fascinating exception to this is SSI.

• 0:19Speaker 00

  Their level of secrecy goes beyond anyone else. One explanation is that they've been really good at keeping it secret and they do have like the next big thing and Ilya really knows what it is. The other explanation is that they actually don't have anything. We're all a little skeptical that things are too bad at OpenAI these days, but why call the code red?

• 0:33Speaker 01

  OpenAI is burning an insane amount of cash. Google, on many dimensions, is in a better position than OpenAI in terms of the depth of their talent bench, their compute resources, et cetera. I predict that by the end of the year, Sam Altman will no longer be CEO of OpenAI.

• 0:53Speaker 00

  How are you, Rob? Great to see you guys again.

• 0:55Speaker 02

  Thanks for having us.

• 0:56Speaker 00

  Great to be back. Fun to run this back, especially, you know, a week before the holidays. Got to get those those last hot takes in before before the end of the year. And then 2026 is a clean slate, right? If you say anything dumb in 25, it was part of your 25 predictions, right?

• 1:10Speaker 02

  I think that's exactly how it works. And, you know, we got to make sure to give the hot takes for people over that break.

• 1:14Speaker 00

  Yeah, I have something to say at the that the holiday family table.

• 1:18Speaker 02

  Where surely the discussion will be AI, right? At the every every holiday family table is going to be talking about the topics we're going to talk about today, right? That's how everyone else's family is, correct?

• 1:27Speaker 00

  I assume so. I mean, certainly, certainly, certainly in Silicon Valley. A bunch of things I want to hit on today, I think, you know, want to talk about the models themselves. You know, I think lots of questions around model progress, the labs and what's going on there. You know, some exciting developments on the hardware side, the market, and then kind of just zooming out and talking about really what our, you know, takeaways are from this year. Getting a little reflective mood at the end of the year. But I don't want to lose the the the main point here of of recent weeks, which was that, you know, you two obviously just came back from NeurIPS. You know, otherwise known as like the the AI Super Bowl in some ways. I'm curious, like what, you know, both of you, your your kind of main takeaways from there or what you kind of thought of the vibes?

• 2:07Speaker 02

  I was reflecting on this at NeurIPS. This was my 11th NeurIPS. My first one was 2015. When I first went to NeurIPS, I believe it was 2,000 people. This year it grew from 22,000 people last year to 30,000 people. So it grew by more than 4X the total attendance in the first year that I joined, which is pretty wild in hindsight to reflect upon. Also just note, something that I was talking to a number of folks about at the conference and that I was really reflecting on is, I remember when I joined the field, because my background, I originally did my PhD in neuroscience. So, you know, I spent the second half of my PhD trying to kind of redo it in machine learning. But when I first went to NeurIPS, I was coming as kind of someone from a different field,

• 2:49Speaker 02

  trying to really break into machine learning and transition field, which was, you know, non-trivial at the time. And I remember the overwhelming feeling I had was that I was late and that I had missed it. In 2015, my main sense was, oh, man, I've missed AI. It happened already. And the reason I bring this up is that I talked to a lot of students at NeurIPS. And all of them, I think, had this very strong feeling that like, oh, man, it's so big now. You know, it's 15 times larger than it was in 2015. Like, surely, we've missed it. And a lot of anxiety around that. And the thing is, I actually feel now like we are still extremely early. And just like I thought I was late in 2015, and in hindsight, that seems so ridiculous,

• 3:34Speaker 02

  and I was so, obviously so wrong about that, I think we're going to say the same thing, you know, five years down the line. So that was kind of one thing I was reflecting on. The other thing I would reflect on that's changed is, unfortunately, there's just a lot less exciting work at NeurIPS these days. As we've seen a transition from kind of very open work to closed work, NeurIPS has become, you know, much more about PR in many ways than it actually has been about sharing work, right? All the work from the posters, not only is it now six months stale, which is always how it has been for these conferences, but now, first off, like, not all the best work is actually getting in, going to conferences. A lot of it's not being submitted, right? At both Meta and Google, you now have these policies where you can't submit work if it's actually meaningful. There's like this Goldilocks zone where the paper has to be good enough that it could get through review, but not so good that it actually matters.

• 4:20Speaker 02

  for the model, otherwise you can't publish it.

• 4:21Speaker 00

  Yeah, I mean, I guess they say from the Google and Meta publications, you can tell explicitly what they're not interested in or or working on too hard, right?

• 4:28Speaker 02

  Yep. Or what you see a lot of times is people, they do results across scales and then they remove all the large scale results for the paper. And they submit it just with the small scale results. That's like when you have like the researchers who like really want to get the publications and are kind of trying to game the the public system. That's how they go about doing that.

• 4:47Speaker 00

  Yeah. And then it's like join, join the company if you want to see what happens at a higher scale.

• 4:50Speaker 02

  Exactly.

• 4:51Speaker 00

  Yeah. Well, I'll just say it's it's amazing how the event having more and more VCs is correlated with its its relative downfall on the research side. Can't say I'm too too shocked there, though, in this case, I don't blame us fully. It sounds like it's a lot of a lot of the labs themselves, you know, being more reticent to share their research.

• 5:09Speaker 02

  It's a combo.

• 5:11Speaker 00

  Yeah, we're not, we're not blameless here. Let's be real. You know, I do want to start the conversation, you know, for this episode really around models. And I think the biggest question that's been in the zeitgeist the last month has been, you know, are we kind of plateauing with models? Have we reached the limits of scaling laws? Is that, you know, are we kind of moving on to new things? I guess I'm curious for both of you, what do you react to that?

• 5:34Speaker 01

  In my mind, I think the answer is clearly yes, they are plateauing. And I like, I don't see how you could disagree with that. I mean, I think if you just think logically about like the leap in capabilities that that happened from GPD one to GPT two, from GPD two to GPD three. And then from GPD three to GPD four, just the slope of that curve, no matter how you define it, was breathtaking. And since GPD four, certainly the incremental improvements have have decreased a lot. And I think that's just characteristic of what S curves look like.

• 6:15Speaker 01

  Now, part of that, you could say is like most everyday tasks that people use LLMs for, like, you just, it's not possible to get any better at them. Like, if you want to ask ChatGPT or, you know, GPT five for a recommendation for travel or for pet care or something like, it's going to give you as good of an answer as there could be. So I, so that's part of it. But I also think that there are. Like fundamental limitations that the current paradigm models have that aren't being addressed. And I think we'll get into the into some of these topics later in the conversation. But things like continual learning or sample efficiency, like those are fundamental problems that that like we're not seeing improvements happening in the current set of models.

• 6:59Speaker 01

  So I think there's no question that the models are plateauing now. At the same time, like the we're just starting to scratch the surface in terms of economic value creation from the model. So like, as we've discussed, and you hear this said a lot, and I think it's really true, like, even if you froze all fundamentally AI progress, like, there's trillions of dollars of value to be created, and there's like years and years of progress to be made on the commercial front, because like change flowing through these technology changes to the enterprise at scale takes a long time. And so there there's no question that we're in the early innings there. But I do think that that that slowing of model progress is a reality, and it's why

• 7:40Speaker 01

  we're starting to see like more of a blossoming of different technical approaches and research approaches compared to, you know, two or three years ago.

• 7:49Speaker 00

  Ari, are we are we plateauing? I mean, it feels like Gemini three comes out, you know, pre-training scaling is alive and well, coding models seem to be getting better every every month or so. You buy the plateau?

• 8:01Speaker 02

  So I saw someone tweet this recently, and I liked this. It was, we're not in an we're not in an AI bubble, we're in an LLM bubble. And I think there's something to that. I'm not sure if I would even say that we are necessarily an LLM bubble, but I think that people sub out LLMs and AI way too easily for my taste. In general, the entire community is pretty myopic when it comes to focus over focusing on on language models. So do I think AI models are plateauing? Actually, almost certainly not. But I would say that that's if you exclude LLMs. If you're asking are LLMs plateauing? I think there is an argument to be made that that, yes, they are in many cases. I personally find sycophancy to be extremely frustrating in models.

• 8:46Speaker 02

  Like, I want models to argue with me, and it's Actually still very hard for me to get a model to argue with me effectively. So I think there are still like rooms for, there is still clear room for improvement along along these models. But you know, along the standard candidate answer test questions sorts of things, we're definitely seeing a plateau. But that's just consumer L L Ms. There's so much more to AI. You know, look at video models over the course of this year. I don't think anyone would argue that they're plateauing. So I would say models are not plateauing, but maybe consumer L L Ms are starting to plateau with respect to their use cases.

• 9:20Speaker 00

  Yeah. Rob, do you think L Ms in the enterprise are plateauing? Like, I feel like that's usually part of the plateau narrative, right? Not just consumer.

• 9:26Speaker 01

  Yeah, I definitely do think that like, because the like, there aren't, there aren't fundamentally different models being base models being built for enterprise use cases. Opposed to consumer use cases, the same base models, maybe there's some are all done and so forth. But then it's really more like in terms of enterprise adoption, it's really a matter of productizing it and deploying it and implementing it, getting it integrated with the right systems and so forth. So I think like, there is a lot of work still to be done and a lot of like wood to chop in terms of getting these models deployed in the enterprise. He had a lot of progress that will happen there.

• 10:03Speaker 01

  But in terms of like the foundational capabilities of the models, whether we're talking about in an enterprise setting or in a consumer setting, like I think for sure a real plateauing has happened.

• 10:15Speaker 00

  Yeah, I mean, I feel like this is the big question, right? Like the, you know, we were getting improvement in these giant pre-trained models for free that were really unlocking all sorts of enterprise use cases, right? I remember when G P T four came out and suddenly all these domains that weren't addressable previously were, or when reasoning models came out, it was like suddenly like, okay, you can revisit a bunch of these spaces that didn't work. We're no longer getting these like massive general improvements, which maybe is what scaling laws would have predicted absent new chips and massive amounts of compute. So not surprising, but it seems that the big question of the next year beyond diffusion is going to be, can you do really good R L within a bunch of these enterprise domains? And then does that actually meaningfully unlock capabilities? And it might, it might just end up playing out right that we have.

• 10:58Speaker 00

  The capabilities are there today and all we have to do is just discover them. And if you do the right prompting and chaining, like you could probably have this class of models solve almost any enterprise problem. Or it might be that, you know, that RL ends up working really well in some of these specific domains. I mean, obviously, I'm curious for both your thoughts. It seems like RL has worked really well in coding. We have great evidence of like people, you know, improving coding models through that. Have you seen it work really well in like other specific domains, either in your portfolio, Rob, or Ari and some of the customers you work with?

• 11:27Speaker 02

  So there's a really great paper that came out last week that we spent a lot of time talking about. That was really trying to understand how RL relates to the base model, how it relates to the different stages of training. One of the things that I've, that I pretty strongly believe is that we think about stages of training as distinct entities far more than we should. Some of that is just organizational. When you have a big team working on pre-training versus mid-training versus post-training versus RL, it's somewhat natural to have org boundaries and kind of think of, hey, the post-training team is a customer of the mid-training team, which is a customer of the pre-training team. But in reality, this is all training. And like one of the things we've consistently seen, right, is that we take things from later stages in training. We bring them earlier and earlier to training and then things work much better.

• 12:12Speaker 02

  So I think that people really need to be asking this question more of how all these things interplay. One of the things that it pointed out, which I think is very true, is that RL really works well when you're in a particular Goldilocks zone. So you need to be in a case where the model understands enough that it's on policy guesses or have some chance of being correct enough where a sparse reward signal can actually teach you something. So it can't be too easy or too hard. It has to be just right. If it's too hard, then the policy is not going to do anything reasonable at all. If it's too easy, then you might be getting great policies. You're getting lots of reward, but it doesn't teach the model. So RL really works when you're at this edge of capability,

• 12:54Speaker 02

  where the model knows enough to be dangerous, but not enough to actually solve the task effectively. I think what we've seen is that for things like coding, models have gotten to that right Goldilocks zone. I think in a lot of other domains and in a lot of other places, this isn't necessarily true, although it definitely works beyond coding. I don't think RL is a one-trick pony and that'll only work just for coding. But I do think that we tend to see this like generally. Whatever the new hotness is, people are like, that's the answer. Like, we really clearly saw the RAG, say, you know, two years ago. Everything was about RAG. Everyone was obsessed with RAG. Is RAG dead? No, RAG's not dead. Is RAG gonna be a part of many of these deployment systems? Of course it is, and it already is.

• 13:39Speaker 02

  Is RAG the answer to everything? Obviously not.

• 13:43Speaker 02

  We're gonna say the same thing about RL. RL will absolutely be a critical piece of the puzzle. It absolutely has a lot of capabilities that can really drive improvements. It is not a panacea, I don't think. I don't think RL alone is going to solve things. And I think we're gonna see increasingly that the question will become, how do you get models to the point where they are ready for RL in a kind of particular domain or use case? I think for things like computer use, where we're training models on the internet, in many cases, these models are already ready for RL. But there are gonna be a lot of use cases and tasks where I think there's gonna need to be a lot more work to be done to get models to the point where RL

• 14:23Speaker 02

  will work really effectively because they get to that Goldilocks zone first.

• 14:29Speaker 00

  Yeah.

• 14:29Speaker 02

  So I'm still very bullish on it. I think there's a lot to be done here. It's not gonna be the answer all in itself.

• 14:36Speaker 00

  For both of you, like, what are you gonna look for in 2026 to figure out, like, are these models getting better?

• 14:42Speaker 02

  Are enterprises deploying them at large scale? Like, from an enterprise standpoint, at least, like, I think that's the big question. Like, that's not happening.

• 14:48Speaker 00

  But I mean, that could be, that could literally just mean the models are the same and people have, like, chained them together in better ways, right? Or, you know, they've, they've passed security reviews or something, right? I actually don't even know if that's a great indicator that the models themselves have gotten that much better.

• 15:01Speaker 02

  I mean, I don't know, just talking to enterprises, and I'm curious if the two of you have had a different set of conversations. Generally, what I see as the challenge with adopting beyond kind of the two and maybe three use cases where we've clearly seen models work in the enterprise, right? Which are like internal document search coding. And then I think the third use case that's still not as quite as strong as the first two, but starting to see good evidence, is customer support. Beyond that, I don't know if we've seen many huge success stories of models being deployed. And when you talk to As to like, why is that? Why haven't you deployed it? The question, the answer I hear most consistently is accuracy. The models actually aren't good at the task. There's a big gap to where they are that that needs to be get done. Now, I don't know if we necessarily need to do a bunch of new research to figure out how to do that.

• 15:45Speaker 02

  If we conduct no more neat research, it's just about applying the techniques and know how we have already. Could we get a lot of those models to the accuracies they need to be deployable in the enterprises? Like, yes, I actually think we can, absolutely. But I think there actually is a big gap around just like the models themselves aren't good enough at the task that they need to be good enough. I'm certainly not when it comes to the ease of getting a model to there. Maybe you can do that, but it's really, really hard right now.

• 16:12Speaker 00

  Do you buy that, Rob? Is enterprise adoption your main criteria for 26 model improvement?

• 16:16Speaker 01

  I think like, from my perspective, maybe a less concrete, more big picture way to think about it is, like, as we come to the end of 2025, I think there are two or three big buckets of like, new research vectors that are really trendy. But a lot of folks are talking about, you know, we're a big topic of discussion in Europe's that a lot of new startups are working on and that the big labs obviously are focused on. And I think it'll be really interesting to see like, is progress made on any of these vectors in ways that are noticeable to the outside world? Which I think they could be. And so the first category I would say is continual learning, which I already touched on.

• 16:56Speaker 01

  And the basic concept being sort of like moving away from this paradigm where they're separate training and then inference phases, but rather you have a model that can continuously learn as it goes. And as it has new experiences, it can update its weights in real time and and kind of, you know, learn on the go throughout its life, the way a human does. That's not how AI models work today. And it is a real source of

• 17:25Speaker 01

  inadequacy of today's AI models relative to human intelligence. I think like, the big labs have come up with Workarounds and ways to address, ways to get some of the benefits of continual learning by incorporating memory into ChatGPT, for instance, or through like regular fine tuning where you can kind of update a model's weights with some frequency. I'm very interested to see like, are there actual meaningful breakthroughs in continual learning that like make their way to the outside world next year? The second category where there's a lot of discussion and where nothing has been demonstrated publicly, but, and this is maybe even a more ambitious direction, but this notion of recursive

• 18:10Speaker 01

  self-improvement, where models, the kind of AI that can develop better AI and kind of, any narrative of like, you know, super intelligence or a fast takeoff generally is oriented around this concept of like, an AI that can recursively improve itself and then that can improve itself and you sort of get this runaway dynamic. It's a concept that has existed in the AI literature for decades, but it has also become a real focus in the AI research community this year. And there are a lot of new labs that are working on it and OpenAI actually has recently publicly started talking, I think they've been thinking about it for a while, they've actually started publicly talking about it recently with Sam saying like, we want to have a working prototype

• 18:53Speaker 01

  of this like AI researcher next year and like have a full-fledged thing soon thereafter. So I think that's like, if that actually starts working, it's kind of unknowable like how fast that could improve. And so that like, that's, you know, that's one like unpredictable variable that could lead to like actually really dramatic model improvements if you just get this runaway phenomenon. And then I think the last one, which Ari will definitely have some, a lot of interesting thoughts on, is just around like data efficiency, sample efficiency, and this notion that like, If you zoom out, it's kind of crazy today that the way we train our most advanced AI models requires like gigawatt scale data centers that consume like the same amount

• 19:38Speaker 01

  of energy as all of the city of San Francisco and it takes months and it costs hundreds of millions of dollars compared to like the human brain and how energy efficient the human brain is and how sample efficient the human brain is like you see a few examples of something and and you learn it. And so I think like if there are real breakthroughs on the efficiency side of things, which could come from the hardware side of things and like, you know, Naveen Rao's new thing is is focused in this direction, but could also come on the algorithmic side of things. Like, all of those areas are frontier areas, but are areas where conceivably there could be real progress next year.

• 20:12Speaker 02

  So I have a couple. So like, first off, I'm going to put for a moment aside the whole data efficiency point, but I definitely want to talk about that, obviously. Amazing restraint. But I want to start actually with the AI researcher one for a moment because because I think that's a really interesting one. I actually disagree on the AI researcher point of view from like that it's going to make things go so much faster. And the reason I would say this is the fundamental idea behind the AI researcher is that the bottleneck is that we don't know what the best things to do are that will work. And that an AI researcher enables us to figure that out faster, more effectively, more efficiently.

• 20:52Speaker 02

  I would argue that I'm not sure that's true. I'm not sure if OpenAI does not have in the set of ideas that all of their researchers have proposed, a very good idea that will make the next model better. Is the problem that it doesn't have enough ideas or that the cost of experimentation, both in time and bandwidth, is too is too high? I don't think in practice, the bottleneck to experimentation is writing the code anymore, right? It's clearly the the actual GPUs and compute to do it. So if you have an AI researcher who's really strong and is coming up with lots of great ideas, but fundamentally, you still got to go and test those ideas, that's actually a really big bottleneck in this process. So the only way you really start to see this accelerate things dramatically.

• 21:37Speaker 02

  Is if the hit rate of the AI researcher is much, much higher than the kind of group of of of strong researchers that we have within one of these organizations, right? So you'd have to assume that the AI system is going to propose an idea with a much higher likelihood of working than the communal organization would would produce otherwise. And I don't think it's impossible that that could happen, but I don't think people think about it in those terms.

• 22:01Speaker 00

  You basically need to try these ideas at such scale.

• 22:04Speaker 02

  You have to try them.

• 22:05Speaker 00

  There's a limited number of ideas. I mean, I'm really, honestly, it's super analogous to bio, right? And this idea of like, you know, sure, you can come up with a bunch of ideas, but how helpful is that actually? There's a limited number you can actually take through a pipeline all the way through. And so you better be having a much higher hit rate or something that's really, really interesting.

• 22:24Speaker 02

  Exactly.

• 22:25Speaker 00

  And that obviously is a much higher bar than just like increasing the surface area of brainstorm.

• 22:28Speaker 01

  But in in theory, Ari, couldn't you come up with a great idea that doesn't require scale? Like, not every idea around how to create intelligence by definition should require a ton of scale.

• 22:40Speaker 02

  In principle, like, that would go against kind of the whole bitter lesson and most of the learnings of the last five years. Like, there have been a couple of these examples that could do really well in Arc AGI and so on and so forth with really small models. There's been, I haven't looked deeply into those, but there's quite a bit of kind of, I think, questions like, is it actually learning it? Is it kind of overfitting to this particular benchmark, et cetera? I'm not sure if there is a way, like, if we do find one, that is a way that you could break out of this. In practice, with the model, modern deep learning, you know, era, a proposal would be bottlenecked by this scale. So I think the analogy to bio makes a lot of sense. I think the difference from bio, I would say, is that, okay, bio, you have this fixed

• 23:21Speaker 02

  factor that you can only evaluate, you know, X thousand reagents in a given time period or whatever the case may be. So, okay, why do we expect that, you know, AI for science, drug development, sort of targeted, small molecule prediction will work really well? It's because the hit rate of our current systems is so absolutely low, it's point zero zero zero one percent or something, that even if you get it to a one percent...

• 23:43Speaker 00

  You don't think our future AI researchers will be like, the hit rate of human AI researchers in 2025 was so low?

• 23:49Speaker 02

  So, like, maybe. That's the analogy you'd have to draw, though, that you're getting this, like, orders of magnitude improvement in hit rate, which I think is clearly possible for bio. I'm not saying it's impossible for for for research, but I think people kind of think about this really quickly.

• 24:03Speaker 00

  Everyone always thinks it's impossible in their space, right?

• 24:05Speaker 02

  Yeah, exactly. So anyway, not to say that this can't happen. I think it absolutely, there's a clear argument for it being possible. But I think that it assumes that the bottleneck is AI researcher time. And I don't actually think that's what the practical bottleneck is in most situations. And an AI researcher doesn't necessarily fix that bottleneck. Now, if we figured out how to do small scale experiments that are really predictive of large scale experiments and the AI researcher figured that out, that actually could change things in a very way.

• 24:36Speaker 00

  It's fascinating, even with these better chips and big fundraising rounds on the horizon, you still think we're like that compute constrained that these AI researcher would only help so much?

• 24:45Speaker 02

  I think there's still a lot to do. I mean, so going to the second point, actually, this is a great segue. Like, the fact that all of these labs are able to raise infinite capital, I think is actually a huge detriment. And I think it actually limits us in many ways. Look at what China has been able to achieve with far fewer resources than any of the individual big labs have, right? Like, XAI alone probably has more funding and compute than all of the Chinese labs combined. Like, and yet, they have accomplished these incredible things. Why? Constraints breed innovation. To the data efficiency point, we've known forever that there is diminishing returns when you put more data into a model. The

• 25:29Speaker 02

  scaling laws work from Kaplan and others that founded Anthropic explicitly said this: Every time you 10x the data, you get a diminishing return. That's what a power law is. And yet, we are surprised that this doesn't scale infinitely. Well, why wouldn't it? Well, if you're in a position where you have infinite capital, then just getting more data and throwing more compute at the problem is actually a perfectly viable strategy. It might be faster than solving the really hard research problem of how do I actually make these models more sample efficient? That is all we think about and all we do at Datology, and it is a hard problem. There's a tremendous amount of alpha. You know, we're already able to get gains of an order of magnitude or more in kind of sample efficiency and get models to, you know, dramatically better performance in the same compute budget through doing this correctly.

• 26:14Speaker 02

  But the reason we have to do that is that we don't have billions of dollars to just build this. If you do, then the easier path might just be to go and buy data. Even if you look at the first kind of prognostication from Ilya that pre-training is dead last year at NeurIPS. That was entirely based on the idea that the reason that pre-training as we know it will die is because we have run out of data. That was the whole idea, that the internet is a finite resource and we have exhausted it.

• 26:44Speaker 02

  I think that is a very nearsighted way of thinking about this problem.

• 26:49Speaker 00

  I mean, has he already been proven wrong with Gemini 3?

• 26:51Speaker 02

  I think very explicitly he's been proven wrong by Gemini 3, and they did a lot on the data side.

• 26:57Speaker 00

  I thought that, that.

• 27:02Speaker 02

  I will say, he has made some incredible bets that were absolutely right. But like this whole idea assumes that we have maximally abstracted all the juice out of the internet and out of our existing data sets. And that is just so patently false. The analogy I like to use here is imagine trying to hand juice an orange. I'm sure we've all done that. You hand juice an orange, you get some amount of juice out, it's pitiful. You're like, that was so hard to get this little amount of juice. And then you go to a juicer and like you get three times as much juice out of the same orange by doing that. We are in the hand juicing era of using our modern data sets. I think we've clearly seen that one of the ways you can do this is synthetic data. And that that can drive tremendous gains and has now become table stakes for building any model of any sort.

• 27:47Speaker 02

  That you can get way more out of, you know, making your existing data sets much stronger. And I think that's something that Ilya pretty clearly missed. So, you know, why do you have to do that, though? You have to do that when, when that's the, the best path forward. And if there's an easier, more expensive path and money is not a limiting factor for you, you might just take that easier or more expensive path. On the flip side, I think we see the Chinese labs do not have that easier, more expensive path available to them. And therefore they have been innovating left and right around file systems, around curation, around all sorts of things that they have to do in order to make things better. So I think we're entering the era where we maybe have to stop just throwing money against the wall. And actually start really focusing on kind of how do we make these things more efficient? And there's what the good news is that we haven't done too much of this.

• 28:32Speaker 02

  And that means that there's tremendous headroom for growth. And there's so much potential. And this, I think, can really change the game.

• 28:39Speaker 00

  I guess, Rob, because you mentioned it, you mentioned kind of this, you know, big push into continue learning and recursive self improvement. And you've had, obviously, the major labs working on it. But now a bunch of people are spinning out, right? And starting new companies. How do you think about like, does this make sense to do as a new company? Like, why is this not just something the labs are gonna, you know, figure out as well?

• 28:58Speaker 01

  Honestly, I think the, I think the labs are in a very good position to lead the way on things like continue learning and recursive self improvement. And so, you know, I think they're probably in a pole position. I think for the benefit of the AI ecosystem broadly and like humanities, scientific developments, like it's a good thing to let more, you know, let more flowers bloom, let more groups approach these problems and have different ideas. You know, I think there's, there's some argument that, that holds some water around like kind of a version of the innovator's dilemma or path dependency where like OpenAI and Anthropic and Google are honest, like have a certain existing tech stack,

• 29:42Speaker 01

  have a certain existing set of incentives, have a certain existing organizational structure that like motivates everyone to operate in a certain way. A big piece of which is like increasing scale and to Ari's points around like not being resource constrained and kind of just taking the easiest path, which is more and more scale. And a brand, a brand new startup with a bunch of super talented people can, can start from first principles and can think about like, starting from scratch, how, how would you go about architecting a system that can learn continually and so forth? I think, I think that's the best argument for why some of these Neo Labs might have a right to win. But I think it's gonna be a real uphill battle.

• 30:21Speaker 01

  Like, I do think that, I think that it will, it feels like the most likely outcome is that those breakthroughs do eventually get driven by the big labs. Or, I mean, the other, the other fact is that like, Silicon Valley is incredibly porous when it comes to ideas. And so I think even if one of these Neo Labs comes up with a breakthrough that really starts working in a way that like is not currently happening at OpenAI, like it feels like it is just a matter of time until a researcher leaves and switches companies or, you know, the secret gets out. The one like fascinating, fascinating exception of this is SSI. And I say fascinating in the sense that obviously

• 31:05Speaker 01

  it's Ilya's company. He's hired a bunch of great people there. And as far as I can tell, literally, nobody outside the company knows what they're doing.

• 31:13Speaker 02

  Yeah. What are your best guesses? What do you think? He's, I mean, obviously, he's alluding to some stuff on Dorkus's pod. What do you guys think?

• 31:19Speaker 01

  I feel like he didn't actually really allude to anything on Dork. I mean, he said a lot of interesting things on Dorkus's pod, but I don't, I don't think he like really hinted at anything concretely in terms of what they're doing. I have no idea. I'm curious if Ari has any speculations. Like, I really don't think anyone knows. And so I think that's like one interesting counter example to the point that like you can keep secrets in Silicon Valley, but it's very much not the norm.

• 31:42Speaker 02

  I mean, I think two comments here. So first off, with respect to SSI in particular, one thing I'll note is that their level of secrecy goes beyond anyone else. Like, you can't bring your phone into the SSI office. Like, you put it in one of those, those bags, like that schools have. Like, if you try to talk to people that work at SSI, like, they are not allowed to comment on AI, period. Like, they could not have this conversation we're having because they might inadvertently, you know, by saying their opinion on the latest Gemini model reveals something. Like, that is the official SSI policy, which I think goes well beyond any of the other labs. I look, Ilya is obviously an incredibly talented scientist. He has called a grand slam multiple times and then

• 32:23Speaker 02

  hit it at a time where everyone, myself included, I will note, were very skeptical. Like, Ilya tried to convince me in 2018 that scaling was the way, and I was like, absolutely not. It's not going to work. Like, there's no way you're going to get a compositionality just by scale. I was wrong. It clearly works. Like, he was right. Like, you know, it was it was tough to internalize that and accept that. The bitter lesson was indeed extremely bitter. And that's a lot of why I shifted to data was I was like, well, crap. Like, it turns out inductive biases are not actually the answer. And like, all that matters is computing data. And like, I don't know how to make GPUs go burr. So I should probably work on data. And, you know, here I am. You know, so clearly, like, I think you should not count Ilya out. You know, just like I would say, generally, you shouldn't count Zuck

• 33:08Speaker 02

  out. Like, there are certain people that just have such an amazing track record that they've done really incredible things. That said, when someone, you know, in general, like, makes big claims without any supporting evidence, like, generally, I don't believe anything until you see it. So, like, one explanation is that they've been really good at keeping it secret, and they do have, like, the next big thing, and Ilya really knows what it is. The other explanation is that they actually don't have anything that's the big next thing. And the reason it hasn't leaked is because there's nothing to leak. I don't know which one it is. I would not be shocked at all if it's the former and they do have something incredible. I would also note, I wouldn't be super shocked if it turns out there isn't anything there. So I don't know. I think this is, in general, just like the downside of being so aggressively secret.

• 33:50Speaker 01

  Can I share one very funny anecdote that I heard recently? I won't attribute it. I would, I just thought it was a great summation of like, the mystique. It's almost like caricature level mystique around this company. So I met someone who had interviewed at SSI, didn't end up taking the job, didn't actually, like, didn't know anything specific about their building. But he said, when he interviewed with Ilya, Ilya told him, there are two words, which, if I told them to you, you would understand what SSI is building. But I can't tell you what those two words are.

• 34:29Speaker 01

  I'm so curious to know what those two words are.

• 34:33Speaker 02

  Spoiler alert, continual learning.

• 34:37Speaker 00

  On the continual learning theme, Ari is the only one of us that's ever been allowed close to a research lab to actually work there. I guess, what do you buy of this like Neolab versus, you know, the kind of traditional folks? I think the argument being like, well, maybe these net new labs, you know, they can really be focused on this, build from first principles versus, you know, I'm sure OpenAI and Anthropic are like, we're happy to have teams go spin off and think about these things from first principles. So what do you make of these new folks going after these spaces?

• 35:05Speaker 02

  Well, continual learning is definitely a problem. Like, I actually think the kludges are pretty darn good for consumers. Like, I'm not sure if actually memory is fundamentally not sufficient for a lot of these use cases. It very well might be. It is easy to get a model into like a bad minimum and like frequently I actually find that like memory annoys me and I want to go into cognit- cognitochats to like have it stop using my memory or things like that. But like clearly it's a problem. Turning the question around a little bit, can one of you articulate the bull case for one of these Neolabs? Okay, so they come up with the real solution to one of these major problems. How do they go from that to having a great business? Like, is it just, is the bull case like, oh, well, they come up with

• 35:49Speaker 02

  a great solution, they like show it and then OpenAI buys them for a lot of money? Like, is that the bull case? Or is there actually a bull case for how these companies get to be an independent business? I personally don't know if I see one, but I also haven't thought about this nearly as much. So I'd love to hear.

• 36:05Speaker 00

  Well, similar to OpenAI, they ask it how they should make money, right? Once it's been produced.

• 36:10Speaker 02

  That's SSI, not that will do that. But yeah, you just ask the god to tell you how to make money and then it does it.

• 36:16Speaker 00

  I think the bull case would be, obviously, it's not some like, you know, binary moment where like you either have it or you don't. Like, I think ideally you're making some progress along the way, which might help you embed within, you know, there's a whole set of companies, right, that are doing more kind of fundamental research or science or discovery that you might be able to embed in because you have a different class of models from like a chatbot. And so maybe along that journey, you've, you know, you have a six-month lead or a three-month lead or whatever it is, but that then embeds you in a bunch of these enterprises. And over time, you're kind of continuing to build more and more things for them. But I doubt people are actually thinking about that. I think what they're thinking is, let's say in Three to five years, like this is the paradigm that matters.

• 37:00Speaker 00

  The first and one of the Neolabs is first. Even if other people copy it, there's going to be this like oligopoly of companies that are like, you know, you've seen it with OpenAI, Anthropic and Google now, right? Or like the oligopoly and maybe X are the oligopoly in this in this kind of paradigm. It feels like the Neolab that gets there first has a decent chance of being in the oligopoly. They're not going to be the only one. I really doubt, you know, somewhat over saying earlier that these ideas aren't going to spread. But it feels like they have a decent shot at being. And what we've already said in this current paradigm is apparently at least a minimum $300 billion valuation. You know, oligopoly to be in. If you can kind of get yourself in that next one, not a not a terrible bet.

• 37:37Speaker 02

  A couple of questions I'd ask, because first off, I think there's a very big situation, very big difference between the situation in which OpenAI and Anthropic started and right now. Right. There is an entirely new product surface there that was completely novel that OpenAI and Anthropic and Gemini and others were able to get to before it had fully penetrated. I think that my understanding of many of the Neolabs is that they're still aiming for the same product surface. I think like, you know, a chip company is a different product surface. I think that changes it out, you know, like someone like Lila or Periodic, where they're trying to solve an entirely different problem than the big labs. I think I can clearly see why there'd be a bull case in those situations. If the goal is to eventually make a better consumer chatbot or make the consumer chatbot better, it seems very hard for me to see how that that works.

• 38:21Speaker 02

  And then going to the enterprise, you mentioned, okay, maybe you're three months ahead of time. Maybe that enables you to penetrate. Enterprises are two, three years behind. Like, there's no way they're going to choose a startup over OpenAI because a startup is three months ahead in practice, I think. So, you know, I really think that like, in order to see this argument, you have to be able to say-

• 38:40Speaker 00

  Maybe, though, like honestly, I don't know. How far ahead are OpenAI and Anthropic of of Microsoft, Google and Amazon in model capabilities?

• 38:49Speaker 02

  Somewhat like relative to Google, not that far ahead of anything.

• 38:52Speaker 00

  But even Microsoft and Amazon.

• 38:55Speaker 02

  A little bit ahead, probably not a ton ahead, I think.

• 38:57Speaker 00

  And people seem to be going with OpenAI and Anthropic over Microsoft and Amazon, who would have been the natural partners.

• 39:03Speaker 02

  I think that's true. At the same time, if you're using an API, that's a little bit different, right? Because you're just swapping it out in this pretty quick way. Maybe there's something around this. But I don't know, I would be skeptical of that argument, that you'd really see that shift. And then to Rob's point about people move so quickly, look at reasoning models. 01 came out, what, in September? There were amazing open reasoning models within three months. I think it's extremely hard to keep a legitimately groundbreaking discovery secret. It's incredibly challenging. Also, just realistically, I used to believe in this great man theory of science, which is very common, right? This is this idea that like, science is waiting.

• 39:44Speaker 02

  And then every so often, a super brilliant person comes along and sees a connection that wasn't there before, and now all of a sudden, humanity leaps forward. You know, people love to think about Einstein as one of these people, or von Neumann, or whatever.

• 39:59Speaker 02

  I have been firmly disabused of the notion that this is true. And I think the best evidence against this, and I would encourage any of the listeners who are interested in this to actually go and look at this. Go to the list of multiple independent discoveries on Wikipedia. And what you'll see is that consistently, every major innovation in history was discovered by at least two people separately within a year of each other. It's crazy. Calculus being Leibniz and Newton, that is the rule, not the exception. They are all like that. Every major innovation was discovered by several people within a year or two. And then what you see is that as you start to get into the 20th century, this happens less and less frequently because communication bandwidth starts increasing.

• 40:42Speaker 02

  And the ability to not know that someone else in your field has made a groundbreaking discovery within a year goes to zero as you start to get to the internet. So like, this doesn't happen anymore, because now, as soon as you do it, you put it out there. But this happened constantly. This really led to me believing that ideas get to a point where they are ready.

• 41:01Speaker 00

  Yeah.

• 41:01Speaker 02

  Like, you have enough of the component pieces that you can now do the thing. I don't actually even think that the reason reasoning models went so fast is that people from OpenAI leaked it and then other people figured it out. People in every lab were already trying these ideas. OpenAI just happened to get it to work a little bit faster than the others. But the reason that you saw this fast follow effect was because, first off, the existence proof is always really nice. But second of all, they already were working on this. There are already a lot of internal efforts. For the next big thing, I think the same thing is going to happen. I don't think you're going to see the same sort of divergence where like one company gets it and nobody else kind of has that information.

• 41:38Speaker 00

  I agree, but I still think someone can build a brand around being first and like actually go to market pretty effectively around that.

• 41:45Speaker 02

  Possibly.

• 41:46Speaker 00

  Well, I guess shifting over to the labs, you know, since we've been talking about whether whether OpenAI will be able to, you know, compete in this new paradigm, potentially coming in 26. In the current moment, we got a code red, right? I guess that was in the last few weeks. I don't know. It's like, you know, everything moves fast in AI world. But we had the code red out of OpenAI and, you know, because we're all really discerning analysts of the industry, it seems like we're all a little skeptical that things are too bad over at OpenAI these days. But why call the code red? Like, if given that dynamic, like, what do you make of it, Rob?

• 42:19Speaker 01

  Yeah, I mean, I think OpenAI has been such a uniquely charmed company for the past few years, basically since ChatGPT launch or even before that. It kind of, you know, had this mystique around it. It could do no wrong. And it felt like it was always one or two steps ahead of everyone else. And so I think the like, this is an interesting moment in the sense that like, OpenAI is kind of coming back down to earth a little bit. And people are realizing like, oh, they may not actually have this like, special unique sauce that no one else can replicate. Like, in fact, Google on many dimensions

• 43:01Speaker 01

  is in a better position than OpenAI in terms of the depth of their talent bench, their compute resources, etc.

• 43:09Speaker 01

  And so I don't, I think the OpenAI Code Red story has been overplayed. Like, I don't think the company is doomed. I mean, they still have, like, they have built the defining consumer product of this generation. But I do think that like their aura of infallibility maybe was shaken a little bit. And one other important point I'll make on this that like could have proved to be more serious for them down the line. Like, I mentioned some of the other advantages that Google has over OpenAI. I think maybe one of the most significant ones is the cash position. The fact that Google has hundreds of billions of dollars on its balance sheet. It's throwing off tens and tens of billions of dollars every year.

• 43:52Speaker 01

  It has a lot of cash to pursue its goals. OpenAI is burning an insane amount of cash and therefore is completely dependent on capital markets for its continued existence. Actually, I saw an interesting graph recently that showed like the like most cumulative capital burned by companies prior to becoming cash flow positive. And like the biggest one in history to date was Uber, which took like $40 billion of cash before it became profitable. Obviously, now it's profitable. Tesla was like $10 billion and then it became profitable. OpenAI, according to its own projections, what it is telling investors is like $150 billion or something like that before it becomes profitable. And that that is projected to happen in like 2029 or something like that.

• 44:37Speaker 01

  And so I think that is maybe the scariest thing for OpenAI is like they don't, at the end of the day, control their own destiny in the sense that like they do need to continue tapping capital markets. And Sam Altman has proved himself a genius at fundraising and there's a lot of money around the world that wants to invest in OpenAI. But like, if there's a meaningful transition in investor sentiment, then OpenAI is in a really tough spot that Google is not. Because Google can continue to fund themselves in perpetuity.

• 45:06Speaker 02

  I think that's a great way to analyze this, Rob. I think that like I view this Code Red as almost like the end of the beginning in some ways, right? It's like, look at where OpenAI's relative market position was a year and a half ago. The model was so obviously better than anything any competitor had, both closed and open source. Now we get to the point where obviously the closed models, both from Google and from Anthropic.

• 45:31Speaker 02

  You know, and others are getting better and better and are very close, if not better on many tasks, you know. So already now they have this strong closed source competition, but also they're now facing real open source competition. Like, a lot of these Chinese models are, you know, just as strong as ChatGPT was less than six months ago. That's a, it's, I think it's unassailable that OpenAI is in a weaker relative market position today than it was a year and a half ago. So I kind of view this code red moment kind of similar to like, you know, Zuck's we're going to war moment when Google Plus came out. I think it's an acknowledgement that OpenAI can't rest on its laurels anymore, that it has to really be competing and there has to be this clear hustle.

• 46:14Speaker 02

  Because like, I think Google has a ton of structural advantages. They're also vertically integrated, right? Like, OpenAI is trying to get vertically integrated, but Google's on like the ninth or 10th generation TPU. They started this in 2016 and TPUs are good. They're, they're, they're not that flexible, but for the things they're good at, boy, are they good. You know, and that gives, and Google, like the combination of that, the cash position, the talent, like clearly there is a real threat. And distribution advantages are absolutely real. And like, my expectation is in the long run, the most used models will be the ones that are default on phones. I think that generally makes sense. Now, I do think that I wouldn't be so worried about the code red because realistically ChatGPT

• 46:57Speaker 02

  has brand velocity and has escaped kind of, you know, any concern there. Like, everybody in the world knows what it is at this point. It's not going to go away because like, oh, the model on my phone is strictly better. That said, there will be a lot of people who, just like Apple Maps, you know, was able to take over a lot of market share over time. The same thing will happen here, I expect 100%.

• 47:19Speaker 00

  I mean, obviously, one of these indicators of like just the brand position OpenAI is in is this big deal they announced with Disney this past week, right? And the, you know, the fact that Disney has decided to exclusively partner with them for some of their IP and putting into their own video models. And it seems like that, that's really a harbinger of the future of a lot of these image and video models and how do you differentiate it's like with interesting IP. I mean, even in the music space, you saw Suno start to do this with some of the labels as well. What'd you both make of this like Disney OpenAI deal? And why, you know, why does it make sense for Disney to go exclusive with one of the model providers here?

• 47:51Speaker 02

  I think really smart for both sides. I mean, the reason it makes sense for Disney is that Disney gets to invest at a favorable valuation that Disney thinks is going to make the company a lot more valuable on the basis of being able to use Disney properties. I think is where a lot of Disney's value comes from. I think from OpenAI, it's obvious why they do this, but I think there's a clear upside for Disney as well. If the deal is big enough, if they think that they can make enough money from it, and I assume that exclusivity is still somewhat limited time, like, everybody knows we're still early, like, if in two, three years or whenever that expires, they want to shift to another model provider, they will gladly, they will easily be able to. And for the time being, I think it's honestly a very smart deal for, rather than letting there just be a void and people doing this

• 48:35Speaker 02

  in a way that is not permitted, actually providing a clear permitted path to enable this is, I think, really smart. And I think suggests that some of these legacy companies are learning from like the transition to streaming where there was like this long in-between for a lot of companies where they're like, oh, we're going to resist it, we're not going to actually get involved. And then that just like created opportunities for piracy and things like that. I think you would see similar things in the AI market, and I'm actually pretty surprised by the forward-looking thinking to actually say like, no, we actually have to have a clear view and a clear way for people to generate content with our licensed properties, you know, beforehand.

• 49:11Speaker 00

  Rob, another deal that made headlines on the OpenAI side was this Thrive Holdings arrangement, right, where they're working super closely with them. Obviously, interesting for Thrive and its portfolio of companies, but why do you think OpenAI was excited about that?

• 49:23Speaker 01

  Yeah, it's an interesting deal. I mean, I think from OpenAI's perspective, it's another example of the reality that OpenAI, first and foremost, is a technology company. It's building state-of-the-art models. It's very interested in helping those models be deployed far and wide across the economy. But first and foremost, you know, it's not a implementation company, it's not a professional services company. And so I think they've found, they've struck a lot of different partnerships that helped them kind of extend their reach through collaboration. So like, a couple of years ago, there were a lot of partnerships announced between OpenAI and Anthropic and like, Daimen Company and McKinsey and BCG.

• 50:05Speaker 01

  More recently, this startup Distill has attracted a lot of buzz as kind of the like palantir for OpenAI to help implement OpenAI's models. And from OpenAI's perspective, I think this partnership with Thrive Holdings is another example of that, where they will work with this holding company to help like push out their models to all sorts of arcane and non-tech forward industries that might otherwise not be exposed to OpenAI's technology.

• 50:32Speaker 00

  I think the data is also a piece here. No, 100%. That's what I think it is. I think it's like you, you know, you, you solve, you know, you work really closely in solving some of these like unsexy, like, you know, in the weeds problems and you just get great, you know, data to go and then improve your models, right? It's like, you know, I think the labs are clearly so, you know, really want to work very closely with a bunch of these end application companies to learn, you know, where, you know, to our earlier conversation, like the evals are kind of useless at this point. Like the only evals that really matter when a model comes out is like, what does a bridge think about the model? What does Legora think about it? What is, you know, cognition or cursor? And so I think that like, this is yet another set of people that they can figure out like, you know, are our models trending in the, in the right direction through, through the work?

• 51:16Speaker 00

  You know, I guess we've talked, obviously, a bunch about the models plateauing. We've talked a fair amount about, you know, continual learning RSI and then then OpenAI. I do want to hit some of the other proper nouns before we move into, you know, next year's take. And most importantly, Ari, I want to get like an update from you on, on Meta's superintelligence team, because I feel like you had some great takes on this in the last pod. And every week there's like some, you know, there's reports of drama, you know, departures, whatnot, like, what's your current temperature take on whether this experiment is working?

• 51:44Speaker 02

  I'm probably more bearish now than I, than I was before, although still like, who knows what's going to happen.

• 51:51Speaker 02

  I mean, we've seen a lot of departures already from MSL with some extremely short tenures, which is, which I think is interesting. Definitely heard a lot of challenges around the culture there and around the divide. One thing that I would say personally that has been really sad to see is that first off, there was that big layoff, which was like pretty clearly a purge of everyone that was involved with Lama 4 in any way, shape or form. And like, I think there were a lot of really exceptional people that got tagged in that that shouldn't have, which is honestly just like kind of crazy from the perspective of like, here you're going to pay so much money for all these great AI researchers and then you have great AI researchers that you're going to let go because of like organizational dysfunction that's like not ideal.

• 52:34Speaker 02

  But so I think that's one part. I'd say that like, you know, FAIR is a place that was very near and dear to my heart. I had many years, which I consider, you know, some of my favorite years working at FAIR. And it's pretty clear that FAIR is now kind of on its last legs. And, you know, everyone I talked to at NERFS, for example, from FAIR was deeply unhappy generally, and looking for ways to exit in various ways. So I think that's just like really sad to me personally. You know, I think there's been a lot of recent reporting around the fact that the new model's probably going to be closed source, not open source. I think that likely is true. You know, whether the model's good or not, who knows? We'll see. It very well might

• 53:18Speaker 02

  be. Like, whether that's going to be enough to really recapture a bunch of the gain, I think that's going to be harder. I think a big part of the tailwind behind Llama and Meta was the open source nature of it. It's kind of wild, actually, to look at like Zuck's reputation before and after Llama. Like, it was also it was around the same time that he had the whole makeover and he went to like the chains and the big shirts that have like laughing phrases on it.

• 53:45Speaker 00

  Surfing video in Hawaii.

• 53:47Speaker 02

  Yeah.

• 53:49Speaker 00

  Yes, it was, yeah.

• 53:50Speaker 02

  Yeah, like all that stuff. That was like that one with all the sunscreen, like all these serious things. But like, he had this like kind of image shift around the same time that like Meta started open sourcing Llama and like became the defender of open source and like got a ton of really great press and it kind of rehabilitated Mark's image and now they're going the other way. Does a model that's like just as good as ChatGPT from Meta, but now closed source, like, does it get the same tailwinds? Are people as excited? Do they want it to win? And like, how does that impact things? I'm not sure. So they still very well might produce a good model, but I think we're clearly seeing that there are mercenaries, that there are a lot of organizational challenges and that there has been a lot of drama and like, that can't make it easier to to build a really strong model.

• 54:33Speaker 00

  Well, you know, moving on from the from the labs to other just news stories I want to make sure we hit. I feel like one of the big ones was around Trump allowing H200 sales into China. And so, Rob, I'm curious, you know, what do you make of that?

• 54:47Speaker 01

  Yeah, I think this is a really interesting dynamic that's playing out right now. Like the rewinding a few years during the Biden administration, the big kind of like breakthrough and diplomatic development was the realization that, oh, the US actually has this really powerful choke point in the form of the chips that US companies like Nvidia design and make. And we should just like unapologetically leverage that choke point and and preclude those chips from going to China and that will give us this really strong advantage. And that has worked. That definitely has worked and been effective. I think the the interesting like short term versus long term dynamic that's played

• 55:30Speaker 01

  out here is people have more recently started to realize if we really totally disallow China from accessing any US chips, eventually it will force them to develop their own internal domestic AI chip industry and become more self sufficient. And this is kind of to Ari's earlier point around like constraints breed innovation. And so like today, China's domestic chip industry is nowhere near at the level of Nvidia and others in the West, but it is really improving because it's been forced to improve. And so this latest round is kind of like an interesting like next development in this three dimensional chess game where the the Trump administration, David Sacks in particular,

• 56:11Speaker 01

  decided like, you know, actually, it's in our best interest to give China chips that are like decently good, even though they're not at the cutting edge, they're not black world chips, but they're like as good as any chips in China in order to compete with China's homegrown chips from Huawei and like prevent their their domestic chip industry from advancing to the same degree. And that was kind of whether or not that ends up working, that was kind of the calculus of like, let's let's open the spigot a little bit. And so interestingly, like, I think this is still playing out in the real time, but interestingly, like, it sounds like China may actually reject the opportunity to buy H200s and it may actually say like, we don't want the H200s like for this exact reason of investing in developing their own domestic industry.

• 56:56Speaker 01

  So anyway, this is a game that this is a long term game that will play out over several years. And I think at least for the next few years, there's no question that Nvidia will have supremacy and China will be at a disadvantage because they're not using the most cutting edge chips. But like the CCP thinks in very long term

• 57:16Speaker 01

  timeframes and I think over a long enough term horizon, like it actually will be to China's advantage to develop this internal industry.

• 57:25Speaker 02

  All right. What do you think? Good policy? I agree with everything Rob just said. I'll take an even stronger stance. I think that forcing China to buy handicap chips will be viewed as one of the stupidest things we've ever done. I think it was an immense like example of shooting oneself in the foot and like completely misunderstanding short term versus long term incentives. It makes a ton of sense when you just think short term. And I think it fundamentally stems from a view that a lot of the the West has of China that is not consistent with modern reality. And most people you speak to, and I think this includes the vast majority of politicians, view China still as a copycat factory. China cannot innovate. China is extremely good at taking technology that was developed

• 58:10Speaker 02

  in the West and then copying it and making it cheaper. That's the thing that China is good at and that fundamentally, so long as we prevent them from having access to the technology, that is an insurmountable barrier. That may have been true of China in the 90s and honestly, I think probably was true of China in the 90s. That is so clearly not the China of the 2020s. I tell people all the time, China is innovating in AI. They are not fast following. They are innovating. When a new deep sea paper comes out, it's must read for everybody at the labs and everybody in Silicon Valley. Like they are really innovating and they're not just innovating on the model layer. They're innovating a lot on the infra layer. And that's really effective because that's the not sexy plumbing stuff that actually

• 58:55Speaker 02

  makes iteration speed faster, makes everything better. And this view that, hey, if we handicap the chips, then China is going to end up now with worse chips, like really kind of only makes sense if you have this view that they can't actually develop. themselves. And I think all we've actually done is catalyze. And I think that like it's hard to run the counterfactual, of course, but I would guess that we've probably accelerated China's chip development through this by at least several years. And if you told me it was five to 10, I would not find that to be like shocking at all. I think if we had just sold the chips from the get go, China never would have even considered doing this. But the gap and the we're gonna do it, we're not gonna do it. I think now leads to them realizing like, oh, actually, this is really good for us. And as Rob mentioned, the one thing that

• 59:38Speaker 02

  the CCP is just incredibly good at, and I'll note the one thing that they are way better than any Western government as is thinking beyond a given term and thinking in the long term and making 10, 20, 30, 40 year bets. And this is obviously one of those long term bets. That's like, frankly, a lot harder to do in a democracy where you have regular elections than it is in a more authoritarian government. And I think we're really going to regret this.

• 1:00:04Speaker 00

  Yeah, I just don't know if I buy the counterfactual. Like, I can't imagine, even if we were selling chips freely, it's such an obvious choke point that at some point could have been activated that I don't know if there would be have been any less urgency, you know, from the CCP to, you know, to make sure chips were really good.

• 1:00:20Speaker 02

  I think the question is, if they actually force the model providers to use non Nvidia chips, then they're going to start getting much better, much more quickly. Because then you're going to end up with a virtuous cycle and feedback loop from those folks. So long as those model providers were using Nvidia chips and continue to use Nvidia chips, it's going to really bottleneck their ability to to get the feedback and actually make that cycle fast. And I think probably what we've accelerated is the Chinese government saying, no, you cannot use Nvidia chips. You have to use the internal chips. And that's probably now happening several years earlier than I think it would have otherwise.

• 1:00:54Speaker 00

  Before we we get into our end of year reflections, I did notice one thing in the news, Ari, that I felt like was very relevant for Datology. So I want to make sure I I fully understand the implications, which was that, you know, Amazon announced this Nova Forge, right, which is, you know, making model customizations, you know, for the enterprise. It's kind of this like central part of their strategy. Seems up your alley. Like, what do you what do you make of that announcement?

• 1:01:18Speaker 02

  Well, we were, I think, very excited to see this in general. I think fundamentally, one of the big bets that we've made at Datology and that I've had for a number of years is that we're going to end up in a world where most enterprises are building in some way or another their own models. Now, that doesn't necessarily mean training a model from scratch. That could mean taking an open source model and customizing it on their own data. But that we're going to end up with a set where enterprises have a bevy of specialized models for their individual use cases. I think that in 2024, for example, this view was pretty non-consensus. And that lots of folks were pretty focused on like, oh, it's just going to be one model to rule them all. Isn't just like fine tuning going to work? And what we've seen consistently is that like people tried a lot of small scale fine tuning. They found it doesn't work that well.

• 1:02:02Speaker 02

  They have to go to kind of larger sorts of trainings. And if you look at like, who really knows what enterprises want and need? It's fundamentally going to be Amazon and Azure being in many cases, the best position to see this. So seeing AWS make such a big bet on enterprise customization. And I'll note, do it in a way that is smarter than what I've seen most other folks. Like something that people really miss is that you do not want to start from a fully pre-trained base model checkpoint for any sort of continued training. You actually want to start from a checkpoint that's intermediate in pre-training before the model's kind of annealed and been overbaked. And in practice, those checkpoints actually don't exist for the vast majority of models.

• 1:02:47Speaker 02

  So for example, all the Chinese models, they just released the kind of final annealed checkpoint. They don't release these intermediate checkpoints. Nova Forge, both kind of working with intermediate Nova models and then enabling this for enterprises, I think, is very, very exciting for this. I think they got like one of the key aspects of this correct, that you have to start with an intermediate checkpoint. And seeing them make this big bet, I think, really shows and validates that this is where things are going to go. And this is something that, like I said, I believe for a long time and I think always made sense from first principles. And very cool and exciting to see that this is really starting to happen. And kind of preempting, I think, what one of the things that I think is going to really happen in 2026

• 1:03:29Speaker 02

  is I'm going to think that 2026 is going to be the year that we really see enterprise AI beyond kind of those three use cases we talked about start to really work and start to take off.

• 1:03:38Speaker 00

  That's a perfect transition into our end of year reflections. I'm going to hit you with three questions. I guess two reflecting on the year and then one looking forward. Maybe to start, Rob, what surprised you most this year?

• 1:03:50Speaker 01

  Yeah, this is an interesting question. I think my answer to this is is a specific event. It's the fact that Daniel Gross left SSI to join Meta and into a lesser degree, Nat Freedom as well. But it really, I mean, like, we talked about SSI earlier in the show, but it's such a unique organization. Daniel was in such a unique position, working closely with Ilya. And there's there was such a culture of like, we have to keep everyone in the family. Everyone stays here. We don't have any leakage. We keep all the information. Everyone's very loyal to Ilya. And so to to see him abruptly leave to go to another lab, it was just not

• 1:04:34Speaker 01

  at all what I was expecting. It was very surprising. And I like, I'm, you know, I'm just don't, I'm just speculating here. It's hard to understand the motivations. Like, I think he's made a lot of money. I don't think like, just the fact that Zuck was willing to pay him a ton of money by itself. You know, I wouldn't expect that to be for enough reasons. So that like, that event, I still am kind of scratching my head over. And it was an interesting move.

• 1:04:57Speaker 00

  Yeah, especially because he probably, he probably knows the two words, right? You know, it seems dangerous to.

• 1:05:01Speaker 02

  Right. This is exactly what I was gonna say. I like, like, like, again, knowing nothing about what's going on in SSI, like, if the two words really were the most groundbreaking words, it makes it harder to explain that move, I would say.

• 1:05:12Speaker 01

  Yeah, yeah, I agree.

• 1:05:14Speaker 00

  Yeah. What about you, Ari?

• 1:05:15Speaker 02

  I think two things that I would say really surprised me. First off, just on the Meta front, like, look at the total about face of Lama and Meta around this. Like, Meta was kind of riding high coming into 2025, like Lama 3 released in the summer was like pretty darn good, was kind of a de facto standard. And then I think Lama 4 was delayed right at the end. And then going into like, honestly, Lama 4 was not that bad of a model. It wasn't amazing or anything, but it like mostly underperformed expectations more than it actually like was not a clear improvement over over others. And it got caught in some of the like, really big model scaling. But definitely, I think it's surprising to see like, the purge, the full about face, bringing in Alexander Wang, all this sort of thing, like.

• 1:05:59Speaker 02

  I don't think that was on anyone's bingo cards coming into the year. But the one I would say that like is actually the, in pairs of this, and somebody the biggest one is the rise of China and the rise of the Chinese models, right? Like this time last year, I think DeepSeek V3 was not quite out yet. And R1 came out in January, right? Like, and then seeing Kimi and seeing the Meituan model and Minimax and all of these various models fully in the open really start. That all happened in 2025, which as I'm actually saying this is like, is that actually possible? This wasn't five years of progress? Been a long year. It's been a long year. But like, boy, how has that changed the landscape across the board?

• 1:06:38Speaker 02

  Right, like, who would have expected that going into 2026, the Paragon and the pioneers of open source are exclusively Chinese companies, basically? Like, that is pretty stunning, I think, and would never have been something that I would have predicted.

• 1:06:58Speaker 01

  That's a really good one. Yeah, I can't believe DeepSeek V3 was, was less than a year ago. That's insane.

• 1:07:04Speaker 02

  Pretty wild.

• 1:07:05Speaker 00

  I guess another, you know, 2025 reflection question. What's one thing, Rob, that you've changed your mind on in the last year?

• 1:07:11Speaker 01

  One thing that I have changed my mind on is the timeline on which the big labs will go public. I think like for, I kind of assumed for a long time, these are companies that are not going public anytime soon. They're super secretive, they're, they kind of march to their own beat, et cetera. And I think it's becoming increasingly clear that like both Anthropic and OpenAI are having to start thinking seriously about IPOing. I think it, it seems plausible that at least Anthropic will IPO next year. It was reported recently that they officially hired a lawyer, Wilson Sancini, to, to help them with it. They're like starting to think about which bankers they want to work with.

• 1:07:53Speaker 01

  And it kind of makes sense if you reflect on it, because both companies have taken on a bunch of capital from a bunch of financial investors, including a lot of crossover investors. And so they will start feeling more and more pressure to go public and give those investors a liquidity event. And their businesses are also like, starting to look like real, normal businesses in the sense of like, Fast growing revenue and you know, they have a P and L that you can dig into. And so I

• 1:08:21Speaker 00

  Fast growing revenue, a P and L you can dig into that may have some things that look a little bit different than your average business.

• 1:08:26Speaker 01

  Yes, yeah, maybe unprecedented burn, but but like they're they're becoming real companies. And like, yeah, I think this time last year, I would not have even been thinking about an IPO for Anthropic or OpenAI, but I do think it like either in 26 or in 27, I think it's going to happen for both companies.

• 1:08:43Speaker 00

  That's a good one. What about you, Ari?

• 1:08:44Speaker 02

  I also have changed my mind on that one in the same way. And I'm very surprised by it.

• 1:08:51Speaker 02

  It's funny, if you were to listen to this podcast and maybe our prior conversations, you might come away with the notion that I'm like, you know, extremely bearish on RL relative to others. And I think that might be true, but relative to the other to others is doing a lot of the the hard lifting there. I've become a lot more bullish on RL this year in a way that that is surprising to me. I mean, I think RL is like, if you look at the story of RL over the last like 15 years, at least, it's kind of like constantly the boy who cried wolf. It's like, it kind of works. Oh, look, it's going to be super big. And then that doesn't really work. It actually just kind of overfits or stays in these situations, right? Like DeepMind made a huge bet on RL in like 2014, 2015. And like, clearly, we're early and not right, you know? And I think that like R1 came out in January, right? So like, last year, we were

• 1:09:36Speaker 02

  still firmly in the RLHF era of things. And I think like, a lot of that actually got killed pretty quickly. And we just shifted to like, oh, just buy more direct data and trying to model the humans. And just it turned from RLHF into SFT. And people just kind of made that transition. I will say that RL has generalized and produced gains, certainly beyond what I would have bet this year. I still think it's not the panacea. So I still embarrass on it relative to kind of people saying, hey, this is the be all and end all. But, you know, relative to, I think, what we had historically seen from RL. Clearly, it works in real practical settings now. And I don't think we could say that, you know, a year ago.

• 1:10:15Speaker 00

  It is amazing how on the AI podcast circuit, like a relatively bullish take, still can end you in like the bottom quartile. Like, I remember when Andre Karpathy was like, agents will automate everything, but it will take a decade. People are like, oh, he's, you know, that, that, he's not, he doesn't believe. And it's like, that's pretty sweet if that, if that actually happens.

• 1:10:32Speaker 02

  Yeah, automate all work in a decade? Like, ah, AI is a bust. It's not going to do anything. Like, it's a bubble. Like, only a decade to automate all human work? Oh, what's the point?

• 1:10:44Speaker 00

  Well, we have to end, you know, to give our listeners something, a treat going into the holiday with some hot predictions for 2026. So, Rob, what's your, what's your spiciest?

• 1:10:53Speaker 01

  Yeah, this is a fun topic. As you guys know, I write an annual column in Forbes with 10 predictions.

• 1:10:59Speaker 00

  Oh, yeah, give us a preview. You have, you, I, I, yeah, I totally forgot you're going to be having a piece come out. So what's, what's the spiciest one of those predictions?

• 1:11:07Speaker 01

  Yeah, yeah, I was actually just working on it before this podcast. And yeah, I will, I will give kind of a sneak preview of one of the spicier ones, which I anticipate will get a lot of interesting reactions. I predict that by the end of the year, Sam Altman will no longer be CEO of OpenAI.

• 1:11:25Speaker 00

  Oh.

• 1:11:26Speaker 01

  And here's my reasoning. Part of it is what we've talked about, which is that, like, OpenAI has been this incredibly charmed company for the past few years that could do no wrong. And it now, for the first time, is really like facing some real headwinds. The narrative has noticeably shifted. Gemini Free's debut really changed the view to like, oh, Google may actually have the stronger position here, and OpenAI may be on its back's foot. And so I think that has kind of, that, that, that changes a lot of the dynamics internally at OpenAI. And it's been interesting and funny to watch just like how all over the place Sam and OpenAI have been recently. Like, he's

• 1:12:08Speaker 01

  You know, he's doing this big BCI company called Merge Labs, which he's having OpenAI invest a big chunk of money into, which Sam is going to be a co-founder of. He's talking about doing space, like he wants to buy a launch company potentially. You know, he's been talking about OpenAI doing their own chips, their own data centers, you know, the consumer hardware thing, of course, with Johnny Ive. And I think like, when the company is this darling that can do no wrong, all of that stuff is exciting and ambitious. But all of a sudden, like if ChatGPT's growth starts to slow, or Gemini is taking share from them. Then all of a sudden, it's like, hmm, is this actually the right guy to be running this company? And like, is he focusing on the right things? And maybe we shouldn't be spending so much time on BCI and space and so forth.

• 1:12:49Speaker 01

  I think the other dimension to it is like, I don't know if Sam really is excited about running OpenAI, the business in all of its day-to-day drudgery. And like, thinking about retention stats for ChatGPT and so forth. And I, so I think it was actually pretty significant that he, this year, he brought on Fiji, who is the CEO of the cart, to join as like, her title is CEO of Applications at OpenAI. But I think she is now running a lot of the day-to-day. And so I, and I think that like, he, I think Sam thrives on being like the big visionary and salesperson and not necessarily the like, making the ships run on, making the trains run on time.

• 1:13:27Speaker 01

  And as we just talked about, like, I do think OpenAI is now very much on a path to, to IPO, and that's kind of on the horizon. And it's what, it's investors will be demanding and expecting. And there are all sorts of organizational things that have to go into that, which again, are, are maybe not things that Sam cares about or is the best equipped for. So I, I think there may, this is like an imperfect analogy, but there may be a bit of a like, the transition from Travis to Dara, where like, you just need to get in the straight-laced person who can like, hire the CFO and like, get the quarterly reporting right and like, get the company ready to IPO, which is not Sam. And so like, if, if this transition happens, like, I don't say like, will it be the board letting him go?

• 1:14:11Speaker 01

  Will it be him stepping down? Like, I think probably the public will never know. It'll be framed as like a, a Sam's decision. But I like, it sounds crazy, but I think there's actually a decent chance that by the end of the year, he's no longer running the company.

• 1:14:25Speaker 00

  I love the spiciness of that prediction. I mean, I completely disagree because I think it's like the top of the first inning in AI and you like mean that like the worst thing you could possibly do is start like optimizing in the for for short term things, given how much we talked about continual learning, we talked about RSI, all these like changes afoot, like there still is a big, you know, a big vision to be climbed if we're going to hit Andre Karpathy's automating everything. But man, that you took the prop seriously, which I respect. Ari, what about you?

• 1:14:54Speaker 02

  I think that there is a greater than 50% chance that by the end of 2026, there will be at least one time where clearly the best model in the world is a Chinese open source model. And where there is a general clear consensus that it is better than all the closed source alternatives.

• 1:15:13Speaker 01

  That's a pretty hard take, yeah.

• 1:15:14Speaker 02

  Like, there will be one that people in the know are using for coding assistance above all the all the commodity players. Like, I think that will happen at least once in 2026. And I think that it is not implausible to me at all. Like, I'd make it less likely, but probably like a greater than 20% chance in my mind that like, at the end of the year, the unequivocally best model is not from one of the closed labs.

• 1:15:40Speaker 00

  Well, I'm making a note for for end of next year's episode. We'll have to check back in on these.

• 1:15:44Speaker 02

  Let's see if we are right on this. And if Rob's right, we have to like give him something big because that one's just like such such an out on a limb sort of prediction there.

• 1:15:54Speaker 00

  We might even let him play host for a day if he's right, you know, we'll let him host one.

• 1:15:59Speaker 02

  Wait, Jacob, we have to get we have to get your your hot prediction. You can't make us accountable for this and then you not be on the hook.

• 1:16:05Speaker 00

  I got to be honest, I didn't expect this level of heat. I guess I'll say it feels, you know, paltry comparison. But I think that in the last few years, like, it was like clearly like the age of AI app investing. And like, actually, I think a really bad time or hard time to invest in AI infrastructure because the underlying substrate was changing so fast. I think because of this plateau conversation we finally had, if you look at like the market cap of value that is created in 26, so I'd say 23. 30 through 25, I'd say apps way over infra, especially if you take out data labeling and inference providers. I think in 26, way more values is going to be created on the infra side than the application side.

• 1:16:48Speaker 00

  Because you finally have enough stability in the way people build things for real infrastructure companies to emerge. And you've seen early versions of this in like some of like the web search companies that connect into agents. And like, I think there's going to be like dozens of these categories finally. And so, yeah, I actually think it's all these infra investors in venture land have become vertical SaaS investors because they're like desperate to get out of infra. And I actually think it's gonna be a great time to invest in it.

• 1:17:13Speaker 02

  Well, as a founder of an AI infra company, I could not agree more.

• 1:17:18Speaker 01

  That's a very well reasoned, but not nearly spicy enough take for me, Jacob.

• 1:17:22Speaker 00

  Yeah. I think there's a 70% chance that I'll be removed as host of this podcast for some.

• 1:17:29Speaker 01

  Never, never. That could never happen.

• 1:17:33Speaker 00

  Well, all right, Rob. This is a pleasure as always. I hope it's a great holidays and thanks so much for stopping by.

• 1:17:37Speaker 02

  Thanks for having us as always.

• 1:17:39Speaker 01

  Happy holidays, everyone.