Can Open Models Keep Up With the AI Frontier? | Nathan Lambert

a16z Partner Lisha Li sits down with AI researcher and Trillium founder Nathan Lambert to ask how durable the gap really is between frontier AI labs and open models. Nathan argues that OpenAI and Anthropic have structural advantages in compute, capital, and the ability to pioneer new training paradigms, but open-model labs, particularly in China, have proven remarkably capable of following close behind. They unpack how much of that comes from distillation versus technical talent, why compute remains a major constraint, and how data, RL environments, and post-training are increasingly shaping model progress. They also discuss Nathan’s new nonprofit research lab, Trillium, and its plans to open up parts of AI research that are increasingly difficult to study from the outside. That includes releasing RL environments and documenting how AI is actually being used to automate AI research, with the goal of demystifying recursive self-improvement and giving researchers and policymakers a clearer picture of what frontier systems can and cannot do.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  When I think about model progress, it comes down to different forms of capital, and that is compute, data and talent.

• 0:06Speaker 01

  How do you see that playing out in terms of like, who's gonna win in the next couple of years?

• 0:10Speaker 00

  The Chinese are so good at building models. To say that China is anything but equal on talent, you're gonna have a hard time. What they don't have is compute.

• 0:16Speaker 01

  Most people regard distillation as sort of like the main lever for them to do this.

• 0:21Speaker 02

  People distill my products every single day. They take it, they strip it down to its raw parts. And that's called competition.

• 0:27Speaker 00

  Previous post-training was much more reliant on the supervised fine-tuning. Now this has morphed a bit into environments where it's RL-heavy. But it's just like hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today.

• 0:41Speaker 01

  The discourse now is just so much fear on like how much the frontier is just hidden and also breaking away.

• 0:46Speaker 00

  The AI capability plot doesn't necessarily worry me. But the AI capability plot in the pretty much non-existing preparation of cross-public cybersecurity and all of these things like.

• 0:56Erik Torenberg?

  The Frontier AI Labs are spending extraordinary amounts of money to build better models. So how does open source keep catching up? In this episode, A16Z partner Li Shili sits down with AI researcher Nathan Lambert, founder of the new nonprofit research lab Trillium, to unpack the race between open and closed AI. They discuss why OpenAI and Anthropic still have major advantages in tribute, capital and speed, but why Nathan believes the technical talent behind open models, particularly in China, is often underestimated. They also separate the reality of model dissolution from the rhetoric around it and look at how post-training, data and RL environments are becoming increasingly important sources of progress. And finally, they turn to recursive self-improvement.

• 1:41Erik Torenberg?

  Nathan explains why Trillium plans to document how AI is actually being used to improve AI, from the tasks models can already automate to the places they still get stuck, and why making that progress visible could lead to a much more grounded debate about how quickly the frontier is really moving.

• 2:00Speaker 01

  So Nathan, really happy to have you here. Thank you for joining in person. Super timely, given you're launching Trillium, which I'd love to talk about. But more sort of on a macro level, I think we chatted a lot about open source and that's a lot of what you care about. And what we've been hearing a lot in the public discourse is like, hey, we have to pace the frontier, the frontier this, the frontier that. And I think an arc of the stuff that I'm really trying to ask experts about is what is in the frontier? What are we pacing? What's there? And then more importantly, kind of threading it with open source and open weight models, like, can they keep up with the frontier? And I think this matters very much and is something that you're an expert on. So maybe to sort of start on that arc, sort of how you would break down what's gotten us

• 2:42Speaker 01

  to the frontier for both closed labs and open source models? You know, how does it break down in your mind? Maybe first at a higher level of pre-training, mid-training, post-training? How post-training has evolved? Which is something you've, you know, done a great amount of education to the public on.

• 2:58Speaker 00

  When I think about model progress, it comes down to different forms of capital that you've put into them. And that is like compute, data and talent. And all of the labs have various amounts of this. And I think the closed frontier labs of OpenAI, Anthropic, Googles have clearly had the most of these. And what we see is like the different abilities of the labs to organize these forms of capital. And like making the org chart work for building models is really hard, which is why you have, companies have to do complete resets from the bottom up. And then it actually kind of works sometimes. Like company like Meta is like totally reset and they're like on a positive trajectory again.

• 3:33Speaker 01

  Seriously?

• 3:34Speaker 00

  It's like, I think I'm very interested in these human aspects of why competition drives the frontier and why the org chart is so important. Because like we see these things and this comes into play in the open-closed dynamic as well, where it's like the closed companies, the true frontier labs, which I say, if somebody wants to work at the frontier of AI, I mostly tell them to work at Anthropic and OpenAI, which is the easiest bet. I think Meta, Google, SpaceX AI, like I think you still might be there, but it's harder to see from the outside. And these companies have just taken the most ambitious bets the earliest in order to get there. And that's why OpenAI has done things like kickstart the reasoning model revolution. And I think that kind of breakthrough and that beautiful plot with the inference scaling laws is peak OpenAI. It's like they're really good at getting this plot

• 4:19Speaker 00

  that just changes how everyone thinks about everything.

• 4:21Speaker 01

  Yeah.

• 4:22Speaker 00

  And on the other side, it's like Anthropic has been like very focused on the types of things the models are going to be able to do. And then their reward was this Claude code explosion with their focus on coding for so long. And there's these subtly different cultures. And I think those two companies in particular have been the most forward-looking and the most willing to spend egregious amounts to make a step towards a direction that people think is impossible. The Chinese companies came into this because they want to have a company that matters. And if they were to build their models and try to sell an API from China, nobody would know about them. Nobody would know about Moonshot AI. Very few people would know about Jipu and there's a very long list after this. It was a very practical decision by them. They're following the discourse. They know there's gonna be a huge market.

• 5:06Speaker 00

  They know there's a huge demand in the US for paying for these things. And it's very practical. So therefore, like releasing open models is a way to shortcut this and get into the conversation. And I think there's a lot of complexity and like, why do they do this? Is the government involved? And not, it was like not the case. They're just trying to build a startup and.

• 5:23Speaker 01

  So practical.

• 5:24Speaker 00

  It's not to say that they will not get there.

• 5:26Speaker 01

  Right.

• 5:27Speaker 00

  The Chinese government is, does things that would blow people's minds in the US. And I think that that makes me think that the Chinese labs have a much more serious approach to security because like, things happen in China that are much more drastic responses to companies doing things in the US. And I don't need to go down this rabbit hole, but between the balance of open and closed, these companies in China are trying to catch up. And this is a very different dynamic than what I said about OpenAI and Anthropic. They're trying to create the new paradigms, that are trying to make true RSI happen so that their company wins with the best models forever if you get there first, as these boom ideas. And you just have to fill the market where there is a void. And then open weight models are trying to do this.

• 6:10Speaker 00

  And I think over time, there will be these tiers of open models where there's the best closed models, which you can demand the best margins for. And then there's the near frontier models, which is a step behind this. And those will be open because it's just like some sort of supply and demand there as you get access to a different market that might be less profitable. For example, like if you have private enterprise data or so on. And these tiers are kind of natural evolutions of companies that are trying to do things. And I think we were talking about the frontier. And it's just if you go back to capital, like they have less capital, they're gonna make less money. And like, that's okay. But I think this drive to catch up is very strong. And once you know, once Anthropic and OpenAI

• 6:54Speaker 00

  show you the type of model that you need to build, it's a lot easier to then be like, oh, we just need to do this thing.

• 6:59Speaker 01

  Yeah.

• 6:59Speaker 00

  They buy similar data sets. It trickles through the data economy where OpenAI and Anthropic spend way more to build out this function of a data industry. And then those data companies, a year later, might sell them to everybody. They might sell them to everybody. They need to maximize their return on their investment and they had some contractual period. And I think there's a lot of dynamics where fast following is going to be very doable. And we've seen the cycle proceed for a while, like longer than many people would expect. I think when ChatGPT was released, we talked about consolidation in the industry and predicting it. And I think we're just starting to see a much more active mergers and acquisitions market. But I thought this was gonna happen two years ago. And the open model companies have lasted

• 7:43Speaker 00

  much longer than I thought.

• 7:46Speaker 01

  So many interesting threads to pull on here. I mean, starting with actually your commentary on the organizational differences between the top two frontier labs. We can maybe table that or go on through that thread. But then, of course, also kind of, are you explaining sort of how the Open Labs in China have really just chosen that as much more of a distribution slash practical commercialization strategy? Rather than, hey, obviously out of altruism. Though it's great that the researchers, maybe for better or for worse, were motivated from sharing the beginning. But at the end of the day, it perpetuated because it was an excellent, excellent commercialization strategy. So the other thread maybe to pull there is sort of, I think most people regard distillation as sort of like the main lever for them to do this so well and kind of increasingly well because the lag between the open source models

• 8:30Speaker 01

  and the frontier models have only shortened with time in the last few years. Now it's like maybe only two to three months. And this has also pushed the frontier labs to maybe want to close off the APIs, close off their best models to do first-party services. I mean, one, probably to capture more value because it's always possible that way. But then two, also because it's harder to distill, not impossible, but much harder to distill. Maybe in those two threads, because they're both very interesting. So we'll try to touch on it. Let's start with is the distillation story really more of a caricature in China's labs? Since you've actually met with them and you've studied this a lot more, is it sort of like they're all-rounded? Like great at pre-training? Also have their own good post-training recipes that they may or may not have revealed? And where do you see the levers kind of accelerating?

• 9:14Speaker 00

  Yeah, so the Chinese labs are very technically excellent. They are more compute-starred, I think.

• 9:18Speaker 01

  Yes.

• 9:19Speaker 00

  Without a doubt, they have a comparable talent pool to draw on. I think there might be like a slight edge based on who was where when AI really got going. But to say that China is anything but equal on talent, you're gonna have a hard time because they just have so many people being trained at such a high level. And. What they don't have is compute. You talk to people there and you're like, oh, I heard an average OpenAI researcher has 5,800 equivalents that they can access at any time. And like their jaw just falls to the floor. They're just like, I can't comprehend. And this is like a top pre-training researcher at one of the labs that you would know. And then you have this interaction a few times. And the compute thing, I think, is the biggest difference.

• 9:58Speaker 00

  And distillation is a way to effectively spend more money to get what is compute equivalent. Like outputs from a language model are a function of compute. And if you know the outputs you want, it's the same as having compute to generate with the model. And I've seen the stages of distillation. And how I see it is that previous post-training was much more reliant on the supervised fine-tuning and other like RLHF era, which is where post-training started. And you could get most of your performance just by you do this next token prediction loss on the outputs of a very strong model. It was very established that you take the best model you can, you generate completions, and you train on it, and your model's gonna be pretty good so long as you have access to the right prompts. Prompts have always been a key to post-training.

• 10:43Speaker 00

  I remember at AI2, like with Hannah, who I co-led post-training there with, we would always say like prompts are the key unit of post-training. And so you need to get the prompts and you can do different things with them at different stages of training. And now this has morphed a bit into environments where it's RL-heavy. But it's just like hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today. And based on the public literature, it's very hard for me to be like, distillation is a major factor. Because, for example, in the fully open literature of people trying to do open post-training recipes, all the open model builders are stuck using like a really outdated Qwen model because you can't figure out how to get the teacher right.

• 11:27Speaker 00

  My hypothesis is that there's some like mid-training and something in the common pre-training data that's open that makes it hard to do this thing, which is get reasoning traces from the absolute best model and do SFT and the numbers go up. So like if I were to have access to Claude's reasoning trace when I was building OMO, I'm actually like Claude Opus, Claude Mythos, whatever, I'm not sure that it would have actually made our model a lot better. So this is like my evidence grounded in the literature, why I think distillation is hard. And then like, how do you take the outputs of a model and build an RL environment to do a big RL run? Because then you have the Zhaomies of the world doing this week-long open RL run where they're like, here's what we did. And then they've dropped the model. And it's like, where exactly does distillation go into this? They're clever.

• 12:12Speaker 00

  They know how to do marketing well. There's some questions of like, did they do this ahead of time? Because they did their RL run and then they dropped the model? And then their tech report said they had an extra post-training stage after RL? So there's some questions. But like, mapping the established distillation literature to that is very hard. But there are question marks that make me support the labs a bit, which is like these Claude proxies and these model proxies in China where there's a ton of tokens that are used and there's intermediaries and they have tons of accounts. And even cases where Kimi was serving Claude in their products to users. And I'm just like, don't really understand how this fits into distillation as a common technical tool in post-training.

• 12:55Speaker 00

  And like, yes, we should watch it to like, there's other weird stuff with Chinese companies using the outputs from OpenAI and Anthropic's models at a vast scale. So I think we have a lot more questions we need to answer. And I think most reasonable people, when they're like, what's the gap from distillation? Are like, there's very big error bars. But when I say that distillation helps them by a few months, it's based on the scientific literature that we just like, best information we have, we should do more open research and enable this so that we can study it. But like, I don't know how you can come to a different conclusion that it's like not a huge effect in current post-training based on this. And this is like, the Chinese are so good at building models as well. I think there's a bit of cope where people are like, only we are good at building models.

• 13:37Speaker 00

  And it's like, there are so many companies, there's like 20 to 30 companies that are building good models.

• 13:42Speaker 01

  You can like read, yeah, yeah.

• 13:44Speaker 00

  And I think you just, and like, it's hard for me to get over. And I think that there's this very prominent paper that's like stealing reasoning traces from closed models. And I know that the Chinese companies are way ahead on doing this since the AI safety community later published it. Like, I think every Chinese lab has a team that knows those sorts of things. And this goes back to what you're saying about like, they don't need to offer the API. And I think there's some characteristic of reasoning models that generate these thinking tokens, where it's easy to just knock them out of the API. So you see the reasoning token and then you can use that to train your model. And so long as they offer these APIs, there's gonna be no way to stop that type of distillation.

• 14:27Speaker 00

  But if they do make it so it's only in products and things where you have to sign enterprise agreements or whatever, it'll be way harder. And I support the labs doing that if they want to secure it. But it's like, they have to secure their products if they want to. Like, I don't, I don't think knowing your customer would be good for anybody. I don't think it'd be good for their business. But also, I don't know why you should get the government involved in a terms of service violation. It's like, not even, like, I don't even think terms of service are a signed contract. It's just like a, I don't think you can sue them in the same way. You just can cancel their access.

• 14:57Speaker 01

  Right.

• 14:57Speaker 00

  So that's why I'm not that sympathetic, but I am, like, I like to listen to the, like, the national security community is loud. But it's like, there is a directional truth to that with where AI is going and how influential it is, that you have to balance many, many things in your head.

• 15:14Speaker 01

  I wonder what your response is. So I heard, I think Jensen said in some, literally either Twitter interview, maybe, you know what I'm saying?

• 15:22Speaker 00

  Put it in, put it in the show.

• 15:25Speaker 02

  People distill my products every single day. They take it and they strip it down to its raw parts. They test everything. That's called competition.

• 15:33Erik Torenberg?

  Do you think that's fair game? You think distillation is not robbery?

• 15:37Speaker 02

  That's called competition.

• 15:38Speaker 00

  I'm very sympathetic. Because I think there's a lot of calls from the AI safety community that we want to have, like US-China dialogue, as we're getting to such capability levels. And if you're gonna ask the company to just like, start hitting their AI industry with slaps on the wrist and more downstream of distillations, you won't regulate distillation with policy from Congress. You will probably just do federal action to make their life annoying. And at the same time, when we're talking about how do we manage frontier risks, where you have closed models in training, going all over the web and accessing things that they shouldn't, and we have valid worries about like, what will happen with open weight models, where this like, hasn't been harms yet? But it's like, why, like, why aren't people using open weight models?

• 16:22Speaker 00

  Like, why do they go to closed models and get around the safeguards? And like, we want to be able to communicate with the Chinese tech companies and everything about this. And I just think like, you're gonna cause more harm than good if you do anything about distillation, because they're not gonna stop. It's all, it's like, they're gonna keep doing it.

• 16:38Speaker 01

  Right, right. And then, and the open source community benefits, and therefore the startup community benefits a lot. And go into, you know, a lot of, it's not just like, hey, China, USA. It's actually like, there's, you know, and we can actually talk about, like, I am sympathetic, obviously, or try to steel man the frontier lab position, which is like, you have just unprecedented amounts of capital that's required to build these frontier models, right? And it's, it's kind of alarming slash not very conducive to this flywheel continuing if, if the, through distillation or other kind of distillation-related methods, open source can catch up so quickly. And so, you know, and what I mean by this flywheel is just kind of a very simple picture

• 17:21Speaker 01

  of, you know, you train the frontier model by putting in a lot of money to get compute, and then they park this compute, of course. OpenAI and Anthropic are very guilty slash, you know, successful in this. Whichever way you wanna like, look and look at. And there is a huge compute, you know, kind of, sort of, lack of compute available because there's actually like, not just compute in the chips, but there's also power, et cetera. So like, all of this scarcity is kind of, you know, bottleneck until 2028. So this is all to say that it's a successful strategy to park that, which leads to actually a lot of revenue. Because once they get to the frontier, because they understand, you know, scaling is the one thing that kind of gives them, or the science of scaling is what gives them a handle

• 18:02Speaker 01

  over how, how well that can translate like into a predictable revenue stream. So they get the revenue and because of that revenue, and they're able to raise more money. It's just unprecedented, like beautiful flywheel. And this happens, we're both OpenAI and Anthropic. Of course, we need exponential amount of resources. And so we've, we've gotten to the point where we're like, okay, we need to IPO. We need to go to the public markets. And so anything that threatens this flywheel from functioning well is, is a huge threat to just the entire, like, being at the frontier. And so-

• 18:32Speaker 00

  The entire AI industry, realistically.

• 18:35Speaker 01

  And exactly that, actually. So like, you know, maybe not just kind of being sympathetic to their version of like, trying to, you know, eke out capitalism for themselves. It's also just like, hey, if, if they don't do well in the IPO, that's actually bad for everyone else. And so I, I want everyone, you know, I want them to do well. On the other hand, you know, it's like the Chinese open source models are a huge benefit to the startup ecosystem. I mean, we see this all the time, just kind of metrics. I remember from like Decagon and Harvey, who was trying to do a lot of fine tuning. There's like a huge amount of startups doing post-training as a service of various degrees and trying to privatize that. And it actually works because you can get to the Pareto frontier

• 19:17Speaker 01

  with these, you know, Kimi K3 or pick your favorite like GLM open source model. So I guess like weighing these two things, how do you, I guess, there's so many questions can fall from that. But I guess, one, how do you see that playing out in terms of like, who's gonna, who's gonna win in the next couple of years versus, you know, what, what does, and then also relate that to, you know, why you're doing, obviously, an open source?

• 19:44Speaker 00

  So I think open and closed models are on different time scales of exponentials. And the closed labs are so integrated where they have efficient inference, they can serve the model at scale when they're ready to, because they have all these resources, they can plan, and they get to these markets first that are very high value. We have seen this with coding, and this has been the story of the year where the revenue has just gone zero, 100 billion, and likely to grow. But the open model labs, partially, this is like an economic issue, being the fast follower, is that you don't have this flow of information, and you don't have this flow of revenue. So the open model, you have an open model builder, then you have the model inference companies, and you have the open model post-training

• 20:27Speaker 00

  companies. And like, there's information from those customers, which are like things you need the models to do. And it's like, you can't, like, the information doesn't just flow back to the model builder very easily. And they also don't have the economies of scale to serve as many users, to serve it as efficiently, to have as much margins on influence, on inference. And these are just like, it's gonna take a lot longer. But to the extent that things can be commoditized, it seems like companies really want to have private data. There's a lot of things that they just cannot send over an API. And we could talk about the like, zero data retention or not, and so on. But like, this just takes a long time to figure out if a company has internal expertise, or they figure out how

• 21:09Speaker 00

  to build an internal evaluation that actually matches to revenue, and then they figure out how to fine tune, or then they go find, they go find thinking machines, or prime intellect, and like, we could make money en masse if we figure out how to fine tune a model that solves this. And like, they have these customers, but there are just so many more companies involved that it takes so much longer. But at the end of this process, they have real ownership, which is model weights, no terms and conditions, that they can run on their hardware, and they figured out they can make an ROI on. So this is why I think the open model economy, it's just, it takes way longer to build. And I'm less concerned about this being kind of bubbly, where it's like, this could eat into the market of the really fast takeoff, as people figure out how to, what is the cheapest model that

• 21:52Speaker 00

  accomplishes a task, and how do you compete on margins. And I don't really know, like, I don't really know what's gonna happen. I think there's, it's too early to know.

• 22:01Speaker 01

  It's, it's so, yeah, it's so hard. I mean, it's definitely something that's always top of mind, thinking about, like, there's, there are things where scaling up general reasoning, just as a general model is so, it's going to beat out any specialized model. And we see, obviously, examples in math, and that's probably the most pure domain that you can see this. But also, you know, even in, say, robotics, you know, with, with Astra, and I, for maybe, however degrees of like, trucey the signal is, it just doesn't sound like they've actually tried to ascend in robotics, is my general.

• 22:37Speaker 00

  They did. I think this is a cycle of like, six to eight months ago, they started hiring people and buying a lot of data in robotics.

• 22:42Speaker 01

  They did.

• 22:43Speaker 00

  And then the model magically, six months later, is really good at it. But I think that the-

• 22:48Speaker 01

  But I actually, well, the other, other, you know, more contrary things I've heard is that it's actually coming out of computer use, because it's, you know, because computer use is like, some level of spatial reasoning, and then getting to have good blender, you know, demos is, is another extension of that. So, you know, we can believe some version of that. But either way, I guess it's like, maybe if it's like, really specialized, you see, it's, the capability gain is really from like, trying to get better in a certain area. But like, in math, I don't even know if that fell out of trying to specialize in math. I mean, I think the whole thesis there was just like, let's just scale up reasoning, and actually got better, in, in, in natural language, it got better than any specialized model in math. I mean, Anthropic wasn't even trying to be good at math. You

• 23:33Speaker 01

  know, OpenAI, you know, certainly, I think, you know, they hired a bunch of mathematicians, but they also weren't trying to like, use specialized, you know, kind of like lean data set to get better at it. Or at least, that's not, sorry. So anyways, this is, this is all kind of fodder for the view that like, it's kind of hard to compete with the frontier models, because eventually, they start ascending so much that, you know, you, you get good enough in respective categories. With that said, I think actually, there's also very competing, like a counterforce of data that you're seeing, which is like, the open source models that are already good enough, allow for fine tunes, you know, to your point of like, these slow percolation diffusion of these very specific benchmarks and, and data

• 24:16Speaker 01

  sets that are only available in these companies that you're being applied to. And it's just, it's just good enough. And it's actually, we're starting to see people adopt that. I mean, in the most maybe pure regime where you would expect frontier models are good, is, I think Cursor published something over the summer, where like, hey, we just, you know, we, we had this long horizon task of writing, rewriting SQLite, and turns out, using two frontier models, you know, one for planner, one for execution, is not as good, or is equivalently as good, but far more costly than just using Composer with maybe Opus, or, you know, for Astra, whatever, as planner. And that, I think that's actually great evidence that the open source fine tunings are very good, or good enough and

• 25:00Speaker 01

  has surpassed intelligence-hungry tasks. There's a long way of asking, what evidence do you find is most compelling, or, you know, pieces of evidence for various versions of this stable state of where the frontier wins versus open source?

• 25:17Speaker 00

  I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in, and he'll climb on. And I think, there's two, there's two things. We have the math and code thing, and we, we don't know how big the coding market will be. And then there's kind of everything else, which is still, they're targeting this knowledge work, this biology, healthcare, legal, and all of these things are not as verifiable. And I think it is much closer to the story of OpenAI bought a lot of robotics data, they're probably gonna tell us it's generalization, magic, poof, people love poof, generalization. But realistically, it's probably some of A, some of B, but a lot of it comes down to data. When I talk to people in frontier labs, there are some people that have amazing

• 26:02Speaker 00

  jobs, there's some people that contribute by doing a very narrow thing, which is you make a data set, and you figure out what the behavior should be for your model in that area, and you figure out how to make the data that helps get that data-

• 26:14Speaker 01

  Important work.

• 26:15Speaker 00

  -and get that behavior, and you figure out what data you need to buy. And that's gonna take, I think we're gonna have years of this-

• 26:20Speaker 01

  Yeah.

• 26:21Speaker 00

  -across different sectors. And that's very capital intensive to buy all of this data. But that's not just like, poof, magic, RSI happens. Where I do see in this math and code, and some like, distributed ML engineering, like, it's very verifiable, you know what your cost is, and the AI models are gonna drive that down at a really magical rate. And that's gonna just have a lot of surplus benefits, and we can do research faster, you could, I'm very interested in like, I think the efficiency of models is gonna drop multiplicatively over years. So we've seen OpenAI have drastic price cuts. And it's like, what does it mean for GPT Sol or Astra and these bigger models to reduce in price 2X every eight months

• 27:02Speaker 00

  for the next three years? You start getting, you start doing this math, and it's like, it's 1% of the cost in whatever, like, four years. And it's like, are they gonna be able to have higher margins due to Jevon's paradox there? Or are their revenues gonna decrease because the market fills up? There's just like, all these really weird dynamics. But fundamentally, like, AI progress is about data and about writing better software to serve your models. And those are very different worlds.

• 27:31Speaker 01

  Yeah. I mean, at least, definitely, I agree with you that, observing from the outside, there's a lot of data RL environment companies that are, you know, doing very, very well, selling both to the, the top two frontier labs, but an increasing kind of longer tail of labs as well. So to your point, I think a lot of the capability gain has really come from just scaling that up. And it's not an easy task. I mean, maybe there's, I'm actually very curious on your opinions between the, the data collection and the environments that are built by hand versus the ones that are, you know, done in sim. There seems to be like, different versions of like, what could be, you know, what, what kind of price they command, etc.

• 28:13Speaker 00

  They all have good things and bad things. And I think there's a lot of talk of like, like, we've talked about distillation, but on the other side is like, the Chinese company buys the RL environment that they use six months later, and like-

• 28:24Speaker 01

  Exactly. Oh, Anthropic, OpenAI bought this, now there's a premium.

• 28:27Speaker 00

  The good, the good data will do this. But the data that people are buying is pretty flawed. Like, I would say that the method is, you buy en masse, and then you figure out what works, and it's not 100% hit rate.

• 28:39Speaker 01

  Yeah.

• 28:39Speaker 00

  And I think, I personally think that this RL environment explosion, where there's just so much opportunity to make money, is having a lot of people like, like, cut corners, and this can induce things like reward hacking. So it's like, the models are so smart that if you put them in a world, and they're like, my job is to solve this task, but they didn't give me the, like, you have a test file, and like, the test file actually has an empty file, instead of the, instead of the maze that you have to navigate. And they're like, the model is like, well, I should still figure out how to get a high reward anyways. And I think a lot of this behavior is downstream of this. Supply chain. And I think it'll fix it, but it's interesting to see the demand from the labs to eliminate reward hacking and all these company researchers spending so much time on it.

• 29:23Speaker 00

  And then all the data companies like buy our data. And I think proportionally spend less time on model behavior and more time on world building and revenue. And somehow these two parties need to meet together to actually make higher quality data.

• 29:36Speaker 01

  Yeah, well, a lot of the labs are actually taking it in-house, partially because they don't want, you know, data leaking. Not that, you know, like literally and figuratively speaking, it's like, you know, signals leaking of what they're doing, but they're trying to send on. But despite that, I think there's a tremendous amount of growth in how much they're purchasing. Do you think that's probably one of the strongest levers? Can you kind of qualify how much that will kind of increase? Do you expect that to increase based on spending? Or are there other, you know, because for instance, if it can, it can be done well in SIM, I think maybe that is more of an argument to that being absorbed by labs and, you know, in particular. And then.

• 30:16Speaker 00

  All the companies are doing a mix of, like, all the companies are selling synthetic data where they're using language models to modify or build the world or generate permutations of it. And like, if a human is guiding this process, it's still synthetic data. So like, there's very few people that are gonna like, sit there and blender and like, we're gonna make a little world for our model to act in. Like, all the companies I think are moving far faster than that.

• 30:40Speaker 01

  Right.

• 30:40Speaker 00

  And they buy them because they, the models are getting better from them.

• 30:44Speaker 01

  Yeah.

• 30:44Speaker 00

  Like, I still have denial about it. I'm like, these numbers are crazy. The billions of dollars research budget on environments. I'm like, they're only doing this because it is helping the models and they see the signal.

• 30:53Speaker 01

  That's true.

• 30:54Speaker 00

  And I think, like, my general observation is that the data industry has a lot of low hanging fruit on improving quality and improving just like opsec. So like, if you look into the details, you can find weird things with like data companies that use contractors overseas. Those people are through an intermediary, they don't know they're selling it, they just upload it to GitHub. There's a lot of stuff out there that if you poke around the data industry, you find weird rough edges. And it's just very early. And that's like, it's, it's the one thing is like, the numbers are too big, the quality is bad. But on the other hand, the labs are improving very fast and know how to onboard it. So I think this will take a few years to transition of like, really pulling the winners into higher quality, higher ROI.

• 31:37Speaker 00

  It just takes less of resources when you have these systems for doing better. And then the labs keep working with them because they get even more training signal and have to do less internal filtering. And I, I think this is gonna just take years.

• 31:48Speaker 01

  How do you think about then using environments and like your purchasing decisions as part of the nonprofit that you're starting?

• 31:55Speaker 00

  I think most of our work will be making open source environments for tasks that are very high leverage. So I think of this work that I advise at UW or AI two state in Hamish Iverson on this T max work, which is you use a few, it's for terminal tasks, which are like super popular at terminal bench, whatever number we're on. And this is fully synthetic data where you prompt a frontier model to generate a type of problem and you use a few different prompts to generate diversity. And then you use model rollouts and other things to find bugs and lemon, eliminate the ones with bugs. And this process is not magic. It's like, you read every data point, your data will get better and you can do this. But there's just not that many institutions that are investing what it takes thousands

• 32:40Speaker 00

  or tens of thousands of dollars to make these environments well and then release it. Because the profit, because the profit motive is so high.

• 32:46Speaker 01

  It's so high.

• 32:46Speaker 00

  Yeah, like, I was joking with the author, like, yeah, you could probably sell this data for a million dollars and then release the paper. And like, nobody like, we care about doing the right thing. But it's just like.

• 32:54Speaker 01

  Oh, my God.

• 32:54Speaker 00

  These open resources are the foundation of so much study. And it's like, you release some RL environments with some RL rollouts, people can do research on say, like eval awareness, you, you have the generations from the model, people can look for different behaviors. If you've done RL on it, they can see how it goes over time. And this is just the, the units of infrastructure that PhD students spend a year looking at and produce increased knowledge. And the tech industry is the place where value is captured from CS research. All CS research, the benefit accrues to the tech industry in like 10 years. And I just think like, this is why more people supporting open, that if they're gonna work, if you're gonna work in the industry for 10 years, like, your peers are gonna be the ones that benefit from it.

• 33:38Speaker 01

  More fossil fuel, let's not deplete it all.

• 33:39Speaker 00

  But it's just like, the opportunity to go short term win is so high. The opportunity to make, what is it, a hundred million dollars, like, nine figure returns for your business within five months of founding a startup with your friends is just like, these are people who may have done more open source work, or these are people who left their PhD or something. And it's just like, pulling back, which is why I came up with this name Trillium for our lab, which is like, Trillium is a spring ephemeral flower, which it blooms before the canopy has grown its leaves. So it's like winters, no trees, leaves on the trees, these Trillium flowers bloom. And they're like essential for spreading nutrients through the cycle of growth that comes through spring and summer. And this is where I feel like the AI industry is doing so much right. It's like, so much is going right.

• 34:24Speaker 00

  But it's just like, we just need a little bit more life to like, spread this out and make things so new people can come in, like, there's so much attention and just like, do research, do education.

• 34:33Speaker 01

  Yeah.

• 34:33Speaker 00

  And then as this scales and becomes more impactful is like, you have these open things which become the standards of study for such important problems, which is like, how do you understand sandboxes? Like, how do you define what loss of control is? How can you see how this changes in a RL round that is a week on a thousand GPUs or 100,000 GPUs? And it's just like, make these places so that people have more trust. And I think this pays so much dividends to everybody that hands the profit more than I do. And it's just like, hard to get people to take that leap. And like, I've been rewarded so much for kind of stumbling into this area. And now I get so much reward, I get to do things like this, which is fun. And it's like, now is the best job because I tried to create space for more people to

• 35:17Speaker 00

  like, grow their independent thought and build this infrastructure.

• 35:21Speaker 01

  Yeah, and maybe one like kind of naive way that it kind of perpetuates is just because it's open source, people get to, you might get more of a distributed contribution from people who are just so, you know, it's like, they're motivated not for just trying to sell to labs themselves, but because, hey, it's out there, they can share their work. And then you actually have a very high caliber amount of contributors to the project. Like, I mean, that, that would probably be the naive hope of how that.

• 35:46Speaker 00

  Yeah. It's a new puzzle with AI agents where people just throw things at you. Like, I have some random research repositories for research papers and I'm like, people open the most crazy GitHub issues now.

• 35:56Speaker 01

  So what do you mean?

• 35:57Speaker 00

  Like, just like, like fake research results. They open an issue and they're like.

• 36:02Speaker 01

  Oh, my God.

• 36:03Speaker 00

  And they're like, I found this thing in your data set. And I'm like, well, if you looked at the paper, we said that, like, the second sentence. And it's just like, a lot of language model output. And that's a new thing for the open community to go through.

• 36:14Speaker 01

  Yeah.

• 36:15Speaker 00

  But this is solvable problem. Like, like, there are people out there that want to contribute to this stuff because they.

• 36:22Speaker 01

  Right, not just clout chasing in another, you know, dimension, but like, actually, hey, it's out there, you know? I mean, I would just hope because it is published and open or, you know, and available, you can actually just get like, PhD students or other researchers are like, okay, I do this on my day job, but maybe I can, you know, still be able to contribute. Yeah, hopefully from the frontier labs.

• 36:41Speaker 00

  Like, I see people like this and I was like.

• 36:43Speaker 01

  That's great.

• 36:43Speaker 00

  Somebody from like, Eastern Europe was helping me improve the code for my book, which is right here. And he's just like, doing it for the right thing. And like, we would talk. And I was like, there are people that do it. And I think a lot of people just take, like, they don't have the time or the stability to take the long time it takes. Because when you're trying to get into a new field, you know nothing. You have to spend like weeks reading and knowing nothing. And then you send an email and nobody replies. And you're just like, it takes you weeks to know the context, like, ask a good question.

• 37:15Speaker 01

  Yeah.

• 37:15Speaker 00

  And now with language models, most people just push a button and they're like, oh, interesting, I read this, I'll send it off. And it's just like, this.

• 37:24Speaker 01

  Oh, yeah.

• 37:25Speaker 00

  This kind of like, sitting in it is the thing that people don't do as much. And that is the key to like, becoming into AI is that like, you just have to focus on it, you have to write some code by hand, and you have to do this for months. And then you have to wait six to 18 months for you to get noticed. It just takes a long time. And like, AI, all the AI agents, I think, just like frazzle that and makes it harder for people.

• 37:47Speaker 01

  That's what the mathematicians complain about too. That's all. It's like, um.

• 37:52Speaker 00

  So it's a hard time to be a PhD student, but.

• 37:54Speaker 01

  Yeah.

• 37:54Speaker 00

  People are doing great work in academia still, and, but it's hard.

• 37:58Speaker 01

  Yeah, but I agree with you. I think it's like solvable as the agents kind of perpetuate some version of the problem, it can also help solve and identify and quality filter. Okay, so environments, you're gonna release open source versions of that help precipitate maybe more people to contribute and reveal that. I think that's like, really key, I mean, especially given how little incentive there is to release this instead of just get, you know, fast revenue from, from the labs. Another area I think I read on your announcement post is RSI, which is, you know, quite, I think, I mean, we could first talk about what is actually recursive about it or how scary it is or whether we're gonna have takeoff. I think you have a very grounded opinion on that. But, you know, another thing I want to kind of talk about is, I think, RSI may be related

• 38:41Speaker 01

  to how much it's kind of hyped is, is a reason why we are afraid the frontier labs will pull away. In fact, it's also a reason why I've heard not just once, but several times that people are, one, convinced that their own jobs are going to be automated as a researcher.

• 38:57Speaker 00

  Yeah, yeah. Like, I've gone through the last few years, some of my best friends, I'm like, you are some of the best people in the world at this. And then they go to the work and they're like, Nathan, I'm struggling. Like, I'm not, like, what am I gonna do?

• 39:08Speaker 01

  Because your point, like, a lot of what they do is verifiable, right? For, but because of that, you know, there's actually quite a lot of other non-verifiable or harder out of distribution tasks, such as, uh, was it alignment and things like that? And so actually, because, I mean, one argument I heard was because there's such a. Difference between how well RSI is doing for model capabilities versus alignment, which is much more out of distribution, maybe creative work, that gives people an ease on how, you know, maybe there is actual dangers, et cetera, et cetera. So anyways, because of this, I'm very interested in hearing from you like how, because I think it's really important that we do have an open source kind of front in showing how RSI is being done, how much maybe then we can, you know, understand the capabilities of like if in within a lab,

• 39:53Speaker 01

  they have a better model, what are the chances they actually pull away? Just better understanding and calibrating. So yeah, I would love to hear more about that.

• 40:01Speaker 00

  Yeah. So I think this, we won't be the only people doing this, but essentially like RSI has so much mystique where it's like, we're using models for everything. And like, what we can do as a open lab and a nonprofit, and I think other companies like a Luther or Hugging Face or Open Athena, which is Marin and Stanford, they will get into this. So this is a good marketing line for them is, and me, that's why I'm doing this. And it's like, we can share what tools we're using. We're going to build an open source stack for SFT for mid-training. We're going to use Skylar, Miles, all these things that you know. And we have our task, we have our configs, we know what type of compute we're running. We can work with Claude and GPT and all these models. And we'll just be like, we can store the trajectories for what we did. We can tell you what our tasks, whatever use linear or whatever these popular things are.

• 40:46Speaker 00

  We'll say what our tasks were and why we chose them and whether models work on them or not. And you just do this for a year of a lab and you're like, here's what we did. And then just check in every six months and you're like, has it changed? And, and it's just like, to be able to have the full stack and not need to go through these classifiers and like, try to like, say what the task is or vaguely and so on. It's just like, people that can share what automation is happening should do this. And we have a different distribution of tasks, but I think that the research is similar. Like, I think the AI research that I've done at AI too, and we'll do at Trillium versus what people do in some sects of academia and the labs, it's not that different.

• 41:31Speaker 00

  You have more resources, you scale it up and it's just like so good for the world when everybody's talking about this to just, to show people like, this is what Nathan asked his model to do, and this is what it got stuck on. And I think that will help a lot of like literal policymakers.

• 41:47Speaker 01

  Exactly.

• 41:48Speaker 00

  Be like, oh, okay.

• 41:50Speaker 01

  It's like, oh, that's why it's.

• 41:52Speaker 00

  Like, that's what it got stuck on. Like, that's the bug that it put in its code.

• 41:55Speaker 01

  Yeah.

• 41:55Speaker 00

  Like the bugs and the catch-ups will reduce over time. But I just think, I think the jaggedness in some of these fundamental things of the models, like some more things will take off. Like I said, efficiency earlier, like I do think the models would be really good at lowering serving costs. These companies have billions of dollars on the line and the models are good at it. They're going to make a lot of money on being serving more efficient and tweaking architecture in order to go into that. But it's not necessarily like, how do you solve knowledge work? Like Claude's not going to, Claude's not going to like, you're not going to be like Claude, make, tell yourself to make better PowerPoints. Like, what, what is.

• 42:32Speaker 01

  Environments or some other, you know.

• 42:34Speaker 00

  Like, it's going to take a while.

• 42:35Speaker 01

  Yeah, yeah. No, I love the, the sort of specified goal of like, let's just demystify this because I think a lot of why the rhetoric is so heated around RSI is like, it sounds scary.

• 42:45Speaker 00

  AI nonprofits can do a lot of things like this. Like, it's hard for academia to be like, as a professor, 15% of my time and 40% of my grad students can be documenting what we work on. But there's like such a great like policy and community service of like, here are our tasks. People can use this data to make evaluations or whatever. And it's just like, just like build things that have rare information so that when people put more high focus work into it, it just has more value for all sorts of things that you can predict in the future.

• 43:17Speaker 01

  It's honestly just also very timely because the kind of, you know, it's not from one person, but several people that I know from the labs are just like, hey, I'm going to get automated. And to your point, to your friends, like I will definitely as a person in RSI will get automated in a year. And so that probably means there's going to be less people who actually are exposed to what's being done on that particular, you know, part of the org chart. And so.

• 43:39Speaker 00

  There's a related academic, there's an open RSI thing, which is a bunch of academic institutions. I'm not affiliated with it. They just launched like last week. So I was like, they sent me a few emails. I'm like, open RSI is a thing. They're trying to do this as well. Like, it doesn't need to just be one, one group doing it. It's good to have different approaches doing this. Because you get, you need the different ways of looking at it and different results.

• 44:01Speaker 01

  Yeah, whole point of it being open. The other thing I also heard was from more kind of like, you know, frontier lab goss. It's just like, the actual methods that are being used in the labs are not that, it's not that they're unknown. It's just sort of like, what are the proportions that need to be used that are successful? And this is partially because they have the capital to scale it, to experiment. So I think that's one heartening to know because like, okay, it's not like there's a big secret of what's going on. So people actually know the techniques. But on the other hand, you know, capital is the one true barrier still, because it's hard to actually know which of, which of the techniques, recipes work.

• 44:41Speaker 00

  That's why I'm not pre-training. I'm happy to be free. I can use any base model, but you go, I'm now get to benefit from competition.

• 44:48Speaker 01

  Yeah. I, do you think that there's like a chance for any pre-training, open source effort? I know the capital requirements are just like orders of magnitude larger. So that is what's difficult. But maybe what are the values of that? And then maybe we can kind of discuss potential commercial strategies.

• 45:04Speaker 00

  Pre-training science is pretty, like the, the tools are pretty accessible and you have what is a pre-training ladder, which is like the decision-making tool you have to test if your idea scale. And the pre-training data isn't growing at like, like the environment scene. Like it's essentially, it's a first approximation, you train on the internet. And like, there's a lot of, there's a little, there's little details and secret things, but I feel like that's not changing that much. The state of their architecture is and the GPUs are, and that's like, that changes. And, but it's not astronomically expensive to make a pre-trained model that is solid. So I think like, you don't have to spend a billion dollars to get a very good base model.

• 45:48Speaker 00

  The more you spend, the better you will get. But it's like, people keep making progress and I'm constantly surprised by a new lab just goes from scratch because they can. And some of this is on open tools. So like, you can go off Megatron, like a lot of new startups, they're like, oh, what, what should we train? We train, they train off the open source tool and they fork it and then they never release it again. This is what all the, all the like even prominent open model labs have done. They take the open source code, they make it their own thing and they just, a lot of them don't push upstream. So I've always been surprised by this. So I'm like not betting against new entrants because like, it's not that you need $10 billion to like have a relevant model.

• 46:31Speaker 00

  The hard thing is like, get a model that's good enough for people to use. And to get a story where if you release it, you can do your next fundraise. But it's like, you have to get past this lock of like, we need our first model to land. If like your whole company is based on models are the best and you're not in the ballpark and you're not releasing, like you're on the clock. And it's like, I think of like thinking machines getting inkling out where it's like, they were very transparent. They're like, this wasn't the best model. We're proud of it, has some cool things.

• 47:01Speaker 01

  Yeah.

• 47:01Speaker 00

  And like, now that they've released their second model and they're getting going and they have Tinker, this, this, like, that's very good for them. They're like off the ground. And it's just like, that, I don't think that's what they were doing. Like they were building models, but open models wasn't their first plan. And then they got there and there's a lot, there's rumors that like Meta said they'll release a new open weight model again, like that'll come. There's a lot of companies out there that couldn't afford models.

• 47:25Speaker 01

  Yeah, yeah, no, I'm definitely glad that thinking machines is, and it makes so much sense for their business model, because they do post-training.

• 47:34Speaker 01

  And, you know, with Meta, I think you mentioned this as well, and in some previous writings, it's just for them, it's just such a good strategy. I think it was also just kind of obvious when they, when they do it, it's like, it's complimentary to my revenue streams or why not?

• 47:45Speaker 00

  Mark wrote the best blog post on this, like when Lama 3 came out, he wrote like the best, like why open this password? The guy got, guy got a little distracted. Seems like he's coming back to the light.

• 47:56Speaker 01

  So I hope they, they keep on being motivated to do that. Getting to the frontier with pre-training, you know, I think there's a lot of help, given that there's such good open source models to maybe, you know, somehow still kind of bootstrap from. I, I keep on getting kind of conflicting signals as to like whether pre-training is actually as saturated or well-understood. I mean, maybe, maybe to your point, like there's a lot of like well-known techniques at work, but what's hard is just actually having access to the 20,000, 30,000 chips.

• 48:25Speaker 00

  People are turning the crank on pre-training and pre-training became adjusted when super long context generation happened. So there's just way more return if you're, if you can, for example, like get a sparse attention architecture to work as well. And then your 100,000 token generation is 40% faster. So I think there's a lot of co-evolution of the whole stack where they have their pre-training stack, they realize that inference scaling is so much higher, but even the fundamentals of pre-training keep improving where you just have a marginally better architecture, the returns on investment. And then it kind of blurs into post-training where this mid-training where you start the reasoning behaviors, you take them from your older models, you do RL on them to seed behaviors. Like this cycle is much more complicated than pre-training.

• 49:09Speaker 00

  Like, I guess, if you don't have the right people, it's a lot harder to get that off the ground than I think pre-training was. Like, I think there's like, I think like to do a controlled, like, let's just start pre-training our models seems a lot more approachable to me than I don't really know much about language models. I'm going to figure out this whole post-training thing where it's just like the Wild West.

• 49:28Speaker 01

  Yeah.

• 49:28Speaker 00

  And that's why I was like. When I left AI2, I felt like our post-training was honestly falling behind just because our cycle time was not high enough and we're going into these agentic things. And these are still some of the like, leading open source post-training recipes. And that's just like, that's what I want to fix is like,

• 49:48Speaker 01

  Yeah, yeah.

• 49:48Speaker 00

  It doesn't take as much capital to make very big compounding gains on the state of the art and the open source because the bar is very low. It's like, it's not like, it's not close.

• 49:58Speaker 01

  Yeah, yeah. No, that that makes sense. It's just like a kind of efficiency where you can put capital and make a huge difference. Yeah, yeah. Kind of related to the whole, you know, post-training side, maybe, you know, pulling away from closed and open is the fact that the agents, you know, the type of harnesses that are being used in the closed labs are very different and slash, you know, actually trained and, you know, post-trained on to, you know, to be specifically good for, their models are specifically good for those harnesses. And so that increases the mystique even more, makes it very hard. How do you, I guess, one, do you agree? And then two, like, how are you sort of hoping specifically to kind of address that? And things I mean are really examples like, hey, spin up like

• 50:41Speaker 01

  thousands of agents, tens of thousands of agents. It's just like a very different workflow than what people are obviously exposed to.

• 50:48Speaker 00

  I think it'll be okay for open because I thought years ago that, say, the closed labs would be like Apple. There'd be the integrated company where they have the model designed for the harness that's involved that gives them like, kind of a kind of scale efficiency. But it turned out that so many people have different harnesses for different things that they like to do, that the models, companies have been pressurized to make it so that their model can work in all sorts of different harnesses. And I thought it would be a weakness of open models that they have to be designed to train on all sorts of types of harnesses. But it ended up being pretty equilibrated. Or like, it's at equilibrium where they all have to do it to some capacity. And then what we were seeing with this like agent swarms is that OpenAI and Anthropic are going to be earlier to it. They're going to know how to design their models. They're going to know how to design, what do we call,

• 51:32Speaker 00

  what do we call, want to call a harness for a swarm? I don't know. I don't know. But there's going to be a word for it. It's not a harness. It's like a coordination system. But then eventually, other people are going to want to use them. And then I think it's going to be like these cycles where the frontier labs, like, they push this out and they have this, to date, they've had huge returns on being first. But like, what is the returns on Navier-Stokes and all these things? Like, we don't really know. Like, we don't know. They push the frontier in the assumption that there will be high returns and that this marginal intent, marginal intelligence has like disproportionate returns. And we don't know if that's always true. But I kind of expect these cycles of like, once you know the shape of how do you control 10,000 agents at one harness, then the open model companies

• 52:16Speaker 00

  will be like, oh, we can do this too. But it's not like Timmy is going to do that first. For one, they don't have the compute.

• 52:22Speaker 01

  Yeah.

• 52:22Speaker 00

  It's an astronomical amount of compute. But these things go through cycles and Nvidia's chips will get way faster. There's more compute built out.

• 52:32Speaker 01

  So you're very convinced that like, there's not too much hidden there that you can probably guess at what they're doing. And then when it becomes relevant, let the open source models actually try to also post-train on similar meta harnesses, whatever.

• 52:45Speaker 00

  Yeah. And I also think like, the, the security at these labs in terms of information is not as high as I think. Like Silicon Valley is famous for like, ideas traveling, like, what is it, like, one beer or something? And it's like, everybody chats after work.

• 52:59Speaker 01

  Right. Which is why RSI is so impressive. Nobody knows what's going on.

• 53:03Speaker 00

  That's the only pass. But it's like, they're, we're all scientists. If they made a breakthrough, they're going to want to tell their friends about it. They might hold it, they might hold it for a few months. But like, I don't think they're, the amount of lock, like, information security you would need is just so high that knowledge spreads. It's hard to keep the secret. Like, what you, what they can win on is putting more inference into their infrastructure, which lets them serve at a lower price and make higher margins, and then continue this over a longer time. So I'm very confident in the frontier labs being able to serve the cheapest intelligence per, per watt, per score and a val. They are

• 53:45Speaker 00

  going to be able to go to the places with the highest demand and have the, like, the cheapest models. And that'll be revenue creative, or like, that'll be a good business for them. The question is, like, they can't do everything. And that's where the rest of the economy fills in is like, is it just coding that is profitable? Then,

• 54:04Speaker 01

  Definitely not.

• 54:04Speaker 00

  Good case for frontier models because they will have the most, like,

• 54:07Speaker 01

  Right.

• 54:08Speaker 00

  They have integration.

• 54:08Speaker 01

  And they'll do very well, but it's not just coding.

• 54:10Speaker 00

  The whole economy of knowledge work is the market for things. And then it might not be good for them because can they do everything at once?

• 54:17Speaker 01

  Or they don't need to. I think there's going to be a lot of winners.

• 54:20Speaker 00

  Yeah.

• 54:20Speaker 01

  Yeah. I mean, that's what I have. I mean, that's why we invest in the frontier models and startups, you know? Not because we want to hedge, but because I actually do think, you know, to kind of corroborate your view, that there's enough harder tasks at the fringe that requires very specialized data that's kind of out of the splash. So for getting to AGI, or whatever you want to call it these days, that makes it possible to have many, many winners.

• 54:44Speaker 00

  Yeah.

• 54:45Speaker 01

  I do like your RSI point, because we talked a little bit about it, which is just kind of say practically, hey, it's exponential cost and resource consumption. And therefore, there is a very obvious kind of limit to how much that the frontier labs can pull away. If that's not too oversimplification of your view. Anything else you kind of want to add there? Because I do feel like the discourse now is just so, so much fear on like, how much the frontier is just hidden and also breaking away.

• 55:17Speaker 00

  I think there's so much spend on inference and that in the short term, we are learning how to corral inference into performance gains.

• 55:27Speaker 01

  Yeah.

• 55:27Speaker 00

  And this is like, a thing that we just had. And like, this is what we're seeing now is that like, people are spending so much. And like, all the employees at the frontier labs are spending exponential amounts relative to the improvements. But like, we're figuring out how to do this and there will be a lot of gains. But I don't think that it's necessarily going to be this like, fast takeoff when the models build themselves, which is human, human bottlenecks. All the scaling laws are exponential increase in compute for linear increase in performance. And like, that's just like, it's hard for me to bet against this. I do say that like, we have Navier-Stokes, and I'm of the opinion that math is exceptionally

• 56:12Speaker 00

  well-studied. Maybe like, the scientific field that is best suited to this shape of intelligence. And like, intelligence can take many different forms. And that's not to say that like, if they do, like, if an AI model does say, discover something that's very different than the transformer, that gives very different properties and unlocks a new type of intelligence. Like, it is worth updating on this and worth being like, okay, like, that is different. But until that happens, I've been pretty happy with my evidence of like, the companies are going to do well. They're spending exponentially more to keep this remarkable pace of progress. That some of the Claude system cards say this. They're like, it would take, say, like, 10x more progress to make this a fast takeoff. And it's like, for now, they're helping us

• 56:56Speaker 00

  sustain the pace of progress. And I think it's a good equilibrium. It's like, it's a good equilibrium. But I do think that like, the labs have people saying this. The, the science is like, it's hard to overcome that. And it's okay. But it's like, we don't know exactly what it'll go from here.

• 57:12Speaker 01

  Yeah, yeah. No, I, I think that's comforting because we know what the inputs and lovers are here. It's just, there is a very successful autoregressive regime that we've been optimizing. And that one just, you know, asks for a certain amount of like exponential resources, which are clearly capped. And therefore, open source, hopefully, you know, can keep it strong as well. It's just, it wouldn't lag behind.

• 57:33Speaker 00

  There is dramatic amount of work to do in preparing for infrastructure. Like, there's the AI capability plot, which doesn't necessarily worry me. But the AI capability plot in the pretty much non-existing preparation across public cybersecurity and all of these things, like, there needs to be more preparation in the assumption that this is coming than there is. And then that's kind of the thing that worries me more than say, like, RSI is coming and unlocking new risks. Like, there are risks that are here and that are going to continue to diffuse. And many times I've said, the closed labs get there and then the open labs get there. It's like, we're in this process for a lot of things that are going to cause some problems. It's just like, figuring out how to get society to prepare is very hard.

• 58:16Speaker 01

  Yeah.

• 58:17Speaker 00

  And I come of the opinion that we need more eyes, we need more science, so that society feels informed.

• 58:22Speaker 01

  Yeah.

• 58:23Speaker 00

  But there are other valid opinions. And like, I do think that not fearing fast takeoff does not mean that you don't worry about risks. I know, like, you should.

• 58:33Speaker 01

  No, I think that's completely, yeah, maybe I'll send in some people your way that are defected from the labs who are worried precisely about this. Well, thank you so much, Nathan. This was a lot of fun. And yeah, I'm really excited to see what Trillium is up to now that it's finally launched.

• 58:48Speaker 00

  Yeah. Thanks so much for having me.

• 58:52Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode.

• 59:18Erik Torenberg?

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.