Ep 94: Applied Compute CEO on the Limits of RL, the New AI Hyperscaler & Why Post-Training Wins Inference

Yash Patil, a former OpenAI researcher who worked on Codex and now runs Applied Compute, argues that post-training is the most overlooked part of the AI stack and the key to owning your intelligence. He explains when companies should optimize their harness and context first, and when it's worth updating the weights: the largest inference workloads, plus tasks that change over time or depend on judgment. That's the heart of why he believes post-training wins inference. The biggest workloads are both the most valuable to train and the most valuable to serve, and how a model is trained shapes how it should be served. Getting production serving running is the easier half; echoing Dylan Patel, he says GPUs plus vLLM plus OpenRouter "kind of" gets you there. Yash is candid about the limits of RL. It's a hill-climbing machine whose hardest part is defining the hill, which makes a company's private evals as worth protecting as its employees. It generalizes far less than pre-training. And continual learning remains blocked by the unsolved problem of extremely data-efficient training from sparse rewards. He lays out his bet on a new AI hyperscaler, built on GPUs the way AWS, GCP, and Azure were built on CPUs. He pictures it as an inverted pyramid with training at the bottom, where few can compete, and inference, routing, and harness stacked above. The conversation also covers whether every firm needs its own model, why Jevons paradox has made cost matter more than he expected, how models reward hack "like water," and why the US needs American open-weight models even though "blanket bans are not the answer."

• 0:00Speaker 00

  How do you kind of characterize where we are today?

• 0:01Speaker 01

  What we have with RL is we essentially have a hill climbing machine. The hardest part is actually defining the hill to climb, which is why evals, I think, is like a really important thing for companies to, A, focus on building and B, actually like safeguard pretty carefully. Your employees are not fungible. You would not like be comfortable with them going to another company and doing work there.

• 0:23Speaker 00

  Same thing with these models.

• 0:29Speaker 00

  Owning your own intelligence and training your own models is incredibly in the AI zeitgeist these days. There's probably no better person to speak with about all these topics than Yash from Applied Compute. Applied Compute is working with a bunch of the most cutting edge AI applications on really interesting post-training examples. Yash and I were able to talk about his perspective on a bunch of the top questions around open models today, including when it actually makes sense to post-train these models, how to think about the future of when this stuff makes sense, especially as the labs get more and more data from different domains over time. We also talked about like when it makes sense to harness optimize versus actually go and post-train, as well as just get Yash's thoughts on a lot of the really interesting things happening in the ecosystem today. All in all, this is a fascinating conversation with a brilliant mind in the space. I hope folks enjoy it as much as I did. Without further ado, here's Yash.

• 1:16Speaker 00

  Well, thanks so much for coming on the podcast.

• 1:18Speaker 01

  Thank you for having me. Yeah.

• 1:20Speaker 00

  Been looking forward to this one for a while.

• 1:22Speaker 01

  Yeah, yeah, no, it's been a long time coming. It's good we can finally do it in person as well.

• 1:27Speaker 00

  Yes, always better in person in our little closet of a studio here.

• 1:30Speaker 01

  Yeah, this is awesome. I mean, like, it's very cozy, built exactly for the setting, so.

• 1:36Speaker 00

  Yeah, definitely not a repurposed library, purely purpose-built for podcasts.

• 1:41Speaker 01

  So thanks for having me, yeah.

• 1:42Speaker 00

  Of course. Well, I feel like there's a bunch of places we could start. But where I thought would be most interesting for our listeners is just owning your own intelligence feels like it's in the zeitgeist. I don't know when the tipping point happened, you know, three, four months ago, but like it is certainly, you know, in the conversation. And I feel like there's been a bunch of different versions articulated of the risks of depending on frontier labs. There's been people saying, you know, you'll never know when they'll pull their APIs away, when they decide something's a safety risk and they don't give you access. There's been, you know, they'll build products and compete with you. They'll be, if you use their models, they'll train on your data. There's all sorts of different arguments. And I'm curious, when you think about like this, you know, the case for this, which of these arguments like resonate for you? Which of them do you think maybe are a little overblown?

• 2:24Speaker 01

  Yeah, yeah. So, you know, I, as you know, my background is I actually worked at OpenAI.

• 2:30Speaker 00

  Yeah, you've seen it from the inside.

• 2:32Speaker 01

  And I think like the risks, or not even risks, just the things that I think about when it comes to owning your intelligence, aren't more the ones that are like presumed that the labs are malicious or something like that where they're stealing your data or whatnot. I think like, you know, these companies are made up of great people. There are contractual obligations. There's actual, you know, there's a lot of engineering that goes in protecting data and whatnot. But I do think that the idea of owning your intelligence really is about flexibility and control. And so you said like, oh, I don't, you know, I don't know when this own your intelligence narrative popped up. I think it really popped up when the open models started getting good and you could actually go and do things with them.

• 3:16Speaker 01

  But the control you get, it can actually be a lot deeper than that, right? Like where these models run, how you deploy them, what you optimize them, whether it's cost or latency or specific domains. The more control you get, that's what we're talking about when we say own your intelligence. So if you want to own your intelligence and you want to be able to train these models to do tasks on your data, you actually need to have access to the weights. You need to have the infrastructure to post-train them, to inference them, to route between them. You know, I do think there are risks about being super dependent on a single model vendor. Luckily, I think like the labs are doing their best to sort of like serve as many people as possible. But there are times when models can get pulled from different products.

• 3:58Speaker 01

  That's happened actually more than once, particularly in the coding space. And so, you know, I think as these labs start to move more into specific verticals, like, you know, it's not clear whether they won't, they'll continue serving companies that are they're directly competing against. And so the safest sort of like most durable thing that you can go and do and the advantage you can build up, I think, is investing in the full stack to be able to make models that are powering your products, that are intelligent, that are doing the things that you want and not having like extreme vendor dependence on any one set of models or companies.

• 4:35Speaker 00

  Yeah. And it feels like this narrative has really won out. I mean, I think maybe one of the most interesting manifestations of this is even yesterday, OpenAI announcing like this partnership with Base10, right? Where you can kind of use OpenAI credits, almost burned them down on open models through Base10.

• 4:49Speaker 01

  So I think first of all, we're big admirers of the Base10 team. And I think the Tuin and Danny and all of those folks, they're really an incredible company. But yeah, I think this was a clear signal that like, hey, this multi-model future. Customers want choice. They want flexibility. I think on the OpenAI side, it's a clear sort of motion towards, hey, we want to be more of a platform. We want people to be able to use, like draw down from their commits on different products. We want to be the gateway to enterprises using AI. And I think it was a big win for open source models. You know, a lot of people want to use this stuff.

• 5:27Speaker 01

  The fact that there is like a sort of gateway now that you can go and sort of use a variety of models, proprietary or open source, I think it's a big step towards this multi-model future. I don't know if you noticed this, but the narrative a couple of months ago was definitely like, hey, like, we are going to talk about and sort of promote and analyze the strongest, biggest, baddest model out there. And now it's like much more talking about a Pareto curve, right? There is like, there's some cost performance trade-off. You don't need a mega model for every single task and you actually get more choice and flexibility. I think open models take that to an even more extreme where it's like, hey, you can,

• 6:13Speaker 01

  you can slide along the Pareto curve by picking different open models of different sizes from different families. But also, you know, what our core thesis is, is you can move that Pareto curve out. You can train models to be better at particular things and you can actually get better models for cheaper cost. And so all of these things, I think, are like, you know, sort of the general trend of people wanting more flexibility. They've gotten more comfortable with deploying AI in production use cases. They've done a lot of exploration and now it's like, hey, this AI thing is really working. Let's go and optimize it.

• 6:47Speaker 00

  Do you think the like OpenAI Base10 thing will work? I mean, it's an interesting, like, are they in a position to be this like neutral partner to enterprises of like, you know, we'll give you whatever model is the best?

• 6:56Speaker 01

  It's something that we're also excited about participating in. The tricky thing is, right, like, it makes a lot of sense for an Amazon, for example, to be a marketplace because you have Bedrock and you can run anthropic models, you can run OpenAI models, you can run open models. Now, like, is OpenAI going to take it to the extreme where you can run anthropic models and AI models? I'm not sure.

• 7:19Speaker 00

  You mentioned kind of this pushing out of the Pareto frontier that post-training enables. And it seems, you know, like there's been a ton of really interesting examples of this. It feels like, most notably, a lot of application builders have realized, wow, when I do similar workloads at massive scale, like that inference bill really runs up. Or, you know, hey, this experience for my customers is so much better when it's faster.

• 7:41Speaker 01

  Totally.

• 7:41Speaker 00

  And so I feel like a lot of the, you know, initial post-training use cases I heard about were like, you know, cost and speed feel like tremendously valuable things.

• 7:48Speaker 01

  Totally, yeah.

• 7:49Speaker 00

  Capabilities, I'm curious if you could talk a little bit about like, you know, they're certainly pushing out a Pareto frontier of you could get for the same size model, probably better capabilities. How do you think about like the current state of being able to post-train to beat frontier models? Like the best mega models at like, you know, capabilities?

• 8:05Speaker 01

  So I think the interesting thing that we're seeing is, you know, there's obviously frontier models that are ahead of the open source models on base capabilities. But the fact that you're able to post-train the open weight models means if you have differentiated data, something proprietary to you, that's out of distribution of the models, that's when you can actually push these models to be better on capabilities as well. Our general take has been like, the more out of distribution your data is, the more likely that post-training is actually going to give you that capabilities lift. But also, again, it's all about that Pareto curve, right?

• 8:44Speaker 01

  So if I'm able to take a cheaper model and train it to be parity with the frontier, but at much better cost, you know. Sometimes you're able to serve more users for that. Your data has to actually be quite differentiated. But most people are actually post-training to basically optimize price performance.

• 9:02Speaker 00

  On the capability side, you know, I think it's an interesting question of like what data remains out of distribution over what period of time, right? Like, obviously, you know, I feel like the labs are laser focused on coding, you know? And so like, you know, maybe they're obviously doing work in other domains, but maybe not as as focused as they might in a few years from now. I'm curious how you think about like, you know, I don't know, five years from now. You know, how many businesses that Applied Compute works with feel like, oh, are data still out of distribution from like what the models do?

• 9:31Speaker 01

  Yeah, so I think like, my honest answer here is that the way we're going to be training models is going to change, right? So we've done, you know, we're in this paradigm of like offline RL. You go and you procure high-quality data sets from data vendor companies. You go and hill-climb those specifically, and you get a model that's very spiky in that domain. RL is reinforcement learning. It's learning from experience. I think a lot of the online RL methodologies are actually using production inference to go and improve your models. That is actually going to be like how most enterprises codify their judgment expertise by basically, the more they use the model, the better that it gets.

• 10:15Speaker 01

  But generally, right, like the proprietary data that will be out of distribution is things that are produced inside of the company, right? So like, you know, the very extreme example of this, right, is a shout out to Liam, who used to be my main manager, Liam Fettis.

• 10:31Speaker 00

  Yeah, that's going to be out of distribution.

• 10:33Speaker 01

  Extremely out of distribution, right? Because they're making that data, they're actually going in the real world, running experiments, getting reward signal, things like that. But I think in the enterprise, what that looks like is like judgment traces, right? You know, the way one company operates can be very different than the way another company operates because, you know, they have different thresholds for risk, they have a different operating model, they have different expertise, historical data, all that sort of stuff. And that contributes to the decisions that they make. The way we think about out-of-distribution data is maybe like, we're putting it in the context of offline RL where it's like, oh, there has to be this like highly verifiable signal from like some math problem or some, you know, rubric-based RL problem or something

• 11:18Speaker 01

  like that. There are times when you're going to go and optimize an AI system by configuration. So you're going to go and prompt the model, or you're going to put things in context. And there are times where you're actually, it makes a lot more sense to optimize the policy. So how it does judgment, how it does reasoning. And these things are actually like harder to codify. And that's where we'll see, you know, in five years from now, I think, you know, a lot of companies will basically have systems to basically gather input from experts and then turn that into better policies, yeah.

• 11:51Speaker 00

  So, you know, as these models get used, you know, you kind of codify a bunch of the like, the way that an organization makes decisions and you kind of use that to create, you know, to improve models. Is there like enough data that's generated to do that? I guess it feels like, you know, to do RL, a lot of these things requires a large amount of data.

• 12:08Speaker 02

  Hi, I'm Anne Maris, and I'm the producer of unsupervised learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership and see you next episode.

• 12:25Speaker 00

  Is there like enough data that's generated to do that? I guess it feels like, you know, to do RL, a lot of these things requires a large amount of data.

• 12:32Speaker 01

  I think like, you know, part of this is, this is like a greenfield space, right? Like, there's new research happening here, and it's all about data efficiency and learning from sparse rewards. So, you know, we had pre-training, it's like, not that data efficient, but very generalizable. Then we had this like supervised fine-tuning, you know, pun on unsupervised learning, right? Is this the name of the podcast? But supervised fine-tuning, it's like more data, like more data efficient, but still not very good. Then there's RL with verifiable rewards, where you actually trade data quantity for quality with these like replayable environments and some sort of verifier propagate that are all taken by, it's still pretty bad, right? Like, it's still not very data efficient.

• 13:15Speaker 01

  The answer to your question is like, there will be more data efficient methods of training, and that's what a lot of these newer methodologies in online RL or on policy self-dislocation or some of these like type of techniques or SAO, these types of things that are, people are pushing on. And I basically think like, what's blocking continual learning is basically like extremely data-efficient training from sparse rewards, which is an unsolved problem. Yeah.

• 13:45Speaker 00

  How do you think about as a CEO, the extent to which you should be like working on that problem within the applied compute context? Or like, it's like, hey, just understand the enterprise problems really well and like, as that research progresses, you can kind of take pieces?

• 13:55Speaker 01

  Yeah, so we're actually doing this today with customers, this sort of like online training where we're actually able to take advantage of, you know, large inference deployments where people are serving tens of thousands, hundreds of thousands of users, and then use that production data to go and improve the models. We aren't married to like any specific type of training. RL and these online methodologies are the best things right now, but these will continue to evolve and get better. That's why a large portion of our team is actually a research systems team, trying a lot of new methodologies in production with real customers.

• 14:31Speaker 00

  There's a ton of potential on updating models and training models. And also, I'm sure anyone that's played around with these tools has realized like, you know, I don't know, if you do some really great prompt engineering or create some skills or like, there's other ways to like take knowledge. And I think there's maybe a feeling sometimes that, you know, researchers can be a little bit of like a hammer looking for a nail, like, oh, we can do it with the model. There's other ways to do it.

• 14:49Speaker 01

  Yeah.

• 14:50Speaker 00

  How confident are you that like, the long-term solution of this involves like many, many different kinds of models than organization versus like stitching together things in other ways?

• 14:59Speaker 01

  Yeah, I think, I think we are pretty confident that the future looks like a non-static set of weights. I think that is something that we deeply believe in. There is going to be some sort of feedback loop from the real world back into the, back into the model. Does that need to be done for every single task?

• 15:21Speaker 00

  There's been some good examples of that in the coding world. But like, what are, yeah, what do you feel like's above the line of where that feedback loop makes sense and maybe what's below it?

• 15:29Speaker 01

  Basically, things that change with temporality, you often want like some sort of closed loop where new information is making its way back into the model. But also, I just think like, there is this expanding umbrella of intelligence that you're getting from the base models out of the box. And at some point, that is going to be like sufficient for a lot of these like manual, you know, process-oriented tasks. The things that will require sort of iterative training are the things that are more judgment-oriented.

• 16:00Speaker 00

  You know, are there, like, to the extent you can talk about it, examples that come to mind of like ways, you know, of kind of taking this like real-world data and using it to kind of improve, I mean?

• 16:09Speaker 01

  Yeah, yeah. So basically, like, you know, one of the examples of customer work, working with today is they essentially have like some sort of assistant product that a lot of users use. You know, they ask this assistant to go and accomplish different tasks and then give feedback on what the model did wrong or right. And so instead of, basically, it's pretty impractical to go and build like some sort of giant skills file or context or something like that. So we're actually able to bias the model based off of the direct user feedback using some of these like online training methodologies. And we're able to see like real upticks in user satisfaction, completion rates, decreased

• 16:54Speaker 01

  tool call failures, things like that. The main question and answer is like, what's the TCO of doing this? Like, how expensive is it to post-train? And that's like a lot of the stuff that we're going and building is how do we actually decrease the cost to go and, you know, RL a model? How can we actually do this automatically and online without you having to go and build hand-tuned RL environments that are fully replayable and things like that? And so what we see happening is the cost to post-train and actually adjust the weights is falling. You're still better off going and doing a bunch of context optimization and harness optimization. But once you reach to, reach a point where you've sort of squeezed a lot out of the

• 17:37Speaker 01

  harness that you've built and the tools that you've made, you actually can still continue to squeeze a lot after, out of optimizing the policy to use those tools better. So, you know, some of the companies that we work with, they have really specialized harnesses with custom tools and things like that. Like, you know, we're working with a company in chips. They've built like a bunch of software actually that the general models don't really know how to use well, to go and do like, you know, ETL verification and things like that. So turns out when you apply this like, you know, Goldilocks algorithm of RL, you can actually push the models to like a lot better performance.

• 18:16Speaker 00

  You talked about kind of like this, this effort to like lower the TCO and, you know, enable like more continuous training. What are the hardest problems to solve there? And like, you know, where are we on that journey?

• 18:25Speaker 01

  Yeah, the hardest problems to solve with RL and infrastructure.

• 18:28Speaker 00

  Yeah, to be able to bring the like the TCO down to the point where, and then like productize it even more.

• 18:32Speaker 01

  My co-founder, Lyndon, who's our chief architect, he is like the wizard with this, he's one of the smartest people I've met. And, you know, you'd be surprised how underutilized GPUs are. Like, if you were to also just aggregate things globally, like, basically, I would say There is a lot more that can be squeezed out of like the available hardware um to to actually go and lower these costs quite drastically. You can invest a lot in algorithms that are a lot more data efficient, so you need to train uh uh uh less uh you know, for less long and and use less compute. Um and so I would say like the system side

• 19:17Speaker 01

  and the algorithm side are like the places where you can squeeze the the the the most right now. In terms of like the data available and things like that, you know, there there there's synthetic data generation techniques to be like, hey, like, can we go and take some amount of tokens and explode it into something much larger and pull learning single signal out of that? Um but I would honestly say that most of the cost optimization and cutting cutting down costs there is gonna happen on the algo side and the system side.

• 19:48Speaker 00

  Satya came out and said that there should be, you know, as many models in the world as there are firms in the world.

• 19:53Speaker 01

  Yeah.

• 19:53Speaker 00

  Um which obviously was a good a good a good selling point for you for you all.

• 19:57Speaker 01

  Yeah.

• 19:57Speaker 00

  Um you know, is that a vision that resonates with you? I mean, how much do you believe in this vision that like every single company is like its own special snowflake with its own

• 20:04Speaker 01

  Yeah.

• 20:04Speaker 00

  unique judgment that requires like its own model?

• 20:07Speaker 01

  Yeah, I think I think in principle, um you know, wh- whether this means that every company needs to like pre-train a model or mid-train a model of their own, um you know, I I don't necessarily think that's the case, but I do think there are closed-loop feedback systems inside of each company that is worth capturing. Right now, there's just a lot of low-hanging fruit, so people are starting there with like harness optimization and context optimization. But the idea that like different companies are unique and they operate differently and they have different judgment structures and are not the same, um that we we fundamentally a- agree with and and that's kind of the thesis of our whole company is that like

• 20:51Speaker 01

  you know, if everybody is using the same model, like great models uh sort of set the the the the floor for everybody. But then how you go and optimize those models and build really amazing AI systems, that's what's gonna set the the ceiling. So there actually is a, you know, a difference um uh in the products that people can build both on the harness and context engineering that they do, but also the models that they put behind those products.

• 21:16Speaker 00

  Certainly, you know, uh uh you take a one extreme like a pharma company, like pharma companies have such different data from each other. It makes so much sense that like, you know, they focus on one disease area, generate a lot of experimental data, like of course they should have their own model. If you take like banks, it's like how different is like one bank from another bank? And like certainly, maybe there's little judgment, you know, calls that are different, but I wonder like, you know, if a frontier lab is is post-training on a bunch of like really good, you know, general finance data, how much uniquely unique things is

• 21:42Speaker 01

  Yeah, I think I think there are certainly things in the market that will benefit from consolidation. I mean, that's this is why you have roll-ups, right? Like uh you basically to, you know, to take like accounting firms or something like that, like they're fragmented just by the nature of the business, I think, as like a very um you you know, like a you have an accountant and you have a client and they're one-to-one match or something like that, so it makes sense, hey, let's go and take the menial sort of labor out of doing the work of an accountant and put that into a model. Those types of things, I think, you will see sort of consolidation. When it comes to, you know, how um different companies operate like, you know, like banks, though, I think there is a whole

• 22:28Speaker 01

  litany of decisions and um sort of like trade-offs that each company makes that is different than the other.

• 22:37Speaker 00

  Yeah.

• 22:37Speaker 01

  Um so they're they're they're less of like, I guess, like a commodity where you can use one or the other. Um and that's kind of why you have like such a wide distribution of banks that are doing well, banks that are doing poorly, I think. Um there actually is like room to to compete.

• 22:53Speaker 00

  It'll be fascinating to see, right? I mean, it like, you know, essentially, we'll kind of learn the extent to which a lot of these industries, the companies are actually particularly different or have different ways of doing things.

• 23:02Speaker 01

  Yeah.

• 23:02Speaker 00

  Um by the the extent to which like having your own model is actually beneficial relative to using whatever the the off-the-shelf average, uh you know,

• 23:10Speaker 01

  Totally.

• 23:10Speaker 00

  Merck or data labelers.

• 23:11Speaker 01

  Totally, and and and I think the way, you know, one of one of the ways that this actually manifests is not always capabilities, right? One of the reasons why we feel very excited about what we're building right now is we are in a supply crunch, right? Like there is like limited compute, everyone's kind of feeling this. Um so the more you can squeeze out of your models and the more you can squeeze out of the compute that you actually have available to you, um the more stuff you can do, right? So so basically, that's why we think having the infrastructure to go and optimize models and make them more efficient, even if you are um pushing the Pareto curve out and you're not necessarily pushing the frontier on capabilities, that delta is some sort of opportunity or advantage, right?

• 23:55Speaker 01

  If I can do something 10x cheaper than my competitor, that is differentiation and that is something that I can I can actually uh uh win against them out on the competitive landscape.

• 24:06Speaker 00

  I guess maybe zooming out and talking about like, you know, people that are at the precipice of of deciding whether they should embark on their own models. I mean, obviously,

• 24:13Speaker 01

  Yeah.

• 24:13Speaker 00

  you've talked about, you know, you should try a lot of the harness optimization stuff first and, you know, a a lot of other things. But as you, you know, I think a lot of folks are trying to figure out how well does RL work in like these non-verifiable domains?

• 24:24Speaker 01

  Yeah.

• 24:25Speaker 00

  How do you kind of characterize where we are today?

• 24:27Speaker 01

  Yeah, so I was, what I would say is like, you know, RL started or like the the way these projects started and the internal models that were being trained were on highly verifiable domains like math and coding. Partially because they're highly verifiable, partially because everyone at the lab is like loves math. It loves good, but it was more of more of the former. Um I think like it it is a surprisingly easy to hill climb on non-verifiable domains if you turn them into some proxy verifiable domain. So like Rubrik-based RL and this sort of stuff um actually works quite well. Um so I would say like the state where we're at right now is like if it's not something that has one definitive answer, basically giving it a expert answer

• 25:11Speaker 01

  and grading against that serves as quite a good proxy. So that's actually why, like, you know, different companies have different different experts. And so optimizing to two different groups and two different companies can actually give you like very different models.

• 25:26Speaker 00

  So to keep all your ex-employees off of the data labeling.

• 25:28Speaker 01

  Exactly. Yeah, yeah, no, I I think I think it's actually um what we have with RL is we essentially have a hill climbing machine.

• 25:37Speaker 01

  The hardest part is actually defining the hill to climb, which is why evals, I think, is like a really important thing for companies to A, focus on building and B, actually like safeguard uh pre- pretty pretty carefully. If you are thinking about competition and stuff like that, if you look at every public benchmark, it's gonna get benchmarked, right? So um if you have things that are very particular to how your business operates, uh and you want to make your models really good at that, it's actually to your advantage not to tell everybody else uh what good and bad looks like, so people can go and make their things really good at that. So that's why I think like, yeah, like it is really important to actually keep your evals and

• 26:21Speaker 01

  employees and all that sort of stuff, like you wouldn't your employees are not fungible, you would not like be comfortable with them going to another company and doing work there. Same thing with these models.

• 26:30Speaker 00

  It's like there's this like tension if you're a leading app company where obviously like you want to just expand the set of capabilities, you you can deliver to your customers and so going to labs and telling them your evals is helpful. And then the other side, it's like a lot of it's hard-earned insights, like.

• 26:44Speaker 01

  Totally, and it's it's a tough position, which is why I think like, you know, investing in open model infrastructure, investing in a multi-modal future is like something absolutely every company should be doing. Because, yeah, you're kind of caught between a rock and a hard place.

• 26:59Speaker 00

  Yeah.

• 27:00Speaker 01

  Yeah.

• 27:00Speaker 00

  Um I guess, you know, I'd love to shift to like just a black and pure like the company.

• 27:04Speaker 01

  Yeah.

• 27:04Speaker 00

  Um and I'm wondering, I'm sure that like there is no shortage of enterprises that would be like, it would be pretty sweet to work with you guys and like, you know, get you rolling up your sleeves and and helping whatever the set of problems is. How do you like qualify business and figure out, you know, who you work with and who you don't at this stage?

• 27:21Speaker 01

  Great question. So, you know, we we um basically see post-training uh and actually like training these custom models as, you know, the differentiation and the part that makes the business sticky, um and actually like provides more value on top of just the off-the-shelf models. But really inference is actually where like most of the value capture happens, right? If you look at the dollars spent on models,

• 27:46Speaker 01

  the vast majority goes to actually serving and using the models than training the models um by themselves. So when we work with with with with customers, we typically go after very high-value use cases and that's either like because the capability increase in the model that we're gonna provide is extremely valuable. So some of these like uh uh companies like that you were describing before, like a pharma company or a cybersecurity company or chip company or something like that. The other is if the inference workload is really big, right? So so basically like amortizing, basically some sort of efficiency or optimization gain or quality improvement distributed over a very large uh user base or or, you know,

• 28:30Speaker 01

  billions or trillions of tokens, that's actually super valuable as well. So we we actually look for like large inference workloads. These are typically the best customers for us or um, you know, very out-of-distribution data where people are wanting to train on things that are kind of only unique to them.

• 28:46Speaker 00

  You you kind of mentioned the inference side of things. Obviously, it feels like a temptation for like any infrastructure company these days is like, you know, having an inference business is pretty nice.

• 28:54Speaker 01

  Yeah.

• 28:54Speaker 00

  Um I feel like there's different schools of thought on this, like how difficult of a product is that to build, like a really good inference engine?

• 29:00Speaker 01

  So so I think like, you know, maybe taking a step back, like why we think um So first of all, we think inference is gonna be like one of the biggest markets in the world, if not the biggest. Tons of people have said this. It's not like a very unique take. I mean, it's kind of obvious if you believe in AI doing a lot of, you know.

• 29:20Speaker 00

  Not gonna get a lot of training dollars if there's not.

• 29:23Speaker 01

  Exactly. And so, so what we, what our view is, is that the most post-training actually wins inference. So, you know, if you are, basically what the trend we've been seeing is like, companies will start off with frontier models, they'll do a lot of experimentation. Then their workload will become mature. They'll be scaling that workload. Then they'll think about open models and how to sort of cost optimize, perform, basically push the Pareto curve out. And the interesting thing is that the most at-scale workload, so the largest workloads where you're spending the most money, is actually where you're gonna wanna go and post-train first, right?

• 30:05Speaker 01

  Because that's actually where the most gains are that we had, and it's kind of power law between all the other use cases. And the interesting thing you could do is you can both co-optimize your inference and your, your, your training. So, you know, basically, the way we train models actually influences how we go and set up inference deployments for customers. You know, if it's like a heavy tool-using model and we train it to do a bunch of like tool-call parallelization, that kind of stuff.

• 30:33Speaker 00

  Can you say more about that? Like, what does that actually mean in practice? Like, how would you, you know, if the model, you know, if, if it was designed that way, how does that influence the inference choices you make?

• 30:40Speaker 01

  Yeah, yeah. So, so it's a great question. So basically, like, if we are doing like a long-horizon tool-using agent, the way we sort of like set up the inference deployment or the inference engine is different than if we were doing like a heavy, like a decode-heavy sort of workload, right? So, you know, and you could, you can sort of set up these like disaggregated setups where you're using multiple different types of chips for pre-fill and decode. And so, having influence on how the models are trained actually can influence how we wanna go and set up the inference deployments for very large-scale workloads.

• 31:13Speaker 00

  It's interesting. I'm really curious to dig into this because I feel like there's been this question for a while of like, how much does creating the model help you run better inference? And it originally showed up in like a lot of the original open-source model pitches, right? Where the idea was, hey, we train these models, so we're able to run like, you know, inference way more effectively on them. And then like, the way it played out, at least in the early days, was actually, you know, someone like Fireworks would be like, "Thank you for, for training a great open-source model, and that will be the ones to run that like as efficiently as possible."

• 31:41Speaker 01

  Yeah. So, so I think, I think what's, what's happened is like, yeah, basically like the kernels that you use for, for training and inference, like you, you obviously want, you know, matches between them. We do a lot of work on the, the training side to go and optimize, you know, how fast we're able to train these things, how efficiently we're able to train these things. And a lot of that work translates over to the inference side. I think also one other really interesting thing to point out, this is kind of unrelated to what we're just talking about. One thing I think is really often overlooked is, you know, you can do a bunch of inference optimization and things like that. If you wanna make your inference, cut your, say you wanna cut your inference bill by 10%.

• 32:23Speaker 01

  You could do a bunch of inference optimization, cut costs by 10%. Or, I think these models are actually quite token inefficient for a lot of things. So what we've been doing with customers is say, "Hey, we can actually cut your, your token bill by making your model 10% more token efficient." Preserving the same performance against some sort of eval that you care about. And so, I think like, one of the, the, the, the things that we've been able to tell customers is like, "Hey, like, these things are two sides of the same coin. You can actually influence both of them if you do it with us." Which is why we're trying to be the centralized provider for both training and inference.

• 33:02Speaker 00

  It'll be interesting to see, yeah, the extent to which doing the training yourself gives you like even more insight into the inference workloads.

• 33:08Speaker 01

  And I think like, you know, back to your original question about, about like inference offerings and how easy they are to spin up. I think like, what we've seen, right, is there's, actually the main thing to have is, is, is capacity, right? So, so, you know, you need the actual GPUs to be able to go and serve the models and things like that. And the open-source projects like VLM and SG Lang are like great starting points. In fact, I think like almost all of the inference clouds have like sunset a lot of their custom inference engines in, in preference for taking one of these inference engines and doing the workload tuning and performance tuning, and maybe having a couple of custom kernels in there.

• 33:53Speaker 01

  So, you know, I think Dylan Patel was on like Dork Asher's podcast, and he was actually talking about how easy it is to basically, if you have some GPUs, you put like a VLM on it, and you throw it up on a, a open router. You know, you kind of have a production inference offering. Now, there's, now, I don't wanna trivialize it too much. I think there is like a lot of optimization and hard work that goes into making these inference deployments like a lot more efficient. You know, faster, faster token speeds. And that matters, especially as you hit scale, right? Because like 1% of savings across tons and tons of chips actually adds up. But in terms of like actually getting production-quality model-serving going,

• 34:38Speaker 01

  it's, it's, it's definitely the easier thing to do out of like training and inference.

• 34:42Speaker 00

  Yeah. It's interesting. I feel like as an infrastructure CEO, you must, it feels like there's a bunch of companies starting in different places, and there's like this desire, you know, or this kind of belief that like over time, a lot of them will go pretty horizontal and like, you know, intrude on each other in all sorts of ways. And I imagine you both have to figure out like the sequencing that makes sense to do here, as well as like figuring out the red line, do you wanna draw of like, okay, like, you know, we wanna do lots of things, but we don't wanna do that. Like, how has that like evolved over the course of the business, and how do you think about it today?

• 35:09Speaker 01

  Yeah, it's a great question. So, you know, our large-scale vision, so what we're actually building is we think there's like a new AI hyperscaler to be built. Like, in the cloud era, you had, you know, you have AWS, GCP, and Azure. They took CPU compute, they built really awesome flexible compute products on top of it, like for networking and storage and databases and data lakes and like stuff for building applications, all that. Now, you have this new compute substrate, GPUs. You have this weird thing, which are, which are, this collection of weights and parameters called models. Turns out, you can serve them with an inference engine, which is the first piece of software

• 35:54Speaker 01

  that has like achieved real, you know, stickiness inside of the AI software stack on top, and you can serve these valuable intelligent tokens. Our view is that there's a ton of stuff in that software layer, actually, besides just inference. Our big bet was, hey, post-training is a massively underlooked part of the stack. How do I take GPU compute and turn that into more intelligent tokens? That's training. You know, how do I serve those tokens? That's inference. How do I pick the right tokens to serve? That's routing. But there's also a ton of stuff like security monitoring and, you know, identity, agent sandboxing, all this sort of stuff. So, when we think about what we're building, we actually wanna be very horizontal.

• 36:36Speaker 01

  But we actually wanna start, we think, kind of think about things as an inverted pyramid. There's things that very few people can do at the bottom of the pyramid, and things a lot of people can do at the top of the pyramid. So, at the very bottom of the pyramid, that's where we think model training is. And then above that is inference, above that is routing, above that is context and harness and all that sort of stuff. And so, our strategy has really been focused on the things that are extremely high leverage and extremely high ROI that are very hard to do, that you need to build super robust infrastructure. You need a GPU fleet that you actually, you know, own and operate and all this sort of stuff. And so, our goal is to, as an infrastructure company, widen the offering as we, as we mature.

• 37:21Speaker 01

  And, you know, we're 16 months old right now, so we're, we're, you know, the, the first step was going from training into inference. But we have a lot of exciting things on the roadmap and things that we wanna go and build.

• 37:33Speaker 00

  What's so interesting is that I imagine when like 80% of your pipeline approaches you, you're like, "This could probably be solved by like, you know, a good harness and like some data work," or something. And so, what, you know, what do you end up doing on like the harness optimization side? Like, I imagine a lot of folks are thinking through this right now.

• 37:50Speaker 01

  Great question. I think, I think we are really going for focus right now. So, we actually, you know, the customers we work with have mature AI products. They have already like, they've built amazing product teams, and they are investing in creating really great harnesses. And so, our current operating model has actually been, we're not focused on the harness layer of the context layer. We're focused on the model layer, and, you know, we, we don't necessarily do the zero-to-one agent builds. We, we optimize the models behind mature products.

• 38:21Speaker 00

  Makes sense. And I guess in terms of like the building the RL environments today, maybe like, I'm sure our listeners would love to hear like, what is actually, you can even just take an illustrative example. Like, what is actually involved in like building one of these? And, you know, how do you think about like, I'm sure there's 500 annoying parts of it, like, the parts that need to get better to, or that you hope get better?

• 38:39Speaker 01

  The, the nice thing is, like I said before, RL is just like a hill-climbing machine, right? So, it actually is, the same way you're gonna go and build your training data is actually the same way you're building your eval. So, hopefully, you know, that can help folks like kind of conceptualize, you know, what is, go, what goes into building great eval is like, hey, you have to have the right, you have to have the right task, and you have to have the right environment, and you have to have the right verifier, right? Those are kind of the three main things when you're going to do RL. What is the problem? What is the scope of tools and services that the model has access to? And then how am I gonna judge whether it's good or bad? So, take for example, you know, we did this Harvey Review Table.

• 39:27Speaker 01

  Custom model that we we train, we put out a case study about it. But for that, right, we basically had worked with, you know, Harvey, Harvey, the Harvey team. Were able to correct, collect a bunch of experts, rubrics and answers. Also supplemented those with like synthetic data and gave the model access to some tools. Were able to say, hey, here's the task that you're supposed to go, you know, extract all this information out. Graded against the expert answers with, you know, real request RL. And that's how we were able to create the task. The environment here was the tools that the model had access to in the document base. And then the verifier was the grading against the expert answers.

• 40:08Speaker 00

  Is the hardest part of that, just like actually getting people to create these rubrics or like, you know, create, you know, what are realistic documents that would live in these things?

• 40:14Speaker 01

  Yeah, so we we we work really closely with all of our customers because they are the experts in the domain, right? We we really come in with the infrastructure and the training team and the inference, you know, just production serving and stuff. So, you know, this was like pretty, pretty collaborative. We oftentimes can do a lot of work on the synthetic side. If it's like some, say, you can take some sort of document and back translate it into, you know, question, answer pairs, for example. That's a very concrete example. We could do that stuff synthetically, but when it involves like real experts and real lawyers, we actually want to work with the teams directly.

• 40:54Speaker 00

  Yeah, and I'm sure like, you know, the the entire thesis that seems to be like that type of work is going to be way higher quality than like whatever lawyers, a data labeling company gets on and tries to like.

• 41:03Speaker 01

  Totally.

• 41:03Speaker 00

  I think, you know, probably distant memory at this point, but like OpenAI shutting off cursor or like, do you expect that kind of stuff?

• 41:09Speaker 01

  Winserve before, right?

• 41:09Speaker 00

  Of course.

• 41:10Speaker 01

  Anthropic with Winserve, yeah.

• 41:12Speaker 00

  Do you expect that kind of stuff to happen more or like?

• 41:15Speaker 01

  I mean, it happened twice. I think competitive dynamics are real and like their incentives are real. So especially as companies start to move more into the application layer and build their own competitive products with their largest consumers of their models. I wouldn't be surprised if it happened more.

• 41:32Speaker 00

  I always like to have a quick fire round where I basically shove in like everything that I, in an unstructured way that I haven't gotten to before. And so maybe to to to start, is RL going to generalize?

• 41:45Speaker 01

  In some ways, in some ways it is generalizing, right? Like a model's ability, models abilities to, you know, do error recovery or, you know, use tools and subagents, things like that. But I think like if you actually empirically, you can like run these experiments, you can train on a math dataset and see how much that translates to something in the legal domain or something like that. The transfer is is not really there, certainly not to the extent of like the generalization that pre-training has. So I think it will, it will bleed into adjacent areas, like intelligence is some sort of sphere or circle.

• 42:30Speaker 01

  It's got jagged points on it, the points that are closer together, you'll see more generalization there. But certainly does not have the same generalization as like a pre-training or something like that.

• 42:40Speaker 00

  Yeah.

• 42:41Speaker 01

  Yeah.

• 42:42Speaker 00

  What have you changed your mind on in the last year?

• 42:44Speaker 01

  Yeah, I mean, I guess one thing that I didn't, maybe I didn't change my mind on it, but I didn't fully appreciate is how real Jevons paradox is.

• 42:56Speaker 00

  Where are ways that are like a particular thing you saw that?

• 42:58Speaker 01

  Yeah, like, like, you know, I think one, one interesting thing is

• 43:06Speaker 01

  we are in, like, basically, like, if you look at some of the open router statistics, whenever there's a price cut for a model, usage just spikes. So I really think we're moving to a world where things are like extremely cost optimized. We're feeling the pressure, like everyone's feeling the pressure right now to like make things as efficient as possible, lower the AI bill. I don't think I fully appreciated how much of a driving factor that would be over something like capabilities. Like, we are reaching a point where I would say

• 43:47Speaker 01

  most, most of the open source models can kind of do everything that you might want them to do. There are obviously a lot of frontier things and, you know, I don't want that to be a blanket statement. Frontier models are amazing, but I didn't, I didn't expect how much people would care about really cost optimizing and using the right model for the right task.

• 44:08Speaker 00

  Yeah, what is interesting because obviously for some of these tasks, like you, you, people were using frontier models because you got a lot of value.

• 44:14Speaker 01

  Yeah.

• 44:14Speaker 00

  But if there is a way to to do it much, you know, much cheaper in the use case.

• 44:17Speaker 01

  Yeah, there's some marginal return of value for like, and and at some point, people are going to do the experimentation, try a bunch of things and then consolidate on the right model.

• 44:26Speaker 00

  When something like Jev comes out, how does, what's that like at applied compute? And like, how is that, how have you guys thought about that?

• 44:32Speaker 01

  I think it's a super cool model. I basically think like there are, there are probably going to be a bunch of things like that coming out. It enables a whole set of new use cases. I mean, it's in the name, that's why they called it Jev, right? Like things that are cheap, fast, designed for like a specific, particular set of tasks. That's how we've been thinking about a lot of things that we're building in applied compute. So models like Jev, you know, we're working on, hey, what does this look like to integrate into the training process? As sort of like massive billion token scale classification on rollouts or things that we couldn't do before that were cost prohibitive.

• 45:14Speaker 01

  So, yeah, when things like that come out, I think it's like super exciting because it basically just expands the amount of things or that we can go and build. And it's some sort of matrix, right? So it's, it's, it's, it's highly multiplicative, like these things actually end up stacking. Jev is like much, I would say it's like much less of a replacement for stuff rather than an enabler for things that we weren't able to do on both training and on like inference for, say, for observability or something like that.

• 45:44Speaker 00

  Yeah, just because like at that kind of like cost level.

• 45:46Speaker 01

  Yeah, at that, at that cost, at that cost, or that price point, it's basically like enables like a whole new set of things that we can actually do that we couldn't have done before. Even with like the smallest, cheapest Qwen model or something like that.

• 46:00Speaker 00

  Yeah, is there like a visceral example for you of that that you've seen?

• 46:03Speaker 01

  So we, so we actually put this out in, in, in a blog post. We basically used Jev to do essentially like massive trace analysis for online training.

• 46:17Speaker 00

  Because I imagine you're on this like exponential slope of growth. So how in the world do you budget for like the amount of compute you want? Because you're probably entering into some of these.

• 46:23Speaker 01

  So, so I think like, you know, we, we, the way we think about it is we're trying to actually make like a very sticky differentiated business. So we aren't going after like every single opportunity that we could that like, oh, like, you know, someone needs inference or something like that. We are very sort of convinced of this thesis that people will use custom models. They will want to invest in the infrastructure that we use to post-train. So we actually are, instead of looking at like the total demand for everything, we actually are like, what are the most strategic companies that we can go and work with?

• 47:02Speaker 01

  That are, that we can grow with them and they're also, you know, the right folks to basically go and prove out this, this thesis that, you know, custom models are much stickier and valuable and the ROI is is persistent besides just some sort of like off the shelf model.

• 47:19Speaker 00

  Two things I wanted to hit on that I've heard you say, you know, one is, you said something that I thought was really interesting is that like the best engineers you know, like all learned to code before AI existed.

• 47:29Speaker 01

  Yeah.

• 47:29Speaker 00

  And I wonder if you expand on that a little bit and kind of like, you know, there's all these, you know, 16 year olds now that are, that are, you know, coding and kind of like native to these, you know, have only learned to code with these AI tools. How do you think about like advising folks?

• 47:43Speaker 01

  Yeah, no, it's, it's, it's a great question. I basically, so to be clear, I'm not against AI coding.

• 47:49Speaker 00

  I'd imagine you use a fair amount of AI.

• 47:50Speaker 01

  Yeah, exactly. And and also when I was at OpenAI, the main thing I worked on was Codex. That was actually like, you know, the thing I was spending all my time on. So I'm like very excited about AI coding. I think there's a difference between like delegating your thinking to coding agents versus using them to go and execute on some sort of like plan and design that you've thought of. So what I would say is like, I, I actually think the best thing for people to focus on is, is like system design. Like I, I, you know, funny coming from like an infra company or something like that, but the, the types of problems you face in like building scalable systems really force you to think about

• 48:35Speaker 01

  the things that you're, you're, you're doing and why you're designing and structuring things that way. I think the models are getting really good, so you don't need to like know the exact syntax and all this sort of stuff. It's more, much more important to get like the, the frameworks and design right than any of the actual coding. So I, yeah, I, you know, I definitely am a big fan of AI coding. It just, you can't delegate your thinking away and, and, and basically rely on it as a crutch. Because then you get these like massive slop code bases that you can't actually explain. So, so probably the best thing is, you know, learn how to explain your code. That's what we, that's what we interview for is like, we'll let someone go ham when they're on their onsite with us.

• 49:20Speaker 01

  But then we'll ask a ton of questions about why'd you build this this way? What were the trade offs you considered? And if they said, oh, Claude did it or like Codex did it, that's like, you know, not something that we're pretty excited about.

• 49:32Speaker 00

  Yeah, interesting. How have you, like, obviously you're, you know, I feel like everyone's trying to learn from the most AI native companies on how they do things. How have you thought about, like, so obviously that's interviewing on the engineering side. You know, I'm sure you hire other roles and I'm sure you want people to be relatively AI native and fluent. Like, how have you thought about, you know, interviewing or changing processes around that for other types of roles outside of engineering?

• 49:50Speaker 01

  Oh, great question. Yeah, I think, I think like, you know, when we. Interview folks, it's kind of the same principles apply. So, you know, for, for, for like finance roles or biz, biz dev roles or things like that. People will often like use AI tools to go and build like a, a, a, a financial model or something like that. And if they can't explain why they built the model that way, that's like basically what we're looking for. Really, it's like, our view is people should use whatever tools that are available to them, but they should have like a complete understanding of everything those tools went and did for them.

• 50:33Speaker 01

  So we, we, we do much more of like, our interview process is unconstrained, but then very particular in like how we audit or essentially assess understanding of the outputs that were produced.

• 50:47Speaker 00

  Yeah.

• 50:47Speaker 01

  That makes sense.

• 50:48Speaker 00

  Yeah, makes a ton of sense.

• 50:49Speaker 01

  Yeah.

• 50:49Speaker 00

  I guess you, you gave this kind of lecture at Stanford where you said, you know, I think they made you say something you were short. And you said you were short the, the general data vendors, right? Like Merckor and Surge. Curious why that is and, and where you might be wrong.

• 51:00Speaker 01

  Yeah, no, I, I, I guess like, I'm not, I'm not short the companies themselves. I actually think they're great companies. We, we serve a lot of them as, as customers. They're amazing companies. Basically, I, my, my main point there is basically like a lot of data is becoming more and more synthetic, uh, synthetically generated. So you can use these smart models to go and create the types of RL. Actually, in fact, actually, a lot of RL environment companies, uh, end up using, uh, the models to go and create data that actually feeds back into the model. So it's a little bit of a, a, a, a circular motion there. Sort of the big point was kind of like paradigms change all the time.

• 51:43Speaker 01

  So, you know, uh, for, for, for data companies, I think the main thing to get really good at is pivoting, like going from each paradigm to the next. So, you know, we used to be in, uh, uh, SFT land, uh, preference tuning. Then we, uh, you know, then like RLHF, human feedback, then RLVR. Now the big things like robotics data. So I think like, um, you know, my, my, my, my point there actually was less like, hey, you know, these aren't good companies. I think they're great companies. It's just that the particular RL and, uh, data creation company is gonna move around a lot. It's gonna be quite fluid and it'll, the, you know, the paradigm will change. The research paradigm will change.

• 52:27Speaker 00

  Among the RL environment companies today, like what makes, what makes an RL environment company better than another?

• 52:32Speaker 01

  I think it really is the, the, the human part. Like, uh,

• 52:36Speaker 00

  They're able to get to.

• 52:37Speaker 01

  Yeah, exactly. The expert networks, so like Merckor and, you know, Handshake and these types of companies, they have incredible expert networks. Actually, like, uh, when I was at OpenAI, um, you know, we, we worked with Merckor and I think they, you know, they obviously still do, um, but we worked with them very closely for some of these like first agent products, like deep research and, and Codex. And that is where you are getting like the best of the best in the world for, um, coding or math or stuff like that. And there's still signal that needs to flow into these models. So models are a function of data. The higher quality of the data, the better your model.

• 53:16Speaker 01

  So, uh, that's what I would say probably differentiates is like the ability to very quickly run a campaign across a wide network and basically narrow that down to the highest quality data points. Because when you're going and training, you're not actually using that much data, right? You're using a few hundred data points, all of those each, a few hundred to a couple thousand per task or something like that. And so if you get, if your, if your quality assurance is like pretty low, that has a material effect on, on the models, like final.

• 53:50Speaker 00

  It feels like a lot of this, like, you know, the power companies, models will be determined by this delta between the like ex-bankers that Merckor can recruit and like figure out what the tasks are.

• 53:59Speaker 01

  Yeah.

• 54:00Speaker 00

  Versus like the, you know, bankers within JP Morgan and like the stuff, the tasks they have and like the evals they create. And like, I don't know, I don't know what the gap ends up being between those two.

• 54:08Speaker 01

  Yeah, well, I think like when banker or like take for example, your, your example, when bankers leave, they don't get access to like a lot of the proprietary data that's been built up, uh, over time. Like they're obviously doing lots of, uh, you know, reasoning and judgment and stuff like that. But it's also have very heavily informed by the data that they look at and that they, that they sort of do their work on. But yeah, I think it's like a real, I think it's like a real source of like, uh, alpha permeating the boundary of an enterprise. I mean, it's always been the case, right? Like, that's why people have like non-competes and things like that, right?

• 54:46Speaker 00

  It almost seems like, uh, yeah, like, you know, obviously you've seen this trend of, you know, a company goes under and people buy it for the data. It's like, if I was a, a bank and a bank went under, I would not want like one of the labs to go acquire like whatever. I don't know what data they can actually train on.

• 54:58Speaker 01

  No, exactly, right? I think, I think it's like, and, and that is almost a sort of argument for why these companies are valuable, right? Like that data is valuable, so that company is valuable. And therefore, the more you can keep things sort of like proprietary to you, um, the better.

• 55:18Speaker 00

  Yeah, and it'll be so interesting to see like what that boundary ends up being. It's like clearly gyms, you know, biology, like that feels like it's hard to get out there. Whether like the labs eventually figure out kind of the basics of running a bank or something else will be.

• 55:30Speaker 01

  Yeah, well, I think, I think there's just a lot of, I mean, running a bank is like, it's like a thousand layer cake, right? I, I think, um, you know, uh, unclear whether a single model or an ASI model or something like that is just gonna be able to do that out of the box. I think people are important, like they've built up, uh, you know, years of expertise doing, doing these jobs. Codifying those into, into models is actually like quite, quite difficult. But yeah, I think there will be large portions of roles at different companies that will get automated away for sure.

• 56:07Speaker 00

  Yeah, super interesting. Well, I was thinking about where to end and one thing I thought would be interesting is just like, because you're doing RL all day, a lot of things like models do some crazy reward hacking sometimes when you set these things up. I'm curious, like, what's the best example you've seen? You know, you don't need to mention the specific customer, but like,

• 56:22Speaker 01

  Yeah.

• 56:22Speaker 00

  you know, in some deployment where, uh, maybe, maybe, uh, some, some run didn't go quite as you expected.

• 56:28Speaker 01

  Yeah, yeah, yeah. No, no, I mean, like, there's even from OpenAI days, there's like all these interesting ways that these, these models are really clever. They figure out, it's like water. They, they find the, the path of least resistance. And so I think like for us, with some of the companies we work with around like coding, for example, we've been able to find some exploits in, in, in like, uh, like packages, uh, that, uh, the model just used to sort of accomplish, get easy reward on the tasks, uh, which is, you know, like, I think there, there are much worse things that are, that are public out there.

• 57:10Speaker 01

  I think that's honestly why that we're seeing, there's so much interest from cyber companies in training RL models and things like that. Because it's like almost the perfect problem to go and just apply optimization pressure to. Cause these models are really great co- coding priors. And, uh, the, yeah, the, the basically the, the offensive cyber capabilities that are also quite good.

• 57:31Speaker 00

  Well, it's been a fascinating conversation. Really appreciate you, uh, coming on the podcast and, uh, and taking the time.

• 57:36Speaker 01

  Thanks for having me. This is great. It's been awesome.

• 57:39Speaker 00

  I'm Jacob Effron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.