Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

We’ve been running a bit of an Agent Cloud series surveying all the top inference/compute/cloud providers, from Databricks https://www.latent.space/p/databricks to Daytona https://www.latent.space/p/daytona to Railway https://www.latent.space/p/railway and, even further back, E2B https://www.latent.space/p/e2b?utm_source=publication-search, but we’re excited to conclude this series returning to Modal, which has just raised a monster $355M Series C https://modal.com/blog/modal-series-c. The cloud was built for developers. But agents are now changing that. The old infra stack was designed for a human who could read docs, reason through YAML, and understand dashboards to figure out what they need when something broke. While this was painful for developers, it worked since they could fill in missing context in their heads. However, agents don’t have that luxury. Now in this new era of agents, everything has to be tighter. They need a place to write code, run it, inspect the output, change the environment, debug failures, and try again. Fast iteration and feedback loops with all the necessary context are crucial for agents to operate properly. Furthermore, sandboxes are a clear representation of this shift as agents can easily spin up isolated environments. This programmatic infra even extends to research: Two years ago, we were one of the first to cover Modal with CEO Erik Bernhardsson and Alessio designed our favorite LS thumbnail of all time: At the time, Modal was just a teeny little company with a $17M Series A https://tracxn.com/d/companies/modal/__tHK2ShUcB0Q1o6j-hbJ-xcZMxDsw0P3kCJ85veVeYjU. Today, fresh off their $355M Series C, Modal is one of the clearest examples of the agent cloud future being built in real time: a cloud platform moving past traditional web app assumptions toward the workloads AI actually creates such as elastic inference https://modal.com/products/inference, sandboxes https://modal.com/products/sandboxes, GPU burst, post-training, background agents, and infrastructure that agents themselves can operate https://modal.com/solutions/coding-agents. In this episode, Modal CTO Akshat Bubna joins swyx and Vibhu to unpack why AI applications don’t fit traditional cloud assumptions, why Kubernetes was never designed for bursty compute-heavy workloads, and why Modal is now shifting from developer experience to agent experience. We go deep on Modal’s AI infra stack: serverless functions, decorator-based infrastructure, elastic inference for custom models, GPU snapshotting, DeFlash, speculative decoding, Auto Endpoints, sandboxes, persistent storage, networked containers, private IPv6, RDMA, multi-node training, and Modal’s capacity pool across 17 cloud providers. Akshat also explains why RL rollouts can require 100,000 sandboxes, why production agents need hard guardrails, why observability may matter more than reading code, and why AI has made infrastructure exciting again. We discuss: * Why Kubernetes wasn’t built for bursty AI workloads * How Modal started as a better runtime before becoming an AI cloud * Why Modal added GPUs before ChatGPT * The shift from developer experience to agent experience * Why observability matters when agents are writing the code * Elastic inference for custom models across audio, video, robotics, and comp bio * GPU snapshotting, cold starts, and why inference workloads are so bursty * Why RL rollouts can require 100,000 sandboxes * DeFlash, speculative decoding, and frontier-level inference performance * Auto Endpoints and making optimized inference easier to deploy * What Modal adds beyond vLLM, SGLang, and raw GPU rental * Modal’s 17-cloud capacity pool and supercloud strategy * Networked sandboxes, sidecars, private IPv6, and RDMA * Serverless multi-node training for post-training and research workloads * Auto-research, model-guided sweeps, and agents launching GPU experiments * Compute strategy, capacity planning, and batch tiers * Why production agents need specialized sandboxes and hard guardrails * Modal’s take on managed agents, CI, Gitpod/Ona, Python, TypeScript, and Modal Bench Akshat Bubna * LinkedIn: https://www.linkedin.com/in/akshat-bubna-188885103 * X: https://x.com/akshat_b Modal * Website: https://modal.com Timestamps 00:00:00 Introduction 00:00:39 Modal’s origin and why Kubernetes wasn’t enough 00:04:32 Developer Experience → Agent Experience 00:06:21 Modal’s AI cloud primitives 00:09:14 Sandboxes, agent loops, and proto-Cognition 00:12:12 Elastic inference, GPU snapshotting, and 100,000 sandboxes 00:15:24 DeFlash, speculative decoding, and Auto Endpoints 00:19:59 Production-grade inference beyond raw GPUs 00:22:00 Background agents, Ramp Inspect, and the agent lifecycle 00:24:08 Modal’s 17-cloud supercloud strategy 00:26:40 Networked sandboxes, private IPv6, and RDMA 00:32:48 Multi-node training, post-training, and auto research 00:37:36 Compute strategy, capacity planning, and batch tiers 00:40:55 Open models, real-time AI, and production agent infra 00:43:06 Hard guardrails, managed agents, and specialized sandboxes 00:46:06 Why AI made infrastructure exciting again 00:48:30 Model APIs, differentiated products, and agentic video 00:51:50 CI, coding-agent infra, SDKs, and Modal Bench 00:57:28 Closing Thoughts Transcript Introduction: Modal, Series C, and the Art Party Swyx [00:00:00]: We’re here with Akshat, CTO of Modal, together with Vibhu. Congrats on your Series C. Akshat [00:00:10]: Thank you. Swyx [00:00:11]: Your party yesterday was amazing. Akshat [00:00:15]: Yeah. Swyx [00:00:15]: From all the photos and all the swag. Akshat [00:00:17]: We had a bunch of art installations, which was fun, seeing, like, our products on pedestals next to, like, Rodin. Swyx [00:00:25]: Very nice. Very nice. When you started, it was not the GPU inference company. Maybe it was in your mind. Take us back to the origin story. Modal’s Origin: A New Runtime Beyond Kubernetes Akshat [00:00:39]: I first met Eric, who’s the CEO, through an investor. Back then Eric was already thinking about building, a new runtime, and he got there thinking through why are workflow orchestration products so hard to use. It’s because you have to run them on Kubernetes. Kubernetes is hard to manage. It’s not built for burstiness and, custom images, Swyx [00:01:03]: Yeah Akshat [00:01:03]: It has a terrible developer experience. Swyx [00:01:05]: And I’ll, I’ll interject Akshat [00:01:06]: Yeah Swyx [00:01:07]: For listeners, who are new, we interviewed Eric two years ago, and there’s a bit more of the story there from Spotify and all those things. Swyx [00:01:14]: And I came across Eric through Data Council because he did that talk on the serverless container stack that you guys did, which was like, that was my first like, “Okay, I need to take Modal very seriously” moment. Akshat [00:01:26]: Yeah. Swyx [00:01:26]: But it was still very unclear, like, do I need all this for just my data pipelines? Akshat [00:01:33]: Yeah. initially what we were thinking about was if we build a better runtime, it’s a very useful primitive in itself. It’s There’s a lot of things that, get solved by serverless functions, like you can do, ETL stuff, you can do job queues, you can do all this, like, bursty processing, which it turns out every company had needs for. but then we also were thinking about this as like, this is a primitive that we can build a whole collection of products on, which are very verticalized. So perhaps data engineering would’ve been the first one, but we were thinking about inference. Back then it was more classical inference, like computer vision stuff and running XGBoosts and whatnot. But we added GPUs to the product a year before ChatGPT came out. From Serverless Containers to GPU Workloads Swyx [00:02:19]: Nice. Akshat [00:02:19]: We just didn’t think it would be that big of a deal. Swyx [00:02:22]: Yeah, just like add A100. Vibhu [00:02:23]: Was there any, like, early key problem that really sparked off why you built it? Akshat [00:02:28]: Yeah. Primarily it’s just, none of the tooling that was out there was built for, one, a really great developer experience, and also there’s a general trend of, a lot of the workloads that we were seeing were very. I wish there was a better word for it, but compute-heavy. Like, they need, one, like, need a lot more resources, so you need to burst up and down a lot, versus like Kubernetes designed for, like, slow scaling and, more for, like, web server use cases. And also there’s just a lot more specialization in, like, what kinds of environments these workloads run in. Like, we had sometimes they need accelerators, sometimes they need different kinds of images, and this is just like a consistent thing that we saw across a lot of companies. That would be the next step. Software-Defined Infrastructure and Decorator-Based DX Swyx [00:03:13]: Yeah. Yeah. Be nice. I don’t know how much this factored into the early story, but I wrote a post when I was at Temporal about infrastructure, software-defined infrastructure or something like that. Akshat [00:03:22]: Yeah, the self-provisioning Swyx [00:03:23]: Self-provisioning. Akshat [00:03:24]: Yeah. Swyx [00:03:24]: Yeah. I can’t even remember my own post. Swyx [00:03:26]: And then you put me on the landing page. Akshat [00:03:28]: Yeah. We really like, the term and so we stole it. Swyx [00:03:32]: Because you had the insight that everything can just be in decorators co-located with the code, right? Akshat [00:03:37]: Yeah. Swyx [00:03:37]: Was that a big part of the original Akshat [00:03:39]: Yes Swyx [00:03:39]: Story or it was just like a DX layer? Akshat [00:03:41]: That was, really important because we really didn’t want people to spend, so much time, writing YAML, and it seemed like you could really condense the surface area of what you’re doing, put it in code so you can operate on it just like you operate on other code, and like build stuff that’s more expressive and dynamic. and so yeah, that was always a very important part. Swyx [00:04:04]: Then the pushback is this is a DSL. Akshat [00:04:07]: Yeah. Swyx [00:04:07]: It’s you’re closed source. I am locked into Modal. Akshat [00:04:11]: Yeah. We never really got pushback for that because the nice thing about Modal is you can bring whatever code you have, and sure, the DSL is at the configuration layer for, what hardware you’re using, how you’re scaling things up, but you still own the code. Akshat [00:04:27]: And that’s, that’s been an important, part of our story, even as we do inference now. Swyx [00:04:32]: Yeah. Vibhu [00:04:32]: How much of do you think still stays the same today? Like if you were to build something today, DevX very important, but I feel like, a lot of this has been changed with just hook it up to an agent, have Claude Code, have Codex implement a tool. there’s very agent native primitives that are different than if I’m doing this myself, right? Developer Experience → Agent Experience Akshat [00:04:54]: We’ve changed our SDK team to think about agent experience instead of, developer experience and we think that the same benefits that apply for DX also apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that’s not even typed when it can make a couple of changes in a decorator and it gets this self-provisioning runtime of, being able to see its changes live in action? yeah, it just seems from the customers we talk to, they find Modal is much faster for agents to use versus operating on a different substrate. Swyx [00:05:34]: Yeah, because like you, again, you co-locate the infrastructure requirements to the code that runs it. Akshat [00:05:38]: Yeah. Swyx [00:05:38]: Well, the negative thesis now is that nobody’s looking at their code anymore, so there’s no point. Akshat [00:05:44]: Yeah, people aren’t looking at code. one thing we still see is really important is observability. Swyx [00:05:51]: Yeah. Akshat [00:05:51]: Like how good is your dashboard? And of course, like we have, we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what’s going on and, make judgment calls and whatnot. and that’s I feel like, Maybe more important now than looking at the code itself. Swyx [00:06:11]: Yes, because like, you can try to treat the code as a black box and then use, see the observable action that comes out of it, and then just prompt a change. What Modal Is For: AI Cloud Primitives Akshat [00:06:21]: Yeah. Swyx [00:06:22]: So I think it takes a bit of restraint to not specialize, to say, “I want to ship a new primitive,” and then just be general purpose. Swyx [00:06:31]: People ask you, “What are you for?” You’re like, “ I don’t know. We can do this, we can do that.” Vibhu [00:06:36]: Well, I’d be curious to see, like, okay, if we were to ask you, like, what is Modal for even at a high level? There’s a lot you guys do, sandboxes, GPUs, everything. How do you answer? Akshat [00:06:46]: Modal is a cloud platform that’s built for, where we’ve built the primitives from scratch for AI applications. and right now it covers, inference, training, batch processing, and sandbox workloads. Akshat [00:07:00]: But we’re building a lot more Swyx [00:07:02]: I noticed you didn’t say web server, so there is still a role for, like, the always-on large-scale Kubernetes type things. Akshat [00:07:09]: Yeah, absolutely. We’re, we’re not trying to compete with the renders of the world, because yeah, we think the differentiator for us is the, are the workloads that need specialized compute, need to scale up and down a lot. yeah, they’re, they’re, they’re just shaped differently. Working Alongside Frontier Startups Vibhu [00:07:26]: I think you’re building a lot of it alongside the startups, right? They’re innovating quite a bit, even in your, like, latest blog post. Like, even in the series C, the customers that you mention here, the cognitions, technical ones, ramps and whatnot, they’re, they’re innovating with you, right? And that’s not something AWS is doing directly with. Akshat [00:07:45]: Yeah, absolutely. I think, this is again classic. We’re a small team. We can move really fast. our engineers are working with our customers and figuring it out. Yeah. Swyx [00:07:54]: So my first week at Cognition, I walked in, there was someone wearing a Modal shirt. I was like, “What are you doing here?” They’re like, “Yeah, I just. I am embedded inside of Cog.” Akshat [00:08:05]: Yeah, I think that was Peyton. We sent him over Swyx [00:08:07]: Yeah. Akshat [00:08:07]: Because, the latency of communication was too high otherwise. Swyx [00:08:12]: Yeah, distributed node, you have to - you have to place one and collocate. Vibhu [00:08:16]: Yeah. Swyx [00:08:16]: So I had a, I had direct personal experience, right? So I worked on smol developer three years ago. it was inspired by Claude 1. I think you onboarded me at some point, like, just before, and I was like, “Oh, like, I need some bursty compute. Like, I was just gonna try using Modal.” And it was a, it was a pretty pleasant experience. apparently, I showed up in the board meeting, like the analytics. smol developer, Sandboxes, and Proto-Cognition Akshat [00:08:39]: Yeah, you blew up on Hacker News and, Swyx [00:08:41]: Yeah Akshat [00:08:41]: We got a big traffic spike. I. I think the way you used smol developer was Modal functions for running stuff, which was. Like, the, that was a good use case. but then, yeah. Swyx [00:08:53]: Yeah. That - So to me, that was proto-cognition. Akshat [00:08:55]: Right. Swyx [00:08:56]: If only I had, like, stuck to it. Swyx [00:08:58]: Like, that was like, if - did you say draw the tech tree Akshat [00:09:00]: Absolutely Swyx [00:09:00]: You’re just like, “Yeah, like, probably this will happen.” Akshat [00:09:02]: Yeah. Like, he was so close. You were just rebuilding upon us Swyx [00:09:04]: I just didn’t realize. Akshat [00:09:05]: But the funny story there is at the same time, we were talking to a bunch of customers who needed something like sandboxing. Swyx [00:09:14]: Yeah. Akshat [00:09:14]: This is like twenty-three. Swyx [00:09:15]: Yeah. Akshat [00:09:16]: So we built Swyx [00:09:17]: You introduced a new API right after that. Akshat [00:09:18]: Yeah. Swyx [00:09:19]: Yes. Akshat [00:09:19]: Like, we built sandboxes in May of twenty-three before anyone was even knew this was gonna be a thing. And the first example we published was, we took smol developer Swyx [00:09:28]: Smol developer Akshat [00:09:28]: And put it in a loop, so the agent can iterate on itself. Swyx [00:09:33]: Loops are hot these days. Vibhu [00:09:34]: It’s the looper. Akshat [00:09:34]: Yeah. Vibhu [00:09:35]: Loops in. When was this, twenty-three? Akshat [00:09:38]: Yeah. Vibhu [00:09:39]: A small check. Akshat [00:09:39]: Yeah. Swyx [00:09:39]: It’s like twenty-three. so the. the, those for listeners, like, the problem was the models are not built for any of this, right? Swyx [00:09:46]: Like, you’re just trying to like. They’re not post-training to understand, like, looping and, like, self-correction and tool calling was there, but, like, also not that great. Akshat [00:09:55]: Yeah. Akshat [00:09:55]: I don’t remember if you used tool calling in this one, but yeah, the models would just diverge after like ten iterations and not produce anything meaningful. Swyx [00:10:03]: Yeah. But like, then. So okay, like now talking to myself three years ago, the answer Vibhu [00:10:08]: Of course they will get better Swyx [00:10:09]: Collect all the failures, build benchmark, and then collect all the, examples, build the RL environment Akshat [00:10:15]: Right Swyx [00:10:15]: Sell it for like ten billion dollars to Meta. Swyx [00:10:17]: And then also train a model and then sell that for sixty billion dollars to Elon. And this is Akshat [00:10:23]: Yeah, of course Swyx [00:10:23]: The funny machine. Like, it’s like, it’s about the hardware. Akshat [00:10:28]: It’s hard to have that inherent conviction that the stuff will get that much better. Swyx [00:10:33]: In retrospect, it’s so f*****g obvious. Akshat [00:10:36]: Fair enough. Swyx [00:10:37]: Like, what else were we doing back then? I don’t know. anyway. Yeah. So this. That was the start of your sandboxing journey, right? I feel like it didn’t blow up until, like, last year. Akshat [00:10:49]: Yeah. Swyx [00:10:50]: So there was like a couple years of quietness. Akshat [00:10:52]: Exactly, yeah. We were Vibhu [00:10:53]: I think very underrated product value. Like, my experience with Modal, Charles, before he had joined Modal, met this guy at a hackathon, and he really insisted we wanted to run some small model, not hosted anywhere, and he’s like, “ there’s this cool company, Modal. They’ll like spin up a GPU sandbox, we can throw it on there. They’ll take a Hugging Face link.” And like there’s so much value just right there, right? Like instant hosting, spin it up, spin it down. It’ll stay cold, but we run the demo a few days later, it’ll come back up and like all this stuff in retrospect, like it’s still what we needed like today. Akshat [00:11:27]: Yeah, it’s still needed today. workload shapes have changed a lot as, we run stuff for people with really massive production scale and, there it’s it’s not about scaling from zero to one, but it’s how do we scale really elastically, from like thousand to fifteen hundred GPUs very quickly in a given region. It’s the same shape problem. Elastic Inference, GPU Autoscaling, and Custom Models Vibhu [00:11:50]: Okay. So you look at, say, Cursor Composer, right? Akshat [00:11:53]: Yeah. Vibhu [00:11:53]: They had a. “We’ll do RL on a model every couple hours.” you guys have a whole version of RL inference gym and whatnot. Vibhu [00:12:01]: When you look at workloads like that, you’re doing train runs where you need to scale up, scale down every hour thousands of GPUs, right? That’s the example for we do need it, right? Akshat [00:12:12]: Yeah. Well, so I’ll, I’ll take a step back and, maybe talk about like how people use Modal today. because our biggest use case is, elastic inference. And the thing we first found product market fit, with was inference for custom models. So we stayed away from the LLM space, and we were serving companies like Suno for audio, Runway for video, robotics, comp bio companies that train their own model elsewhere. But Modal is the best black box that for deployment, scaling to however many GPUs you need as your traffic pattern changes. And we saw all of them like have a very unpredict- predict- predictable, traffic pattern. it’s like diurnal. It’s Some days, like the company will do a launch and, they’ll need like, way more. And it’s not just one model that they deploy. They-- all these companies deploy, lots of different models in different regions, and so the autoscaling problem becomes even harder because then you have to scale within a certain region, and those cycles are offset. So different times you scale up in different regions. Akshat [00:13:20]: So that’s like our sort Vibhu [00:13:22]: And that Akshat [00:13:22]: Yeah Vibhu [00:13:22]: That in and of itself is a huge category. There’s a bunch of inference providers which, provide this fireworks, does this as a service together, whatnot, Base10. that’s carved into its own niche for language models, at least right now. Akshat [00:13:36]: Yeah. the thing that we have specialized in is the autoscaling aspect. Vibhu [00:13:41]: Yeah. Akshat [00:13:41]: Because we found that it’s not universally true that everyone else can autoscale, and we’ve gone deeper into it on the tech side by, we’ve incorporated GPU snapshotting into the product so we can take the GPU state, like your torch.compile model, snapshot it, and the next cold start is way faster. And so going back to your question, it’s That’s why you need a lot of burstiness for inference. But then people also do a lot of demand training, like for RL stuff, your rollouts are bursty, as you said. People also do a lot of batch jobs. So we’ll see, a lot of companies, before they have a training run, they’ll need thousands of GPUs to run encoding or something like that. And I think those things are much more bursty than. I agree that agents are not that bursty. sandboxes are, except when you’re doing RL. RL is just RL, Batch Jobs, and 100,000 Sandboxes Vibhu [00:14:28]: Or commerce Akshat [00:14:28]: Insanely bursty. Vibhu [00:14:29]: Yeah. Akshat [00:14:30]: Yeah. Like when you’re doing, rollouts, you sometimes need a hundred thousand sandboxes in your sandboxes. Vibhu [00:14:37]: Yeah. I’m curious if you’ve seen early sparks of continual learning. There are some people, like our friends, ngram, recently announced this Akshat [00:14:45]: Yeah Vibhu [00:14:45]: They’re, they’re trying to do training. That also seems like a different workload, right? If you’re doing training twenty-four/seven per se, there’s a very weird dynamic of how you’re using GPUs between people and whatnot, but seems like something you guys would work for. Akshat [00:15:00]: As you said, we’re, we’re fortunate to work with a number of, customers at the frontier and grab some of our customers. and they are taking the primitives we have, and trying to use them in very interesting ways, like continual learning. It’s possible as the stuff gets better, some of that will be part of, our offering as well if, more people need it. but we’re, we’re just waiting to see Vibhu [00:15:23]: Yeah Akshat [00:15:23]: How it shakes out. Vibhu [00:15:24]: Is there a primitive that you added after sandboxing that was the next step in the story? LLM Inference, DeFlash, and Speculative Decoding Akshat [00:15:32]: I guess we’ve been going much deeper into LLM inference Vibhu [00:15:35]: Yeah Akshat [00:15:35]: Because we realized that some of the advantages we have with like autoscaling, again, especially in different regions and whatnot, are, not present elsewhere. and the place where we had a gap was we weren’t, working on the model layer itself. Like we were a black box. And, we realized that, we can get to frontier-level model performance, with, by having great people who work on this. And, we’ve been open sourcing a lot of our work, in terms of, Recently, we, shared our work on DeFlash, which is a block-based, speculator, and we’ve open sourced, all of it. So, you can - By using open source DeFlash, you can get the same performance as you would with one of the proprietary providers. And the next thing we’re thinking about here Vibhu [00:16:23]: I thought this was Akshat [00:16:24]: Yeah Vibhu [00:16:24]: An interesting blog post as well, right? Like, I think in here you make a claim that. Not a claim, just that how effective speculative deco-decoding really just get to. Akshat [00:16:33]: Yeah. Vibhu [00:16:33]: Anything you wanna point out from this around, what people should know? Akshat [00:16:39]: Yeah, absolutely. the high-level summary is, it would help to describe what speculative decoding is. Vibhu [00:16:44]: Yes. Akshat [00:16:44]: I will, yes. Vibhu [00:16:45]: I think, like Akshat [00:16:46]: Yeah Vibhu [00:16:46]: So we’ve covered like Eagle and all this Akshat [00:16:47]: Yeah Vibhu [00:16:47]: Like Hydra and all those things, but it was like two years ago. Akshat [00:16:51]: Yeah. Vibhu [00:16:51]: I think it doesn’t hurt, right? Akshat [00:16:52]: Yeah. Speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model, verify all of this, all the tokens are predicted. And the reason it’s faster is if you’re predicting, one token at once, you’re bound by memory bandwidth. But if you can batch the verification of, the draft model, then you’re much more efficient using compute, and it’s faster, and as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get a speed up that’s, multiple times of, the original model speed. and well, that’s what we highlight here. It’s Like people talk a lot about we made these kernels faster and whatnot, but improving kernel will only give you like few percentage points of improvement, and, increasing accept length, literally is a multiplicative decrease Vibhu [00:17:47]: Like two to four X. Akshat [00:17:48]: Yeah, exactly. Vibhu [00:17:48]: Without much head-on performance. Akshat [00:17:50]: Yeah. I think it may - you are running a second model, right? So it may be something more expensive in the compute, Vibhu [00:17:57]: I meant quality performance Akshat [00:17:58]: Probably not by much Vibhu [00:17:58]: But yeah. I think Akshat [00:17:59]: So there’s no drop in quality performance Vibhu [00:18:01]: Yeah Akshat [00:18:01]: Because you’re always. You’re never accepting a token that the big model Vibhu [00:18:04]: It’s strictly better Akshat [00:18:05]: Yeah Vibhu [00:18:05]: Or it’s same. Akshat [00:18:06]: Exactly. Vibhu [00:18:07]: Right. Yeah. Akshat [00:18:08]: And so we’ve been working a bunch on DeFlash, which is a block-based speculator. so it’s instead of predicting, one token at a time, it’s predicting a block. And we’ve been open sourcing our work with it. The next thing for us here is for helping people train speculators and custom models. it’s it’s something that traditionally is very forward-deployed engineering driven, support deployed, engineer driven, like you work with customers and help them do that. And our vision for. This is why we launched Auto Endpoints, is we want to make frontier-level performance available to everyone. And so, we mentioned this in the announcement, we teased it. The next thing we’re, we’re launching is, as you run an auto endpoint, we shadow traffic Auto Endpoints and Frontier-Level Performance Vibhu [00:18:54]: Do you want to explain what auto endpoints are? Akshat [00:18:57]: Yeah. Vibhu [00:18:57]: I lovely, yeah. Akshat [00:18:58]: Yeah. So, this is, I guess, going back to your Modal is you touch the code, but, sometimes people don’t wanna touch the code, and they wanna get started with an endpoint that works and has all the great performance and, scalability that Modal has. So we’ve made that easier with, a way to create an endpoint from our UI, from the CLI, that has all of our optimizations that we talked about, like the DeFlash stuff already baked in, and there’s full transparency. So we give you the code, you can go run it yourself, and if you want, you can eject out into the full Modal experience, which we see as people get sophisticated, they do wanna tweak the models, they wanna, fine-tune stuff. You can still do all of that. It’s it’s not a black box. And yeah, the next thing, as we teased later in the post, is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again, without having to talk to a person and, yeah. Vibhu [00:19:59]: I guess just to understand it directly, you have the GPUs, you have an endpoint that’s compatible, you serve open model. If someone was to do this themselves, what’s the delta that you guys provide? So you do a lot of open source great work on effective inference. how does it compare to, say, I take the same model, 5.2 FP8, take shelf inference engine, vLLM, SGLang, get compute of similar capacity, similar cost. What’s the delta that plugging into something this, like this offers outside of the benefit of, scaling? Production Inference Beyond Raw GPUs Akshat [00:20:34]: It’s interesting because we’ve taken the approach of open sourcing our contributions and upstreaming them. we work closely with the SGLang team. We want the improvements that our team, comes up with to be, there in open source for others to use, even outside of Modal. The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance, first. the other thing is with these endpoints, we are way more elastic, as you said, than, anyone else, and you have true scaling to zero. you have true, burstiness, and in practice, that matters a lot more to people than just finding, the GPU and, running Modal code on something. Vibhu [00:21:20]: Yeah. And I will say it’s not that straightforward to just. like what I said is easier said than done, right? Akshat [00:21:26]: Yeah. Vibhu [00:21:27]: It’s I think still for the average person, still hard to just gut check using different. There’s, there’s quite a bit of combinations you can make there. the trade-offs aren’t really known at face value. Akshat [00:21:40]: Yeah. it’s it’s not just that. I think it’s it’s that running production-grade inference is a hard infer problem. Vibhu [00:21:49]: Yeah Akshat [00:21:49]: Even if you subtract out the autoscaling Vibhu [00:21:50]: Yeah Akshat [00:21:51]: Is controlling things like tail latency and, making sure every, request is delivered at least once and whatnot. The Model and Agent Lifecycle Vibhu [00:22:00]: There’s a lot of innovation that you can do here. I think, it’s very interesting that you’re starting to encroach on, like as you become a full cloud, you’re starting to encroach on other people’s turf. Vibhu [00:22:09]: What will you not do? Akshat [00:22:13]: Well, we wanna follow our users and, make sure they get like a platform that has everything that works well together. so right now we’re focused on the model lifecycle and the agent, lifecycle. so both like going from data prep to training to inference, and then also if I want to deploy a background agent, let’s say, sandbox, do persistent storage, a whole bunch of other stuff. Vibhu [00:22:38]: We talked to Cole, who did, OpenInspect. Yeah. Akshat [00:22:42]: Yeah. Vibhu [00:22:42]: And RealInspect also is on Modal. Akshat [00:22:44]: Yeah. So Ramp Inspect was a great example of a background agent that was really successful because they, were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well. Ramp Inspect and Background Agents Vibhu [00:23:02]: Yeah. That’s the new CTO of, Ramp right there. Akshat [00:23:05]: Yeah, Rahul. Vibhu [00:23:08]: It was really fun. yeah, okay, I think, all very bullish. Like, one of my reflections was also I did not originally. So when I met you guys The Inference Inflection: CPU, GPU, and Co-Location Vibhu [00:23:19]: You weren’t that much in the GPU game, and now you’re all about, inference. And one of the points that I hinged on for Jensen’s keynote at GTC this year was, what we’re calling like the inference inflection, right? That let’s say in AI workloads or machine learning workloads, it used to be like, let’s call it eight to one GPU to CPU, and now it’s more like one to one, which is like a interesting. Like, - because of how much agents are blocked or call out to this, to CPU heavy stuff the actual, like, limiting factor, like, swings back and forth from GPU to CPU a lot more than it used to be all GPU and then occasional CPU. Akshat [00:24:01]: Yeah. Vibhu [00:24:02]: GPU, CPU. And now it’s like just constantly, and you just have to locate everything. Seventeen Clouds and the Supercloud Strategy Akshat [00:24:08]: Yeah. And that’s one of the things that, again, we see as, something appealing about Modal, which is we’ve built this capacity pool that spans, 17 cloud providers, so we’re, we’re very good at Running on various kinds of cloud capacity across the world Swyx [00:24:24]: You don’t have your own data centers? Akshat [00:24:25]: We don’t have our own data centers. We just run across a lot of neo clouds Swyx [00:24:29]: Yeah. Are Akshat [00:24:30]: Metal providers. Swyx [00:24:30]: Yeah. Question mark. Swyx [00:24:31]: Yeah. You’re, you’re running the math, and you’re like, “What’s the cutover point where you’re like.” Akshat [00:24:36]: Yeah, it’s a good question. part of it is we see our differentiator in the software layer, and, being capital light and focusing on the software helps us move really fast. so far it’s worked out well because there are so many other people building data centers that we’re able to work effectively with them, and again, focus on what makes us, special. Swyx [00:24:55]: Yeah. Swyx [00:24:56]: 17 gets you into, like, the local providers sometimes. Like Akshat [00:25:00]: The, Swyx [00:25:01]: Which was the most interesting one? Akshat [00:25:02]: There are a lot more neo clouds than you expect, and they all have various degrees of, various levels of reliability. And, that’s why it’s something we’ve invested a lot of time in, is building our own reliability layer on top. so if the GPU falls off the bus or something happens, we user workloads are not affected, and that lets us use a lot more capacity than, Swyx [00:25:30]: Yeah Akshat [00:25:30]: You as a user would be able to. Swyx [00:25:32]: It’s a useful thing to have because like now everyone knows, like, what layer you are and, like, you optimize for being the super cloud of all clouds. Akshat [00:25:41]: Yeah. That’s, that’s, that’s the idea. and so I guess when you mentioned colocation, that’s, that’s another interesting thing where, one thing we’ve seen is people come to us when they want, very specifically located, CPUs or GPUs, like they want Swyx [00:25:57]: Oh, they pin it in like Akshat [00:25:58]: Yeah Swyx [00:25:58]: EU? Akshat [00:25:59]: Exactly. Or EU, US. Swyx [00:26:01]: Right. Data resiliency Akshat [00:26:02]: Australia Swyx [00:26:02]: Locality thing or performance or what? Akshat [00:26:04]: It’s either data locality or latency, yeah. Swyx [00:26:07]: Yeah. Akshat [00:26:07]: Like, you want your. They’re running sandboxes and model. They want them to be right next to a Swyx [00:26:10]: Yeah, it’s easy then Akshat [00:26:11]: Yeah Swyx [00:26:12]: To. That is important in all those things. and so, like, you’ve accidentally, I don’t know if it’s accident, but, like, you’ve built the perfect primitive for agents to express themselves. And then, like, it’s almost very funny how every extra development just involves more file system, just involves more CPU. Akshat [00:26:30]: Yeah. Swyx [00:26:31]: Just like the things that you already have. I don’t know much about, if there’s any, like, networking usages that are interesting, but you’ve also done some good work on networking. Networking, Sidecars, Private IPv6, and Sandboxes Akshat [00:26:40]: Yeah, that’s exactly right. Like, we’re just taking compute storage and networking and building stuff on that layer, for, again, the stuff people need. Swyx [00:26:49]: Yeah Akshat [00:26:50]: We see a few interesting networking things coming up. one is people want networked sandboxes. so we have Swyx [00:26:57]: For like a Docker cluster type thing. Akshat [00:26:59]: Yeah. Swyx [00:26:59]: Sorry, Docker Swarm. Oh, f**k. What is it called? Akshat [00:27:02]: Compose. Swyx [00:27:03]: Compose type thing. Akshat [00:27:04]: Yeah. So if you want Docker Compose, our sandboxes now support, this thing called sidecars. So you can. A sandbox is a pod of containers, and you can run multiple containers in, a sandbox. also useful because, going back to networking, people want a lot of control over, outbound networking from a sandbox. Swyx [00:27:23]: Yeah. Akshat [00:27:23]: Like, they might wanna run a middle proxy for, like, maybe logging stuff for RL or, controlling how egress can happen to a domain, injecting credentials. and yeah. So we’ve, we’ve had to build a lot of that stuff ourselves. Swyx [00:27:38]: Yeah. Akshat [00:27:39]: But then also sometimes people want, sandboxes spanning multiple nodes to talk to each other, which is an emerging thing we’re seeing. We have support for that for a different reason, and yeah, we’ll see if that becomes stable. Swyx [00:27:52]: Like, just an open socket. It’s a. This is directly like mTLS. Akshat [00:27:56]: We do support that, which is you can, expose a tunnel inside a sandbox. Swyx [00:28:01]: Yeah. Akshat [00:28:01]: And then you can either expose it to public internet or it can be, you can add like a HTTP, auth layer above it. But we have this thing called I6PN, which we haven’t talked about, which is this, like, overlay network using IPv6 addresses. so if Modal containers, within the same workspace, when this is enabled, can address each other using this private IPv6 address, and no one else can. Akshat [00:28:28]: So it’s like private networking, for containers. We built it because we needed it as a primitive for our distributed training product. so we have this other feature, which is you can add a decorator to a function, and you get a cluster of GPUs. and they have RDMA networking. so you can run a distributed training job, that’s truly serverless. and we did the overlay network for that. But then we’ve seen that people are using it for other reasons, and, I’m intrigued to yeah, what would people do with it. Swyx [00:28:59]: Build primitives and let people figure it out, right? Akshat [00:29:01]: Yeah, exactly. Swyx [00:29:02]: You put out a pretty interesting Akshat [00:29:03]: They’re like, they read the docs webpage. Let me use that Swyx [00:29:06]: Yeah Akshat [00:29:06]: Something they never intended to work. This is literally not even in our docs page. People somehow found it, and they’re using it. RDMA, Memory Movement, and Distributed Training Swyx [00:29:12]: Huh. Swyx [00:29:14]: The way you portrayed it with, like, RDMA versus TCP, like, very well laid out, but just the transfer speed change at scale for RL, like yeah, you have it, you have it built in. I’m sure someone found it. It’s found it to be a lot more efficient before you made a thing out of it, right? Akshat [00:29:32]: Yeah. And not to split hairs, I guess the overlay network is the TCP overlay network. Akshat [00:29:39]: The reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that. but then people found the TCP part. Swyx [00:29:48]: Can I tell you, this is like a big aha moment for me because Akshat [00:29:51]: Yeah Swyx [00:29:51]: So I review 2,200 submissions for the World’s Fair. Akshat [00:29:56]: Yeah. Swyx [00:29:57]: And then I got this from John Osterhout Akshat [00:29:58]: Huh Swyx [00:29:59]: Who I don’t know if. Do John Osterhout by name? Akshat [00:30:01]: The name sounds familiar. Swyx [00:30:02]: He published a. He’s a well-known professor, published a lot of interesting software design books, and this is the talk he chose to submit, is on RDMA at Inference. And I’m like, you wouldn’t think that this guy, who is like operating systems guy, would care about RDMA. Akshat [00:30:20]: I, it makes sense to me because I, Swyx [00:30:24]: This is the cloud, right? Yeah Akshat [00:30:25]: Like, the way you move around your KV cache and how efficiently you can do it, how efficiently you move, your weights from your training GPUs to your inference GPUs in RL is there’s a lot of degrees of freedom, and it is a systems problem Swyx [00:30:41]: Yeah Akshat [00:30:41]: Moving memory around Swyx [00:30:42]: Yeah Akshat [00:30:43]: Scheduling. Swyx [00:30:44]: This shows you how primitive my understanding of networking stuff is. Swyx [00:30:46]: Is this like the domain of WireGuard as well? Akshat [00:30:50]: Not quite. Swyx [00:30:51]: It’s adjacent? Swyx [00:30:53]: Explain everything. Akshat [00:30:54]: Sure. Swyx [00:30:56]: How do we move memory around GPUs? Akshat [00:30:58]: Well, so sorry. Yeah, that is memory. Sorry, I was talking more, and maybe I was talking like five minutes back, about the private IPv6, addressing that you’ve set up. Swyx [00:31:09]: Yeah. Akshat [00:31:09]: Is it like it’s a VPN? Swyx [00:31:10]: Yeah, it is like a VPN, and yeah, WireGuard is, yeah, you’re right. It is, Akshat [00:31:16]: Right. Yeah, you already moved on to new topics Swyx [00:31:17]: A similar Akshat [00:31:18]: Okay Swyx [00:31:19]: In the same space, WireGuard is, encrypted and this is, Akshat [00:31:23]: And you don’t need encryption. Swyx [00:31:23]: Yeah. Akshat [00:31:24]: Yeah. Swyx [00:31:24]: This is not encrypted. that’s the main difference. This is TCP and we have eBPF programs that will reject or allow the TCP connection based on whether you’re allowed to do it. Akshat [00:31:35]: Used to involve a full sidecar, but now you have eBPF in the Linux kernel. Swyx [00:31:39]: Yeah. Akshat [00:31:40]: Yeah. I don’t know if this is a natural follow-on to the topic of like my skepticism on distributed training is that while, like, people spend a lot of money on, like, cables to hook up GPUs, and even that is not, like, fast enough, and that’s the bottleneck, is your networking fast enough? Swyx [00:31:59]: Yeah. So I guess you’re talking about fully distributed training like, Dialog or something which is like cross data center Akshat [00:32:06]: That would be, yes. Swyx [00:32:07]: That’s the extreme. Akshat [00:32:08]: Yeah. Swyx [00:32:08]: You’re in the middle, and then other people would have like the Mellanox cables up in, like, their actual data center. Akshat [00:32:14]: When you run multi-node training on Modal, RDMA, I think Mellanox, is, or InfiniBand is like a, is all seen as RDMA. but it’s a way to bypass the TCP networking stack and, transfer, stuff much faster, between one node, to the other. And we have I think like 3 terabit per second, internal networking Swyx [00:32:40]: Okay Akshat [00:32:40]: Which is the standard that’s needed. Swyx [00:32:42]: Okay. So I misunderstood what Akshat [00:32:43]: 50 Swyx [00:32:43]: What part of the stack you were Akshat [00:32:44]: 50 gigs over Swyx [00:32:45]: Yeah Akshat [00:32:45]: If you went Swyx [00:32:45]: Yeah Akshat [00:32:46]: RDMA. Swyx [00:32:46]: Okay. Swyx [00:32:48]: Yeah. I, very impressive work. Multi-Node Training, Post-Training, and Auto Research Swyx [00:32:52]: So effectively you’re extending like the model philosophy to the training cluster, like, yeah. Akshat [00:32:59]: Yeah. And we’re, we’re not going for like large scale training runs. the thing that we’ve built multi-node training for is, we see a lot of, smaller scale post-training. like, people are post-training like medium sized fund models, so they can, get higher quality on inference. this is a perfect fit, for something like that. Swyx [00:33:21]: Yeah. That is my impression of how a lot of these labs explore branches in post-training and then eventually merge whatever they find in. Akshat [00:33:31]: Yeah. The other use case we’ve seen for multi-node training is even if you have a big cluster, your researchers are still doing small runs Swyx [00:33:38]: Yes Akshat [00:33:39]: Having elasticity there Swyx [00:33:40]: Right, sure Akshat [00:33:40]: Matters a lot more. Swyx [00:33:41]: Yeah. the, like, this is like the current limiting factor for auto research, which is like you need to give your model some GPUs in order for it to completely run. Akshat [00:33:51]: We have a blog post on auto resource and model is, Swyx [00:33:55]: Yeah Akshat [00:33:56]: Yeah, like, turns out to be pretty good substrate for that. Swyx [00:33:59]: So my impression is auto research means many things, like Akshat [00:34:01]: Yeah Swyx [00:34:01]: Anything that Andrej coins. Right now it’s still science fair, right? Like not like, I don’t know how many people are doing this. Akshat [00:34:08]: We’re having a golf. Swyx [00:34:08]: Yeah. Akshat [00:34:09]: I thought the same thing. Swyx [00:34:11]: Yeah, you would know. Akshat [00:34:12]: We, like, our internal both training and inference teams use this the general shape of this quite a bit. like we have this one internal repo called auto inference, which essentially we’ve automated our own forward-deployed engineering efforts using, this harness, which is, the agent will just spin up a sweep of different things. It’ll even run like, NVIDIA inside profiler and it’ll like tweak configs and it’ll arrive the right thing. it’ll change your GPUs both from H200 to B200, and works really well. Swyx [00:34:47]: Nice. Akshat [00:34:47]: So yeah. Swyx [00:34:48]: By the way, I enjoy that your forward-deployed engineering is so technical that you have to do these things. Swyx [00:34:52]: It’s very different from forward-deployed engineering from other people. Akshat [00:34:54]: Yeah. For our forward-deployed engineering team is, essentially they’re like applied inference researchers or applied training researchers. Swyx [00:35:02]: Someone told me like they have to be able to build, but they also have to be able to sell. do they have to sell or are they like they’re good, they’re just like post-sale type of thing? Akshat [00:35:09]: It does, being able to talk to a customer and engage effectively with them Swyx [00:35:13]: Yeah Akshat [00:35:13]: Matters a lot. Swyx [00:35:14]: They want the same thing. Akshat [00:35:15]: Yeah. Swyx [00:35:15]: ? Akshat [00:35:15]: But it’s it’s not really a sales, thing. We pair them with-- We have solution architects as well that are more on the sales side. Swyx [00:35:23]: Okay. Let’s spend a bit more time on auto research. This is a big focus for for this year. Where does this go? like, have people explored enough? Like, there’s all these beautiful charts of like improve and then level off a bit and then you find the next thing. Is this one abstraction up from normal training? Is that how we think about it, or do you think about it differently? Like model level training versus high, like driven hyperparameter search. Auto Inference and Modal Bench Akshat [00:35:51]: Yeah, like, Swyx [00:35:51]: Someone, some people call it like neural architecture search or whatever, right? Like. Akshat [00:35:54]: Yeah, - So the stuff I’ve seen people do with it is nowhere on the architecture level. It’s pretty much tweaking parameters, but it’s it’s a hyperparameter sweep that’s guided by some model intuition, so it’s like much more efficient than, whatever other, sweep you would have. Swyx [00:36:12]: Yeah, it’s just, it’s just a question of where you want to spend your compute? Akshat [00:36:16]: Right. Swyx [00:36:16]: ‘Cause yeah, you can just throw infinite amounts of money on this and somehow you’ll bang out Shakespeare? Akshat [00:36:22]: Yeah, infinite monkey. Swyx [00:36:24]: Yeah, so like the very good for model. and I think it’s also very important that agents can spin up other agents, can spin up their infrastructure. Like very good for you. how good is our LLMs at generating model code? Like the benefit of existing LLMs is that you are in the data. Akshat [00:36:42]: Yeah. They’re, they’re surprisingly good. I think like pre Cloud 4 they were not, and then now they’re able to shot, stuff out of the box. But we’re playing around with releasing like a Modal Bench for like the harder Swyx [00:36:55]: Yeah Akshat [00:36:55]: Things, that the LLMs cannot do yet and maybe Swyx [00:36:59]: What’s an example of that? Akshat [00:37:01]: I think the things that- Sometimes agents struggle with, without right guidance and a skill is, how to, use the rest of our observability. Like how to. Something is failing, like how do you look at the logs and then update the right thing? It’s reasoning about that. But they’re able to shot, like Swyx [00:37:23]: Yeah. You can just add a skill to it? Compute Strategy and Capacity Planning Akshat [00:37:26]: Yeah. So we have a Modal skill now that. Which is why we built this Modal Bench. It’s to find things like that, so we can address them in our tool. Swyx [00:37:35]: Tune a skill. Yeah. Akshat [00:37:36]: Yeah. Swyx [00:37:36]: No. it’s it’s good. are you facing any shortages? like we talk a lot about GPU shortages, but also CPU, also memory. Swyx [00:37:44]: Yeah. Akshat [00:37:45]: We have had a lot of growth, which means that, there’s - we’ve had to be much better about Swyx [00:37:53]: Planning Akshat [00:37:54]: Proactive capacity planning. Swyx [00:37:55]: Yeah. Akshat [00:37:55]: So we have, Swyx [00:37:57]: Which by the way, like it’s like a MBA’s like dream Akshat [00:38:00]: Yes Swyx [00:38:00]: Is like just planning this stuff. I think last time you and I talked about something maybe about this. Akshat [00:38:03]: Yeah. we have a really competent team of people that we call, The role is called compute strategy. so yeah, if anyone listening here or wants to work on that Swyx [00:38:13]: Compute strategy? Akshat [00:38:13]: Yeah. Swyx [00:38:14]: I think, Akshat [00:38:14]: I feel like, Swyx [00:38:15]: I think the normies call it FP&A or something. Akshat [00:38:18]: Well, it’s more It’s it’s not FP&A. It’s it’s There’s a lot of interesting financial questions of like what is the blend between one year and three-year reservations? how do we forecast our own capacity? how do we. especially since our capacity is very fungible across different GPU types and different regions, like you have to model a lot of it. and you also have to have an opinion on how the supply chain is gonna evolve, and then you have to like, take bets, Swyx [00:38:49]: Yeah Akshat [00:38:49]: Based on that. Swyx [00:38:50]: Tokenomics. Akshat [00:38:50]: Yeah. Swyx [00:38:51]: This is like probably a not a real point, but, I was trying to think about like what other industries. I was trying to think about like, we cannot be first to like these kinds of problems. Akshat [00:38:59]: Yeah. Swyx [00:39:00]: And what other industries have had this? And I was like, airlines with fuel and like they have to hedge their fuel and like, I think for a long time Southwest because they made like a hero fuel bet, they like were like super low cost because Akshat [00:39:12]: Oh Swyx [00:39:12]: Compared to everyone else. Akshat [00:39:14]: Yeah. I hadn’t thought about that. Vibhu [00:39:16]: We’re at a fun time too? Akshat [00:39:18]: Yeah. It’s. A lot of the compute business in general, for us is also about being very good about capacity management. That is how you have great unit, economics. but also over time it’s how you can unlock more value for customers. Like, one of the things we’re building now is like a way for customers to get, If they don’t care about latency, like get much cheaper pricing and they’ll get results back in like next 24 hours or something, like a batch tier essentially. Batch Tiers and Latency-Insensitive Workloads Swyx [00:39:47]: Yeah. Akshat [00:39:47]: And those are levers we have because we control the whole stack and scheduling and whatnot to give people a sufficient Swyx [00:39:53]: Yeah. I feel like they’re not as popular. Like those, like the Frontier Labs have all those APIs. They’re not as popular as they should be. Akshat [00:40:00]: The demand that we see for something like that is not for LLMs. although sometimes people wanna run evals and Swyx [00:40:08]: Okay Akshat [00:40:08]: Synthetic data prep and there it makes sense. Swyx [00:40:10]: Okay. Akshat [00:40:11]: But it’s from a lot of LLM companies, like people who are doing computational bio, like they have to run really big batch jobs and they don’t care about when they get it back. Swyx [00:40:22]: Yeah. And like they have a reasonable. It’s it’s also like a cousin to the stopping problem of like, will this finish in time? Akshat [00:40:30]: Yeah. You can bound it. Swyx [00:40:33]: Yeah. Akshat [00:40:33]: Like you can give people Swyx [00:40:34]: Yeah Akshat [00:40:34]: SLAs on it. Swyx [00:40:35]: Yeah. I think what’s, what’s interesting is like the next phase of model. Swyx [00:40:38]: Like what, do people expect from you, now that you’re established and you’re like well-known compute player among all these leading companies. You had an inference launch week, and we talked a little bit about the launches. like what else? Like what else should people know? What Modal Builds Next Akshat [00:40:55]: We are building primitives that make our users’ lives much easier. So, I think for example, with LLM inference, thousands more companies are gonna post-train their own models and, deploy open source models for inference. so we’re thinking a lot about what is the best product shape for that. And, that involves everything from our training gym to, then, endpoints that get frontier-level performance. again, but I haven’t talked to anyone. It looks somewhat different on other verticals. Like, we’re also seeing a lot of real-time, audio-video stuff in there, which is why like, we’re working on things like regional routing, with fallbacks. So you can get GPUs that are as close to users as possible. so you get like low latency for video streaming and whatnot. And then on the agent side, it’s, Akshat [00:41:52]: We’re still working very closely with our customers because stuff is changing so fast in terms of what they need. And, I think beyond sandboxes and persistent file systems, there’s a lot of other things people will need from this agent stack as they build production agents. So yeah, we’re thinking about those other things that fit in there. Swyx [00:42:13]: I want to ask what the other things are. Akshat [00:42:15]: Yeah. I probably should share right now. Swyx [00:42:17]: I think-- I think, okay, so, I do think a lot about the principal components of cloud, and you do talk about compute storage networking. Akshat [00:42:25]: Yeah. Swyx [00:42:25]: Because so far for me, it’s fine. so far for the. the first couple generations of cloud, it’s fine. What’s different, qualitatively different about agents that you need some new permission level? Like a lot of people, okay, and I’ll just kinda spew tokens at you until it like hopefully sparks something. Akshat [00:42:43]: Yeah. Swyx [00:42:44]: Like the new level now is whatever Claude Code does, which is dangerously scope permissions or like allow list by command or like whatever, right? And sometimes they’re like, “Well, okay, we have like this adaptive thinking mode where like, just trust me, bro. I will make the calls for you.” Is that it? like mediated permissions. Hard Guardrails vs. LLM-Mediated Permissions Vibhu [00:43:03]: Now you’re looping it with a goal and letting it roll. Akshat [00:43:06]: Yeah, I’m, I’m skeptical of LLM media permission for stuff that is at the sandbox level because you do want hard boundaries. Swyx [00:43:16]: Yeah. Akshat [00:43:16]: Otherwise, someone can exfiltrate stuff. Swyx [00:43:20]: But like Akshat [00:43:20]: Yeah Swyx [00:43:20]: Maybe that’s old school thinking. Maybe we’re the dinosaurs. Swyx [00:43:23]: Maybe the AI OS or the LLM OS is really the kernel is a goddamn LLM. Swyx [00:43:30]: Like it makes you feel uncomfortable. Akshat [00:43:31]: Yeah, I’m, I’m told Swyx [00:43:32]: But that’s what trusting the LLM is. Like imagine a spherical cow perfect LLM. Akshat [00:43:36]: Right. Swyx [00:43:37]: That it. Akshat [00:43:39]: Maybe. Swyx [00:43:41]: I wanna test the boundaries, right? Akshat [00:43:42]: Yeah. Swyx [00:43:42]: Like, and I don’t believe that, but I wanna see where I’m wrong ‘cause that’s, that’s the consensus. Akshat [00:43:49]: Yeah. I think you always need hard guardrails when you want, And you can pair those with softer guardrails, right? And that’s gonna be a lot of mediated. Managed Agents and Specialized Sandboxes Swyx [00:44:00]: There. I’ll also get you a end with a couple of your commentary on like the ecosystem outside of Modal. Manage agents. Everyone has one. Gemini, OpenAI, Claude, very useful for you, but also like it is their way of starting to edge into your space. Akshat [00:44:17]: Yeah. Swyx [00:44:17]: What’s going on? Akshat [00:44:19]: Yeah, we’re, very excited to partner with Anthropic and some of the other foundation labs, will not name who we’re also working with. the way we see it is the manage agent thing is a great place to start if you’re starting out building an agent and, But then when you get to, building something more production grade, like you’re a company that’s like Ramp that’s building their own, Ramp also runs their accounting agent on us, so their external-facing agent. You need a lot more control over, your compute primitive on things like, what sort - how do you persist different files that the agent has access to, and how do you snapshot and restore? How do you control the networking? maybe you want GPUs. When you get to that point, you kinda want, a specialized sandbox provider, that gives you those things, and that’s the role that we are trying to play. Swyx [00:45:15]: Yeah Akshat [00:45:16]: We don’t really have an opinion on the harness, whether it runs - it’s a cloud-managed agent, and you hook it up to Model Sandbox, or you run the harness in Model Sandbox. We’ll see where people converge with that. Swyx [00:45:26]: Yeah. Do you any opinions on like the meta harnesses, or just another layer on top of these things? Akshat [00:45:31]: You mean like the OpenPipe Swyx [00:45:33]: OpenPipe is one. I think Vercel had one, which I can’t remember the name of right now. Fredshot had one. and then, to me, most recently was Data Databricks that had Omnigen. All these are meta harness. Like it’s kinda pseudo agent cloud type things. Akshat [00:45:50]: I personally have not played around with them. Swyx [00:45:53]: Yeah. Akshat [00:45:53]: Build agents with them. Swyx [00:45:54]: Everything’s bullish Modal, as long as it consumes more infra. Akshat [00:45:57]: That’s why we’re focusing on the infra layer. It’s somewhere where our, relative competence is and, also it’s a hard problem to solve. Swyx [00:46:06]: Yeah. I will say like just generally reflecting on that, I don’t know if - if there’s other topics on Modal, but like just generally reflecting as an infra person, not as intense as you, but in that field, this has like been the most exciting time in infra. Like it was boring for a while, and you couldn’t really get people excited about data infrastructure. Like Eric would get on Data Console, everyone just watched the video and like say, “Look at how many sandboxes I can spin up,” and no one gave a crap. Why Infrastructure Became Exciting Again Akshat [00:46:39]: Yeah. Swyx [00:46:40]: And like now everyone gives a crap. Akshat [00:46:42]: That’s true. It is a very exciting time, and I think a lot of that’s driven by just the amount of scale all of this stuff needs. Swyx [00:46:50]: I think the, like a lot of your initiatives or a lot of your like product directions make sense in retrospect, which is like the best kind, but I wouldn’t necessarily have thought about it myself, which. Akshat [00:47:00]: We need the predictions. Swyx [00:47:02]: I think there’s a lot that you just don’t even see, right? Like you have the batch, you have the voice, you have the multimodal, but what else? Akshat [00:47:10]: What else is coming up for us Swyx [00:47:11]: Yeah. Where do you see things going? Akshat [00:47:13]: Yeah. I, in general Biotech, Robotics, and Non-LLM AI Workloads Akshat [00:47:15]: It’s it’s clear that there’s there’s a huge shift happening. I think one thing that’s not as obvious to people because LLM inference gets talked about so much and is also we work a lot of companies that are, doing things like drug discovery and computational bio, like the Chai Discoveries of the world. Big things are probably gonna happen there. we work a lot of robotics companies that are putting robots in like active deployments and getting good results out of them. Swyx [00:47:45]: Is there Air Gap Modal? Is there a version that is like prem air gapped whatever? Akshat [00:47:50]: No. We, Swyx [00:47:51]: You should cloud only. Akshat [00:47:51]: Yeah. Swyx [00:47:52]: Yeah. Okay. But yeah, so what you’re saying is like because you’re focused on primitives and they’re good primitives, you find use cases in all these kinds of things. Akshat [00:48:01]: Yeah. Swyx [00:48:01]: Probably diversifies you a little bit away from LMS all the time. Akshat [00:48:05]: Yeah, absolutely. We’re, we’- our goal isn’t to only serve the LLM inference market. Swyx [00:48:10]: There are a lot just on the website, the audio, Akshat [00:48:12]: Yeah. We said both on Swyx [00:48:14]: Computational bio images. Yeah, there’s a lot here. There’s QTA TTS, customizing. Oh, Chatterbox. there was customizing Whisper. Akshat [00:48:24]: Okay. Yeah. Swyx [00:48:25]: This screen reminds me of a fallen competitor, which Replicate. Model APIs vs. Differentiated AI Products Swyx [00:48:31]: What’s your postmortem on what happened? Akshat [00:48:34]: This is one thing we’ve stayed away from is providing an API for models because I think providing model APIs is some of it ends up serving like a really hobbyist market, which is much less sticky. Swyx [00:48:50]: Yeah. Akshat [00:48:50]: And we’ve always wanted to build for companies that are building products and need more flexibility that’s not just an API. Swyx [00:48:57]: Which you can build an API for a model and this is clearly what it is. But you - but what you’re saying, you can wrap it into a more fully functioning back end that you run. Akshat [00:49:06]: Yeah. So all of our examples, it’s not that spin up this model, here’s an API token, use it. They’re all code. Swyx [00:49:13]: Okay. Akshat [00:49:13]: And so the point is that this is just an example. Swyx [00:49:16]: Starter code. Akshat [00:49:17]: Yeah. But you can tweak it however you want. Swyx [00:49:20]: Yeah. Akshat [00:49:21]: And if you’re like a company building a product, like, computational bio whatnot, yeah. Swyx [00:49:26]: I guess I’m trying to tease out for listeners Akshat [00:49:28]: Yeah Swyx [00:49:28]: When does it stop becoming, oh, you’re just an API call and you’re just a wrapper on API to becoming what you call a product, right? Swyx [00:49:36]: Like, what is that layer? Like what-- Like, more lines of code, but like beyond that, what is the substance that people add that qualifies it to be something more? Akshat [00:49:46]: I think there’s a little bit of like a selection effect of like a lot of the companies who do wanna get deeper into that level are probably building something that’s more differentiated. And, I think, an example is like - with LLM inference, originally we, worked with companies that were building their own post-training frameworks or they were, - Ramp early in the day was training their own tokenizer and like swapping out the tokenizer in Llama and whatnot. I’m not saying that’s, that successful, in that case. But a better example is like, let’s say Suno. because Suno, does not use Modal for training. Swyx [00:50:26]: Mikey on the pod. Yeah. Akshat [00:50:27]: But they use Modal for all their inference and that’s because they have like a custom-- They have completely custom model architecture and that means that they have to be at the code level and tweak things that are not, just an API. Swyx [00:50:41]: It’s interesting as well, like we had, Ethan, most recently on the xAI Groq team make a prediction that like the next tier in video gen is not a better video model, it’s a better model or agent that orchestrates video models. Video Agents and Production Workflows Akshat [00:50:56]: Oh, interesting. Vibhu [00:50:56]: Language model backbone that can use tools Akshat [00:50:58]: Right Vibhu [00:50:59]: And write code. Akshat [00:51:00]: Like, yes, I can make my second video or my second video from Groq, but I want my minute video. Akshat [00:51:06]: And I’m not going there through normal video gen. Swyx [00:51:10]: Yeah, that’s interesting. I - So we have GPU sandboxes and recently have seen a few companies doing agents that do video manipulation or, Akshat [00:51:22]: Yeah. Give it FFmpeg and just do it. Swyx [00:51:23]: Run FFmpeg. But like Akshat [00:51:25]: That’s not enough. Swyx [00:51:25]: Yeah. Akshat [00:51:26]: You need to give it Adobe. Swyx [00:51:27]: Yeah, I hadn’t put it together with like it would be a video production thing. in my mind these things were going more towards editing Akshat [00:51:36]: Yeah. Vibhu [00:51:36]: Well, shout out Mantis. Akshat [00:51:37]: I think about this a lot. Swyx [00:51:38]: . Akshat [00:51:41]: Yeah. Sorry. Vibhu [00:51:41]: Luma. Luma Agent is a version of this for video production, but it’s a off. Swyx [00:51:46]: I was gonna get your quick takes, on some other stuff that happens Gitpod/Ona, CI, and Runtime Sandboxes Swyx [00:51:50]: In recent news and just-just see if you have anything interesting. Gitpod, very like-- somewhat like, different market. They’re in like the CI/CD market, but technically very impressive. I don’t know if you’ve like taken a real look at them. Akshat [00:52:03]: Yeah. we’ve, - People on our team have talked to the Gitpod team and they’- they’re technically very strong. Swyx [00:52:10]: Yeah. Akshat [00:52:10]: I - We’re, we’re very bullish at Modal on the CI market as well because Swyx [00:52:15]: Okay Akshat [00:52:15]: There’s, there’s more agents, coding agents. Swyx [00:52:18]: Yeah. Akshat [00:52:19]: They’re gonna run a lot more CI and the primitives there can be much better. Swyx [00:52:23]: I think there’s a lot of wasted CI. Akshat [00:52:25]: Yeah. Swyx [00:52:25]: So is it just like let’s filter? Like what is the highest order bid here in improving CI for agents? Akshat [00:52:32]: Well, there’s a lot of wasted time in CI on like Swyx [00:52:36]: Preparing Akshat [00:52:36]: Preparing your artifacts and like, getting you to the preparing your dependencies and whatnot. Swyx [00:52:44]: Oh. Akshat [00:52:44]: And, like build systems help with that. But like if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently? Swyx [00:52:55]: Oh, okay. Okay. Okay. Interesting. Yeah. another form of like, demand compute. Akshat [00:53:02]: Yeah, exactly. Swyx [00:53:03]: Yeah. Akshat [00:53:03]: It needs the same again, platform. Swyx [00:53:06]: Yeah. So, for those who don’t know, Gitpod rebranded to Ona. Swyx [00:53:09]: It was like there was this whole thing. I - I like semi-sounded the alarm at Cognition. I was like, “You should take these guys seriously because their infra is very good.” Akshat [00:53:17]: Yeah. Swyx [00:53:18]: And but, then they join OpenAI and, presumably we’ll, we’ll see Codex Cloud from the Ona team. Swyx [00:53:26]: Like which I think would be very strong. - To me, like teams like that can set up the networking and like the secure boundaries for like, and your like agents to have their own cloud each, effectively is what you’re doing and I’m just trying to draw the analogy or the differences if you have studied them. Like what is the philosophical difference? Akshat [00:53:47]: My sense is maybe they didn’t go after the right market at the right time because - I guess also got lucky with like agent use cases really taking off and, needing, like more of like a sandbox shaped thing than like, my understanding is, yeah, Gitpod Swyx [00:54:06]: Really sandboxes work Akshat [00:54:07]: Never mind Swyx [00:54:07]: Like CI/ Akshat [00:54:08]: Yeah Swyx [00:54:09]: Is sandboxes. Akshat [00:54:09]: Yeah. Swyx [00:54:10]: It’s just like build time sandboxes versus runtime sandboxes and it turned out runtime was better. Akshat [00:54:15]: Right. And the difference there is runtime sandboxes have a different configuration surface of like how you configure images, how you like attach like storage Swyx [00:54:25]: Yeah. It’s it’s fascinating. Other people, Astral also OpenAI. Python, TypeScript, and the Future of SDKs Swyx [00:54:30]: Also like Python tooling ecosystem people. Are you still bullish build- building on top of Python? Also recently Modular also got bought by Qualcomm. Just any of your takes there? Akshat [00:54:43]: Yeah. we had Python as our first SDK language because that was the language that people did data and ML in. I now have Go and TypeScript SDKs as well. and our runtime is completely language- It is written in Rust, but it’s it’s not tied to Python by any means. We haven’t seen-- I think with like inference and training stuff, people are still very Python and the interesting thing with like the agent stuff is people use our TypeScript SDK a lot more because they’re not doing anything that needs ML. Akshat [00:55:13]: I don’t think we’ll have to go beyond that super soon Swyx [00:55:16]: Yeah Akshat [00:55:16]: ‘cause Python and TypeScript is still Dominant. Swyx [00:55:19]: The last two languages in the world. Akshat [00:55:21]: Yeah. Swyx [00:55:21]: That’s it. Akshat [00:55:22]: Well, English and prompting is the fourth language. Swyx [00:55:25]: English and prompting. I occasionally talk to people who try to build new languages. They’re like, - Even, what’s his face? Brett Taylor, who’s chairman of OpenAI was like, “We need a new language for LLMs.” So no one has come across one, and I keep looking. Python and TypeScript - You have a lot of data plus, but then also they are very imperfect as just as languages themselves. Then my close is, I think Modal used to be a big bet on developer experience. Agent Experience as a Company-Building Wedge Swyx [00:55:52]: And you’ve pivoted the team to agent experience. Is it like the way now, like, do - do, - can entire companies and unicorns, multi-unicorns be built on just having better agent experience? Do you need something else? Akshat [00:56:05]: It’s a big part of our identity. it’s not just, like the very tactical, how does an agent use the CLI, but it’s also how easy is it to spin something up? Like, what is your iteration time when you wanna spin up a new service and, you wanna get something going in prod? in practice, that matters a lot, to people. And, I think it will continue to matter. Like, people are building stuff even faster, and if you give them ways to do it quickly not have overhead, then. Swyx [00:56:37]: I think the debate for me has been, do you do anything differently that is, like, very fundamentally different for developer experience versus agent experience? Swyx [00:56:44]: You seem to be on the side of they’re, they’re like this. They’re like cosine Akshat [00:56:48]: Yeah. We also have a blog post on that. Swyx [00:56:49]: Cosine similarity on, like, zero point nine or whatever. Akshat [00:56:53]: Yeah. pretty much it’s the main shift for us has been, as I said, like, we built this, benchmark, Modal Bench, to see where agents are lacking Swyx [00:57:02]: Yeah Akshat [00:57:02]: Literally add surface areas to a product if they’re reaching for something, like maybe this should just be a CLI. Swyx [00:57:09]: They halluc Oh, yeah. They hallucinate their own features. Akshat [00:57:11]: Yeah. And sometimes it makes sense. Like if they’re reaching for this thing, it’s product feedback. Like, give it to them. And then, yeah, moving-- we used to only have, like, logs and metrics in our UI, just moving all those things to the CLI as well, so they’re accessible in that form. Swyx [00:57:26]: Simple as that. Closing: Modal Bench, AX, and Execution Swyx [00:57:28]: Cool. Thank you so much. Yeah. Akshat [00:57:29]: Yeah. Thank you. Swyx [00:57:30]: This was great. Akshat [00:57:30]: This was fun. Swyx [00:57:30]: Yeah. It was a great update and, I can see why you guys have succeeded so much. it is really, focus, but also really good execution. Akshat [00:57:39]: Thanks. we have a long way to go. Swyx [00:57:41]: All right. Thank you. Akshat [00:57:42]: Cool. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:04Speaker 00

  We're here with Akshat of Model, CTO of Model, together with Veevoo. Congrats on your CVC.

• 0:10Speaker 01

  Thank you.

• 0:12Speaker 00

  Your party yesterday was amazing.

• 0:15Speaker 01

  Yeah.

• 0:15Speaker 00

  From all the photos and all the swag.

• 0:17Speaker 01

  We had a bunch of art installations, which is kind of fun seeing our products on pedestals next to Rodan.

• 0:26Speaker 00

  Very nice, very nice. When you started, it was not the GPU inference company. I mean, maybe it was in your mind. Take us back to the origin story.

• 0:39Speaker 01

  I actually first met Eric, who's a CEO, through an investor. And back then Eric was already thinking about building a new kind of runtime. And he got there thinking through why our workflow orchestration products are so hard to use. It's because you have to run them on Kubernetes. Kubernetes is hard to manage. It's not built for burstiness and custom images. And it has a terrible developer experience.

• 1:05Speaker 00

  And I'll introduce for listeners who are new. We interviewed Eric two years ago. And there's a bit more of the story there from Spotify and all those things. And I actually came across Eric through Data Council because he did that talk on the serverless container stack that you guys did, which is like, that was my first like, okay, I need to take Models very seriously moment. But it's still very unclear, like, do I actually need all this for just my data pipelines?

• 1:33Speaker 01

  Yeah, I mean, initially what we were thinking about was if we build a better runtime, it's a very useful primitive in itself. It's a, there's a lot of things that get solved by serverless functions. Like you can do ATL stuff. You can do job queues. You can do all this bursty processing, which it turns out every company had needs for. But then we also were thinking about this as like, this is a primitive that we can build a whole collection of products on, which are very vertical lives. So perhaps data engineering would have been the first one. But we were thinking about inference. Back then it was more classical inference, like computer vision stuff and running XGBoost and whatnot.

• 2:15Speaker 01

  We added GPUs to the product a year before ChatGPT came out.

• 2:19Speaker 00

  Nice.

• 2:19Speaker 01

  We just didn't think it would be that big of a deal.

• 2:21Speaker 00

  Yeah, just like add A100.

• 2:23Speaker 02

  Was there any like early key problem that really sparked off why you built it?

• 2:28Speaker 01

  Yeah, primarily it's just none of the tooling that was out there was built for, one, a really great developer experience. And also there's a general trend of a lot of the workloads that we were seeing were very, this is, I wish this better word for it, but compute heavy, like they need, one, like need a lot more resources, so you need to burst up and down a lot versus like Kubernetes designed for like slow scaling and more for like web server use cases. And also there's just a lot more specialization in like what kinds of environments these workloads run in, like we kind of sometimes need accelerators, sometimes need different kinds of images. And this is just like a consistent thing that we saw across a lot of companies. That would be the.

• 3:13Speaker 00

  Be nice. I don't know how much this factored into the early story, but I wrote a post when I was at Temporal about infrastructure, software-defined infrastructure or something like that.

• 3:22Speaker 01

  The self-revisioning.

• 3:23Speaker 00

  Self-revision, yeah, I can't even remember my own post. And then you put me on the landing page.

• 3:28Speaker 01

  Yeah, we really like the term and so we stole it.

• 3:32Speaker 00

  Because you had the insight that everything can just be in decorators next, co-located with the code, right? Was that a big part of the original story or was it just like a DX layer?

• 3:40Speaker 01

  That was really important because we really didn't want people to spend so much time writing YAML. And it seemed like you could really condense the surface area of what you're doing, put it in code so you can actually operate on it, just like you can operate in other code and like build stuff that's more expressive and dynamic. And so, yeah, that was always a very important part.

• 4:04Speaker 00

  Then the pushback is, this is a DSL.

• 4:07Speaker 01

  Yeah.

• 4:07Speaker 00

  It's your closed source. I am locked in to Moto.

• 4:11Speaker 01

  Yeah, we never really got pushback for that because the nice bit of the model is you can bring whatever code you have. And sure, the DSL is at the configuration there for what hardware you're using, how you're scaling things up. But you still own the code and that's been an important part of our story, even as we do inference now.

• 4:32Speaker 02

  Yeah. How much of do you think still stays the same today? Like if you were to build something today, DevX is obviously very important. But I feel like, you know, a lot of this has kind of been changed with just hook it up to an agent, have Cloud Code, have Codex, implement a tool. There's very agent native primitives that are kind of different than if I'm doing this myself, right?

• 4:54Speaker 01

  We've actually changed our SDK team to think about agent experience, sort of developer experience. And we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that's not even typed? When it can basically make a couple of changes in a decorator and it gets this sort of self-revisioning runtime of being able to see its changes live in action. Yeah, it just seems from the customers we talked to, they actually find the model is way faster use for agents versus operating on a different substrate.

• 5:34Speaker 00

  Yeah, because again, you collocate the infrastructure requirements to the code that runs it. While the negative thesis now is that nobody's looking at their code anymore, so there's no point.

• 5:44Speaker 01

  Yeah, I mean, people aren't looking at code. One thing we actually still see is really important is observability, like how good is your dashboard? And of course, like we have, we push a lot of it to the CLI so that agents can do their own investigation. But you still need humans to go interpret what's going on and, you know, make judgment calls and whatnot. And that's, I feel like. Maybe more important now than looking at the code itself.

• 6:11Speaker 00

  Yes, because like, you know, you can try to treat the code as a black box and then use sort of see the observable action that comes out of it and then just prompt to change. Yeah. So I think actually it takes a bit of restraint to not specialize. To say, I want to ship a new primitive and then just be general purpose. People ask you, what are you for? You're like, I don't know, we can do this, we can do that.

• 6:36Speaker 02

  I'd be curious to say, you know, like, okay, if we were to ask you like, what is Model for even at a high level? There's a lot you guys do, sandboxes, GPUs, everything. How do you answer?

• 6:46Speaker 01

  Model is a cloud platform that's built for, where we've built the primitives from scratch for AI applications. And right now it basically covers inference, training, fast processing and sandbox workflows. But we're building a lot more and I noticed you didn't say web server. So there is still a role for like the always on large scale Kubernetes type of things. Yeah, absolutely. We're not trying to compete with the renders of the world. Because yeah, we think the differentiator for us is our other workflows that need specialized compute, need to scale up and down a lot. Yeah, they're shaped differently.

• 7:26Speaker 02

  I think you're building a lot of it alongside the startups, right? They're innovating quite a bit, even in your like latest blog post, like even in the series. See the customers that you mentioned here, the cognitions, tech go ons, ramps and whatnot, they're innovating with you, right? And that's not something AWS is doing directly with.

• 7:45Speaker 01

  Yeah, absolutely. I think, I mean, this is again, classic, we're a small team, we can move really fast. Our engineers are working with our customers and figuring it out.

• 7:54Speaker 00

  My first week at cognition, I walked in, there was someone wearing a Model shirt. I was like, what are you doing here? They're like, yeah, I just, I am embedded inside of Cog.

• 8:05Speaker 01

  Yeah, I think that was patent. We sent him over because

• 8:08Speaker 00

  the latency of communication was too high, otherwise.

• 8:12Speaker 00

  Yeah, you know, distributed node, you have to place one and co-locate. So I had a direct personal experience, right? So I worked on Small Developer three years ago. It was inspired by Cloud One. I think you onboarded me at some point, like just before, and I was like, "Oh, like, I need some bursty compute." Like, I was just gonna try using Model. And it was a pretty pleasant experience. Apparently, I showed up in the board meeting, like the analytics.

• 8:39Speaker 01

  Yeah, you blew up on Hacker News and you got a big traffic spike. I actually, I think the way you use Small Developer was model functions for running stuff, which was, like, that was a good use case.

• 8:53Speaker 00

  Yeah, to me, that was proto cognition.

• 8:55Speaker 01

  Right.

• 8:56Speaker 00

  If only I had, like, stuck to it. Like, that was like, if you just like draw the tech tree out, you're like, yeah, like, probably this will happen.

• 9:02Speaker 01

  Yeah, like, you were so close.

• 9:03Speaker 00

  I didn't realize.

• 9:07Speaker 01

  The funny story there is, at the same time, we were talking about some customers who needed something like sandboxing. This is like 2023. So we built-

• 9:17Speaker 00

  They should use a new API right after that.

• 9:18Speaker 01

  Yeah. Like, we built sandboxes in May of 2023, before anyone was even knew this was gonna be a thing. And the first example we published was, we took Small Developer and put it in a loop, so the agent can iterate on itself.

• 9:33Speaker 00

  Loops are hard these days.

• 9:34Speaker 01

  Yeah.

• 9:35Speaker 02

  Loops in, when was this, 2023?

• 9:38Speaker 01

  Yeah.

• 9:39Speaker 02

  A small chick.

• 9:39Speaker 01

  Yeah, it's like mid 2023.

• 9:41Speaker 00

  I mean, so, I mean, there's sort of, obviously, for listeners, like, the problem was the models are not built for any of this. Right? Like, you're just trying to, like, they're not post-trained to understand, like, you know, looping and like self-correction. And tool-calling was there, but like, also not that great.

• 9:55Speaker 01

  Yeah, I don't remember if you used tool-calling in this one, but yeah, the models are just diverse after like 10 iterations and not produce anything meaningful.

• 10:03Speaker 00

  Yeah, but like, then, so, I mean, okay, like, now, talking to myself three years ago, the answer is, collect all the failures, build benchmark, and then, collect all the, you know, examples, build the R O environment,

• 10:14Speaker 01

  Right.

• 10:15Speaker 00

  Sell it for like $10 billion to Meta, and then, and then also train a model and then sell that for $60 billion to Elon. And this is-

• 10:23Speaker 01

  Yeah.

• 10:23Speaker 00

  The money machine, like, it's like, it's actually about that hard, though.

• 10:28Speaker 01

  I mean, it's hard to have that kind of inherent conviction that the stuff will get that much better.

• 10:33Speaker 00

  In retrospect, it's so fucking obvious.

• 10:36Speaker 01

  Fair enough.

• 10:37Speaker 00

  Like, what else would we be doing back then? I don't know. Anyway, yeah, so, so, I mean, that was the start of your sandboxing journey, right? I feel like it didn't blow up, blow up until like last year.

• 10:49Speaker 01

  Yeah.

• 10:50Speaker 00

  So there was like a couple of years of quietness.

• 10:52Speaker 01

  Exactly, yeah.

• 10:53Speaker 02

  But like, very underrated product value. Like, um, my experience with Model, Charles, before he had joined Model, met this guy at a hackathon, and he really insisted, we want to run some small model, you know, not hosted anywhere. And he's like, ah, there's this cool company, Model, they'll like spin up a GPU sandbox, we can throw it on there. They'll take a hugging face link and, you know, like, there's so much value just right there, right? Like, instant hosting, spin it up, spin it down. It'll stay cold, but, you know, we run the demo a few days later, it'll come back up. And like, all this stuff in retrospect, like, it's still what we needed, like, today.

• 11:26Speaker 01

  Yeah, I mean, it's still needed today. Obviously, workload shapes have changed a lot as we run stuff for people with really massive production scale. And there, it's not about scaling from zero to one, but it's how do we scale really elastically from like 1,000 to 1,500 GPUs very quickly in a given region as the same shape problem.

• 11:50Speaker 02

  Okay, so you look at, say, Cursor Composer, right? They had a, we'll do RL on a model every couple hours.

• 11:58Speaker 02

  You guys have a whole version of RL inference gym and whatnot. When you look at workloads like that, you're basically doing train runs where you need to scale up, scale down every hour, thousands of GPUs, right? That's the example for we do need it, right?

• 12:12Speaker 01

  Well, actually, I'll take a step back and maybe talk about like how people use Model today. Because our biggest use case actually is elastic inference. And the thing we first found product market fit with was inference for custom models. So we kind of stayed away from the LM space. And we were serving companies like Suno for audio, Runway for video, robotics, comp bio companies that. Train their own model elsewhere, but Model is the best black box for deployment, scaling to however many GPUs you need as your traffic pattern changes. And we saw all of them actually have a very unpredictable traffic pattern.

• 12:53Speaker 01

  It's like Diurnal, it's some days like the company will launch and they need like way more. And it's not just one model that they deploy. They, all these companies deploy lots of different models in different regions. And so the auto scaling problem becomes even harder because then you have to scale within a certain region and those cycles are offset. So different times you can scale up in different regions. So that's like our sort of.

• 13:22Speaker 02

  And that in and of itself is a huge category. There's a bunch of inference providers which provide this fireworks, does this as a service together, whatnot, Base10. That's kind of carved into its own niche for language models, at least right now.

• 13:36Speaker 01

  Yeah, I mean, the thing that we have actually specialized in is the auto scaling aspect because we found that it's not universally true that everyone else can auto scale. And we've gone deeper into it on the tech side, but we've incorporated GPU snapshotting to the product so we can actually take the GPU state, like your Tars compiler model, snapshot it and next call starts way faster. And so going back to your question, that's why you need a lot of burstiness or inference. But then people also do a lot of on-demand training, like for our stuff, your rollouts are bursty, as you said. People also do a lot of batch jobs. So we'll see a lot of companies, before they have a training run, they'll need thousands of GPUs to run encoding or something like that.

• 14:19Speaker 01

  And I think those things are much more bursty than, I agree that agents are not that bursty. Sandboxes are, except when you're doing RL. RL is insanely bursty.

• 14:29Speaker 02

  Yeah.

• 14:30Speaker 01

  Yeah, like when you're doing rollouts, you'd sometimes need 100,000 sandboxes.

• 14:36Speaker 02

  Yeah. I'm curious if you've seen early sparks of continual learning. There are some people, like our friend Zengram recently announced this, they're trying to do training. That also seems like a different workload, right? If you're doing training 24/7 per se, there's a very weird dynamic of how you're using GPUs between people and whatnot, but seems like something you guys would work for.

• 15:00Speaker 01

  As you said, we're fortunate to work with a number of customers of the frontier and grab some of our customers. And they are taking the primitives we have and trying to use them in very interesting ways, like continual learning. It's possible as the stuff gets better, so that will be part of our offering as well if more people need it. But we're just waiting to see how this shakes out. Is there a primitive that you added after sandboxing that was the next step in the story? I guess we've been going much deeper into LLM inference because we've realized that some of the advantages we have with auto scaling, again, especially in different regions and whatnot, are not present elsewhere.

• 15:45Speaker 01

  And the place where we had a gap was we weren't working on the model there itself, like we were a black box. And we realized that we actually can get to frontier level model performance by having great people who work on all of this. And we've actually been open sourcing a lot for our work in terms of recently shared a work on D-Flash, which is a block-based speculator. And we've open sourced all of it. So you can get, by using open source D-Flash, you can get the same performance as you would with one of the proprietary providers. And the next thing we're thinking about here.

• 16:23Speaker 02

  I thought this was actually an interesting thought process as well, right? I think in here you make a claim that, you know, not a claim, just that how effective speculative decoding really just gets you. Anything you want to point out from this around, you know, what people should know?

• 16:38Speaker 01

  Yeah, absolutely. I mean, the high-level summaries would have helped to describe what speculative decoding is. Yes. I think so we've covered Ego and Hydra and all those things, but it was like two years ago.

• 16:51Speaker 02

  I think it doesn't hurt, right?

• 16:52Speaker 01

  The speculative decoding is you have a smaller model called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model verify all of this, all the tokens are predicted. And the reason it's faster is if you're predicting one token at once, you kind of bound. By memory bandwidth, but if you can batch the verification of the draft model, then you're much more efficient using compute and it's faster. And as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get a speed up that's multiple times of, you know, the original model's speed. And that's what we highlight here. It's like people talk a lot about we made these kernels faster and whatnot,

• 17:37Speaker 01

  but improving kernel only give you like a few percentage points improvement. And increasing accept length literally is a multiplicative decrease. Like two to four X. Yeah, exactly.

• 17:48Speaker 02

  Without much head on performance.

• 17:50Speaker 01

  Yeah. I think it may be, I mean, you are running a second model, right? So maybe it's something more expensive in the compute, but.

• 17:57Speaker 02

  I meant quality performance, but yeah, I mean, I think.

• 17:59Speaker 01

  So there's no drop in quality performance because you're always, you're never accepting a token that. It's strictly better or it's same. Exactly. Right? Yeah. And so we've been working a bunch on D flash, which is a block based speculator. So it's instead of predicting one token at a time, it's predicting a block. And we've been open sourcing our work with it. The, the next thing for us here is for, for helping people train speculators and custom models. It's, it's something that traditionally is very FDE driven, support deployed engineer driven. Like you work with customers and we can do that. And our vision for, this is why we launched Auto Endpoints is we want to make frontier level performance available to everyone.

• 18:43Speaker 01

  And so we mentioned this in the announcement, we kind of teased it. The next thing we're, we're launching is basically as you run an auto endpoint, we, we shadow traffic and.

• 18:54Speaker 02

  Do you want to explain what other endpoints are?

• 18:57Speaker 01

  Yeah.

• 18:57Speaker 02

  Lovely.

• 18:58Speaker 01

  Yeah, so this is, I guess, going back to your model is, is you touch the code, but sometimes people actually don't want to touch the code and they want to get started with an endpoint that works and has all the great performance and scalability that model has. So we've made that easier with basically a way to create an endpoint from our UI, from the CLI that has all of our optimizations that we. Talk about like the D flash stuff already baked in. And there's full transparency. So we give you the code. You can go run it yourself. And if you want, you can eject out into the full model experience, which we see as people get sophisticated, they do want to tweak the models. They want to find it stuff. You can still do all of that. It's not a black box.

• 19:42Speaker 01

  And yeah, the next thing as we tease later in the post is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves. Again, without having to talk to a person and yeah.

• 19:59Speaker 02

  I guess just to kind of understand it directly. I mean, you know, obviously you you have the GPUs, you have an endpoint that's compatible, you serve open model. If someone was to do this themselves, what's the delta that you guys provide? So you do a lot of open source, great work on effective inference. How does it compare to say, I take the same model GLM 5.2 FBA take off the shelf inference engine VLM SG lang, you know, get compute of similar capacity, similar cost. What's the kind of delta that plugging into something this like this offers outside of the benefit of, you know, scaling?

• 20:34Speaker 01

  It's kind of interesting because we've taken the approach of open sourcing our contributions and upstreaming them. We work closely with the SG lang team. We actually want the improvements that our team comes up with to be there in open source for others to use, even outside of model. The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance first. The other thing is with these endpoints, we are way more elastic, as you said, than anyone else. And you have true scaling to zero. You have true burstiness. And in practice, that matters a lot more to people than

• 21:17Speaker 01

  this finding the GPU and running model code.

• 21:20Speaker 02

  Yeah. And I will say it's actually not that straightforward to just like what I said is easier said than done, right? It's, I think, still for the average person, still hard to just gut check using different, there's, there's quite a bit of combinations you can make there. The trade offs aren't really known at face value.

• 21:40Speaker 01

  Yeah, I mean, it's not just that. I think it's that running production-grade inference is a hard-inferior problem. Even if you subtract out the auto scaling, it's controlling things like tail latency and making sure every request is delivered at least once and whatnot.

• 21:59Speaker 00

  There's a lot of innovation that you can do here. I think it's very interesting that you're starting to encroach on, like, as you become a full cloud, you're starting to encroach on other people's turf.

• 22:11Speaker 00

  What will you not do?

• 22:13Speaker 01

  Well, we want to follow our users and make sure they get like a platform that has everything that works well together. So right now, we're kind of focused on the model lifecycle and the agent lifecycle. So both like going from data prep to training to inference. And then also, if I want to deploy a background agent, let's say, you know, sandbox, super persistent storage, a whole bunch of other stuff.

• 22:38Speaker 00

  We talked to Cole, who did OpenInspect, yeah. And obviously, RealInspect also is on Model.

• 22:44Speaker 01

  Yeah, so Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and works well.

• 23:02Speaker 00

  Yeah, that's the new CTO of Ramp right there.

• 23:05Speaker 01

  Yeah, Raoul.

• 23:08Speaker 00

  It was really, really fun. Yeah, I mean, you know, I think all very bullish. Like, you know, one of my reflections was also, I did not originally, because obviously when I met you guys, you weren't that much in the GPU game, and now you're all about inference. And one of the points that I hinged on for Jensen's keynote at GTC this year was what we're calling like the inference inflection, right? That, let's say, in AI workloads or machine learning workloads, it used to be like, let's call it eight to one GPU to CPU, and now it's more like one to one, which is like a interesting, like, because of how much agents basically are blocked or call out

• 23:51Speaker 00

  to CPU-heavy stuff, the actual, like, limiting factor that swings back and forth from GPU to CPU a lot more than, it used to be all GPU and then occasional CPU.

• 24:01Speaker 01

  Yeah.

• 24:02Speaker 00

  GPU, CPU. And now it's like just constantly, and you just have to co-locate everything.

• 24:08Speaker 01

  Yeah, and that's one of the things that actually, again, we see is something appealing about Model, which is we've built this capacity pool that spans seventeen cloud providers. So we're very good at running on various kinds of cloud capacity across the world.

• 24:24Speaker 00

  You don't have your own data centers?

• 24:25Speaker 01

  We don't have our own data centers. We just run across a lot of NeoClouds and other providers.

• 24:30Speaker 00

  Yeah, you're running the math and you're like, what's the cut over point where you're like?

• 24:36Speaker 01

  Yeah, it's a good question. I mean, part of it is we see our differentiator in the software layer and being capital light and focusing on the software helps us move really fast. So far, it's worked out well because there are so many other people building data centers that we're able to work effectively with them and, again, focus on what makes us special.

• 24:55Speaker 00

  Yeah. Seventeen gets you into like the local providers sometimes, like, who's the most interesting one?

• 25:02Speaker 01

  There are actually a lot more NeoClouds than you expect and they all have various degrees of various levels of reliability. And that's why something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls out the bus or something happens, user workflows are not affected. And that actually lets us use a lot more capacity than, you know, you as a user would be able to.

• 25:32Speaker 00

  It's a useful thing to have because now everyone knows like what layer you are and like you sort of optimize for being the super cloud of all clouds.

• 25:41Speaker 01

  Yeah, that's the idea. And so I guess when you mentioned colocation, that's another interesting thing where one thing we've seen is people come to us when they want very specifically located CPUs or GPUs like they want.

• 25:57Speaker 00

  Or to pin it in like EU?

• 25:59Speaker 01

  Exactly, or EU, US, Australia.

• 26:02Speaker 00

  Data locality thing or performance or what?

• 26:04Speaker 01

  It's either data locality or latency.

• 26:06Speaker 00

  Yeah.

• 26:07Speaker 01

  Like you want your, they're running sandboxes in Model, they want them to be right next to it.

• 26:11Speaker 00

  Yeah, it's easy, you know. That is important and all those things. And so like you've kind of accidentally, I don't know if it's accident, but like you've built the perfect primitive for agents to to express themselves. And then, you know, like, it's almost very funny how every extra development just involves more file system, just involves more CPU.

• 26:30Speaker 01

  Yeah.

• 26:31Speaker 00

  Just like the things that you already have. I don't know much about if there's any like networking usages that are interesting, but you've also done some good work on networking.

• 26:40Speaker 01

  Yeah, I mean, that's exactly right. Like, we were sort of just taking compute through the networking and building stuff on that layer for, again, a lot of stuff people need.

• 26:49Speaker 00

  Yeah.

• 26:50Speaker 01

  We see a few interesting networking things coming up. One is people actually want networked sandboxes. So we have.

• 26:57Speaker 00

  For like a Docker cluster type thing.

• 26:59Speaker 01

  Yeah.

• 27:00Speaker 00

  Docker Swarm, what is it called?

• 27:02Speaker 01

  Compose.

• 27:03Speaker 00

  Compose type thing.

• 27:04Speaker 01

  Yeah, so actually, if you want Docker Compose, our sandboxes now support this thing called sidecars. So you can, a sandbox is actually a pod of containers, and you can run multiple containers in a sandbox. Also useful because going back to networking, people want a lot of control over outbound networking from a sandbox. They might want to run a man in the middle proxy for like, maybe logging stuff for RL or controlling how your request can happen to a domain injecting credentials. And, yeah, so we've kind of had to build a lot of that stuff ourselves.

• 27:38Speaker 00

  Yeah.

• 27:39Speaker 01

  But then also sometimes people actually want sandboxes spanning multiple nodes to talk to each other, which is an emerging thing we're seeing. We have support for that for a different reason. And, yeah, we'll see if that becomes.

• 27:51Speaker 00

  Like just an open socket. This is directly like MTLS.

• 27:56Speaker 01

  We do support that, which is you can expose a tunnel inside a sandbox.

• 28:01Speaker 00

  Yeah.

• 28:01Speaker 01

  And then you can either expose the public internet, or it can be mine, you can add like a HTTP odd layer above it. But we have this thing called ISXPN, which we haven't talked about, which is this like overlay network using IPv6 addresses. So if model containers within the same workspace, when this is enabled, can actually address each other using this private IPv6 address, and no one else can. This is like sort of private networking for containers. We actually built it because we needed it as a primitive for our distributed training product. So we have this. Other feature, which is you can add a decorator to a function,

• 28:44Speaker 01

  and you get a cluster of GPUs, and they have RDMA networking. So you can run it to do your training job that's truly serverless. And we need the overlay network for that. But then, we've seen that people are using it for other reasons, and I'm kind of intrigued to see, yeah, what would people do with it?

• 28:59Speaker 00

  Build primitives and people figure it out, right?

• 29:01Speaker 01

  Yeah, exactly.

• 29:02Speaker 02

  You put on a pretty interesting.

• 29:03Speaker 01

  They're like, they read the docs, they think, let me use that for something.

• 29:06Speaker 00

  Yeah, they never intended to.

• 29:08Speaker 01

  This was literally not even in our docs page. People somehow found it and they're using it.

• 29:14Speaker 02

  I mean, the way you portrayed it with like RDMA versus TCP, like, very well laid out, but just the transfer speed change at scale for RL, like, yeah, you have it, you have it built in. I'm sure someone found it, it's found it to be a lot more efficient before you actually made a thing out of it, right?

• 29:32Speaker 01

  Yeah, and not to split hairs, I guess the overlay network actually is the TCP overlay network. The reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that. But then people found the TCP part.

• 29:48Speaker 00

  Can I tell you, this is like a big aha moment for me? Because, so I, you know, I reviewed 2,200 submissions for the World's Fair.

• 29:56Speaker 01

  Yeah.

• 29:56Speaker 00

  And then I got this from John Osterhout, who I don't know if, do you know John Osterhout?

• 30:00Speaker 01

  The name sounds familiar.

• 30:02Speaker 00

  He published, he's a well-known professor, published a lot of interesting software design books. And this is the talk he chose to submit. It's on RDMA and inference. And I'm like, you wouldn't think that this guy, who is like kind of operating systems guy, would care about RDMA.

• 30:20Speaker 01

  I mean, it makes sense to me because.

• 30:23Speaker 00

  This is the cloud, right?

• 30:25Speaker 01

  Like, the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL, is, there's a lot of degrees of freedom, and it is basically a systems problem of love.

• 30:41Speaker 00

  Yeah.

• 30:41Speaker 01

  Moving memory around.

• 30:42Speaker 00

  Yeah.

• 30:43Speaker 01

  Scheduling.

• 30:44Speaker 00

  This shows you how primitive my understanding of networking stuff is. Is this like the domain of WireGuard as well?

• 30:50Speaker 01

  Not quite, so.

• 30:51Speaker 00

  Adjacent?

• 30:52Speaker 01

  So.

• 30:53Speaker 00

  Explain everything.

• 30:54Speaker 01

  Sure, sure.

• 30:56Speaker 02

  How do we move memory around GPUs?

• 30:58Speaker 00

  Oh, sorry. Yeah, that is memory. Sorry. I was talking more and maybe I was talking like five minutes back about the private IPv6 addressing that you've set up. It's basically a VPN.

• 31:10Speaker 01

  Yeah, it's sort of like a VPN. And yeah, WireGuard is, yeah, you're right. It is.

• 31:16Speaker 00

  Yeah, you already moved on to the topic.

• 31:17Speaker 01

  Similar, in the same space, WireGuard is encrypted and this is.

• 31:23Speaker 00

  You don't need encryption.

• 31:23Speaker 01

  Yes, it's not encrypted. That's the main difference. This is TCP and we have EBP programs that will reject or allow the TCP connection based on whether you're allowed to do it.

• 31:35Speaker 00

  It used to involve a full site card, but now you have EBP in a Linux kernel?

• 31:39Speaker 01

  Yeah.

• 31:39Speaker 00

  Yeah. I don't know if this is a natural follow on to the topic of like my skepticism on distributed training is that while people spend a lot of money on like cables to hook up GPUs and even that is not like fast enough and that's the bottleneck, is your networking fast enough?

• 31:59Speaker 01

  Yeah, so I guess you're talking about sort of fully distributed training, like a dial local or something, which is like cross.

• 32:06Speaker 00

  That would be, yes, that's the extreme.

• 32:08Speaker 01

  Yeah.

• 32:08Speaker 00

  You're kind of in the middle and then other people would have like the Mellanox cables up in like their actual data center.

• 32:14Speaker 01

  When you run multi node training on Model, the RDMA, I think Mellanox is or InfiniBand is like, is, it all sees RDMA. But basically it's a way to bypass the TCP networking stack and transfer stuff much faster between one node to the other. And we have, I think, like three terabit per second internal networking, which is the standard that's needed.

• 32:42Speaker 00

  Okay, so I misunderstood what part of stack you were.

• 32:44Speaker 02

  Fifty gigs of TCP.

• 32:45Speaker 00

  Yeah.

• 32:45Speaker 02

  Two.

• 32:45Speaker 00

  Yeah. Okay.

• 32:49Speaker 00

  Yeah, I mean, very impressive work. So effectively, you're extending sort of like the modal philosophy to the trading cluster?

• 32:59Speaker 01

  Yeah, and we're not going for, obviously, like, large-scale pre-training runs. The thing that we've built multi-trainer training for is we see a lot of smaller-scale post-training. Like, people are post-training, like, medium-sized Qwen models, so they can get higher quality on inference. This is a perfect fit for something like that.

• 33:21Speaker 00

  Yeah, that is my impression of how a lot of these labs explore branches in post-training and then eventually merge whatever they find in.

• 33:31Speaker 01

  Yeah, the other use case we've seen for multi-trainer training is even if you have a big cluster, your researchers are still doing small runs. And you're having elasticity there better a lot more.

• 33:41Speaker 00

  Yeah, this is actually like the current limiting factor for auto research, which is like, you basically need to give your model some GPUs.

• 33:51Speaker 01

  We have a blog post on auto research and modal is, yeah, like, turns out to be a pretty good substrate for that.

• 33:59Speaker 00

  So my impression is auto research means many things, like anything that on J-Coins. Right now, it's still Science Fair, right? Like, not actually, like, I don't know how we do actually doing this.

• 34:09Speaker 01

  I thought the same thing.

• 34:11Speaker 00

  Yeah, you would know.

• 34:11Speaker 01

  We, like, our internal, both training and inference teams, actually use the sort of, the general shape of this quite a bit. Like, we have this one internal repo called AutoInference, which essentially, we've automated our own FDE efforts using this harness, which is the agent will just spin up a heap of different things. It'll even run like Nvidia inside profiler and it'll like tweak configs and it'll arrive the right thing. It'll change your GPUs from H200 to B200 and actually works really well.

• 34:46Speaker 00

  Nice.

• 34:47Speaker 01

  So.

• 34:48Speaker 00

  By the way, I enjoy that your FDE is so technical that you have to do these things. It's very different from FDE from other people.

• 34:54Speaker 01

  Yeah, yeah, for our FDE team is essentially they're like applied inference researchers or applied training researchers.

• 35:02Speaker 00

  Someone told me like they have to be able to build, but they also have to be able to sell. Do they have to sell or are they like, they're good, it's just like post-sale type of thing?

• 35:09Speaker 01

  It does, being able to talk to a customer and engage effectively with that matters a lot.

• 35:14Speaker 00

  They all want the same thing.

• 35:15Speaker 01

  Yeah, but it's not really a sort of sales thing. We pair them with, we have solution to our connects as well that are born on the pre-sales side.

• 35:23Speaker 00

  Okay, let's spend a bit more time on auto research. This is a big focus for me for this year. Where does this go? You know, like have, have people explored enough? Like, you know, there's all these beautiful charts of like improvement, improvement to the level off a bit. And then you find the next thing. Is this basically sort of one abstraction up from normal training? Is that how we think about it? Or do you think about it differently? Like model level training versus basically like AI driven hyper parameter search. Someone, some people call it like neural architecture search or whatever, right?

• 35:54Speaker 01

  Like, yeah, I mean, I, so the, the stuff I've seen people do with it is nowhere on the architecture level. It's pretty much tweaking parameters, but it's, it's basically a hyper parameter sweep. That's guided by some sort of model intuition. So it's like much more efficient than whatever other sweeper you would have.

• 36:13Speaker 00

  Yeah, I mean, you know, it's just, it's just a question of where you want to spend your computer, you know?

• 36:16Speaker 01

  Right.

• 36:16Speaker 00

  Cause you can just throw infinite amounts of money on this and somehow you'll bang on Shakespeare, you know?

• 36:22Speaker 01

  Yeah, infinite monkey.

• 36:24Speaker 00

  Yeah, I mean, so like very good for model. And I think it's also very important that agents can spin up other agents. You can spin up their own infrastructure, like very, like very good for you. How good are L M's are generating model code? Like, you know, the, the benefit of existing pre L M's is that you are in the data.

• 36:42Speaker 01

  Yeah, they're, they're actually surprisingly good. I think like pre Claude four, they were not. And then now they're able to one shot stuff out of the box. We're playing around with releasing like a model bench for like the harder things that the L M's cannot do yet. And, and maybe.

• 36:59Speaker 00

  What's an example of that?

• 37:01Speaker 01

  I think the, the things that. Sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability, like how to, something is failing, like, how do you look at the logs and then update the right thing? It's sort of reasoning about that. But they're able to one-shot, like.

• 37:23Speaker 00

  Yeah, you can't just add a skill to it.

• 37:26Speaker 01

  Yeah, so we have a model skill now that, which is kind of actually why we built this model bench. It's to find things like that, so we can, we can address them in our.

• 37:35Speaker 00

  Tune a skill, yeah. No, no, I mean, it's good. Are you facing any shortages? You know, like, we talk a lot about GPU shortages, but also CPU, also memory.

• 37:44Speaker 01

  Yeah. We have had a lot of growth, which means that there's a, we've had to be much better about proactive capacity planning.

• 37:55Speaker 00

  Yeah.

• 37:56Speaker 01

  So we have a.

• 37:57Speaker 00

  Which, by the way, like, it's like MBA, it's like dream job, it's like just planning this stuff. I think last time you and I talked about, maybe about this.

• 38:03Speaker 01

  Yeah, I mean, we have a really competent team of people that we call, the role's called compute strategy. So, yeah, if anyone's listening here, wants to work on that.

• 38:13Speaker 00

  Compute strategy?

• 38:13Speaker 01

  Yeah, yeah.

• 38:14Speaker 00

  I mean, the normie is called FPNA or something?

• 38:18Speaker 01

  Well, it's more, it's not FPNA. It's, it's, there's a lot of interesting financial questions of like, like, what is the blend between one-year and three-year reservations? How do we forecast our own capacity? How do we, basically, especially if our capacity is very fungible across different GPU types and different regions, like, you basically have to model a lot of it. And you also have to have an opinion on how the supply chain is going to evolve. And then you have to, like, take bets based on that.

• 38:50Speaker 00

  Tokenomics. This is like, probably not a real point, but I was trying to think about, like, what other industries, I was trying to think about, like, you know, we cannot be first to, like, these kinds of problems.

• 38:59Speaker 01

  Yeah.

• 39:00Speaker 00

  And what other industries have had this? And I was like, airlines with, with fuel. And like they have to hedge their fuel. And like, I think for a long time, Southwest, because they made like a hero fuel bet, they like were like super low cost because compared to everyone else.

• 39:14Speaker 01

  Yeah, I hadn't thought about that.

• 39:15Speaker 02

  We're at a fun time too, you know?

• 39:17Speaker 01

  Yeah, it's a lot of the compute business in general for us is also about being very good about capacity management. That is how you have great economics, but also over time is how you can unlock more value for customers. Like, one of the things we're building now is like a way for customers to get, if they don't care about latency, like get much cheaper pricing and they'll get results back in like next 24 hours or something, like a batch tier essentially.

• 39:47Speaker 02

  Yeah.

• 39:47Speaker 01

  And those are levers we have because they control the whole stack and scheduling and whatnot to give people a sufficient.

• 39:53Speaker 00

  Yeah, I feel like they're not as popular, like those, like the frontier labs have all those APIs, they're not as popular as they should be.

• 40:00Speaker 01

  The demand that we see for something like that is actually not for L M's. Although sometimes people want to run evals and do synthetic data prep and there it makes sense.

• 40:10Speaker 00

  Okay.

• 40:11Speaker 01

  But it's from a lot of non L M companies, like people who are doing computational bio, like they haven't run really big batch jobs and they don't care about when they get it back.

• 40:22Speaker 00

  Yeah, and like they have a reasonable, it's also like a cousin to the stopping problem of like, will this finish in time?

• 40:31Speaker 01

  Yeah, you can, you can bound it.

• 40:33Speaker 00

  Yeah.

• 40:33Speaker 01

  Like you can give P P S on it.

• 40:35Speaker 00

  Yeah, yeah. I think what's, what's interesting is like the, the next phase of model, like what, you know, do people expect from you now that you're sort of established and you're like a well-known computer player among all these leading companies? You had it in France launch week and we talked a little bit about the launches. Like what else, what else should people know?

• 40:54Speaker 01

  We are building primitives that make our user's lives much easier. So I think, for example, LLM inference, thousands more companies are going to post training their own models and deploy open source models for inference. So we're thinking a lot about what is the best product shape for that. And that involves everything from our training gym to then endpoints that get frontier level performance. Again, but I haven't talked to anyone. It looks somewhat different on other verticals. Like, we're also seeing a lot of real time audio video stuff in there, which is why like, we're working on things like regional routing with fallbacks.

• 41:39Speaker 01

  So you can, you can get sort of GPUs that are as close to users as possible. So you get like low latency for video streaming and whatnot. And then on the agent side, it's, we're still sort of working very closely with our customers because stuff is changing so fast in terms of what they need. And I think beyond sandboxes and persistent file systems, there's a lot of other things people need from this agent stack as they build production agents. So yeah, we're thinking about those other things that fit in there.

• 42:13Speaker 00

  I wanted to ask what the other things are.

• 42:15Speaker 01

  Yeah. I think, okay, so I do think a lot about the principal components of cloud.

• 42:22Speaker 00

  And you do talk about computerized networking.

• 42:25Speaker 01

  Yeah.

• 42:25Speaker 00

  Because so far for me, it's fine. So far for, I mean, the first couple of generations of cloud, it's fine. What's different, qualitatively different about agents that you need some new permission level? Like, a lot of people, you know, obviously, okay, and I'll just kind of spew tokens at you until it like, hopefully sparks something.

• 42:43Speaker 01

  Yeah.

• 42:44Speaker 00

  Like, the new level now is whatever Cloud Code does, which is dangerously skip permissions or like allow this by command or like whatever, right? And, and sometimes they're like, well, okay, well, we have like this adaptive thinking mode where like, just, just trust me, bro. I will make the calls for you. Is that it? You know, like basically like sort of LLM mediated permissions.

• 43:03Speaker 02

  Now you're looping it with a goal and flooding it row.

• 43:06Speaker 01

  Yeah, I mean, I'm skeptical of LM media permission for stuff that is at the sandbox level because you do want hard boundaries.

• 43:16Speaker 00

  Yeah.

• 43:16Speaker 01

  Otherwise, obviously, someone can exfiltrate stuff.

• 43:20Speaker 00

  But like, maybe that's old school thinking. Maybe we're the dinosaurs. Maybe the AI OS or the LM OS is really the kernel is a goddamn LM. Like, it makes you feel uncomfortable.

• 43:31Speaker 01

  Yeah.

• 43:32Speaker 00

  But that's what trusting the LM is. Like, imagine a spherical cow perfect LM.

• 43:36Speaker 01

  Right.

• 43:37Speaker 00

  That it?

• 43:39Speaker 01

  Maybe, maybe.

• 43:41Speaker 00

  I want to test the boundaries, right? Like, obviously, I don't believe that. But I want to see where I'm wrong because that's the non-consensus.

• 43:49Speaker 01

  Yeah, I mean, I think you always need hard guardrails when you want, and you can pair those with softer guardrails, right? And the US can be LM mediated.

• 44:00Speaker 00

  And with a couple of your commentary on like the ecosystem outside of Model, managed agents, everyone has one. Gemini, OpenAI, Claude, very useful for you. But also, like, it is their way of starting to edge into your space.

• 44:16Speaker 01

  Yeah.

• 44:17Speaker 00

  What's going on?

• 44:19Speaker 01

  Yeah, I mean, we're very excited to partner with Anthropic and some of the other foundation labs. We'll name who we're also working with. The way we see it is the managed agent thing is a great place to start if you're starting out building an agent and but then when you get to

• 44:39Speaker 01

  building something more production grade, like you're a company that's like RAP, that's building their own, RAP also runs their accounting agent on us, so their external facing agent. You need a lot more control over your computer on things like what sort of, how do you persist different files that the agent has access to? And how do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to that point, you kind of want a specialized sandbox provider that gives you those things. And that's the role that we are trying to play.

• 45:15Speaker 00

  Yeah.

• 45:16Speaker 01

  We don't really have an opinion on the harness, whether it runs in it's a cloud managed agent and you hook it up to Model Sandbox, or you run the harness in Model Sandbox. We'll see where people converge with that.

• 45:26Speaker 00

  Yeah. Do you have any opinions on like the meta harnesses? It's just another layer on top of these things.

• 45:31Speaker 01

  You mean like the Open Pi and?

• 45:33Speaker 00

  Open Pi is one. I think Vercel had one, which I can't remember the name of right now. Fred Schott had one. And then to me, most recently was Data Bricks that had Omnipage. All these are sort of meta harness, like it's kind of pseudo agent cloud type things.

• 45:49Speaker 01

  I personally have not played around with them.

• 45:53Speaker 00

  Yeah.

• 45:53Speaker 01

  I mean, everything is bullish model as long as it consumes more infra. That's why we're focusing on the infra layer. It's somewhere where our relative competences and also it's a hard problem to solve.

• 46:06Speaker 00

  Yeah. I mean, I will say like just generally reflecting on, I don't know if there's other topics on model, but like just generally reflecting as an infra person, not as intense as you, but in that field, this has been the most exciting time in infra. Like it was boring actually for a while and you couldn't really get people excited about data infrastructure. Like Eric would get on Data Council, everyone just watched the video. And like, let's say, look at how many sandboxes I can spin up and no one gave a crap.

• 46:39Speaker 01

  Yeah.

• 46:40Speaker 00

  And like now everyone gives a crap.

• 46:42Speaker 01

  That's true. It is a very exciting time and I think a lot of that's driven by just the amount of scale all of this stuff needs.

• 46:50Speaker 00

  I think a lot of your initiatives, a lot of your like product directions makes sense in retrospect, which is like the best kind. But I wouldn't necessarily have thought about it myself.

• 47:00Speaker 02

  We need the predictions, you know? I mean, I think there's a lot that you just don't even see, right? Like you have the batch, you have the voice, you have the multimodal, but what else, you know?

• 47:09Speaker 01

  What else is coming up for us?

• 47:11Speaker 02

  Where do you see things going?

• 47:13Speaker 01

  Yeah, I mean, in general, it's clear that there's a.

• 47:18Speaker 01

  Obviously, there's a huge shift happening. I think one thing that's not as obvious to people because L M influence gets talked about so much and is also we work a lot of companies that are doing things like drug discovery and computational bio, like the Chinese discoveries of the world. Big things are probably going to happen there. We work a lot of robotics companies that are actually putting robots in like active deployments and getting good results out of them.

• 47:45Speaker 00

  Is there air gap model? Is there is there a version that is on prem air gapped, whatever?

• 47:49Speaker 01

  No.

• 47:51Speaker 00

  It's a cloud only?

• 47:51Speaker 01

  Yeah.

• 47:52Speaker 00

  Okay. But yeah, I mean, so what you're saying is like, because you're focused on primitives and they're good primitives, you find use cases in all these kinds of things. Actually, it probably diversifies you a little bit away from L M's all the time.

• 48:05Speaker 01

  Yeah, absolutely. We're our goal isn't to only serve the L M influence market.

• 48:10Speaker 02

  Just on the website, the audio, the.

• 48:12Speaker 01

  Yeah, we said both on the.

• 48:13Speaker 02

  Computational bio, images. Yeah, I mean, there's a lot here. There's Q T T T S, customizing, oh, chatterbox. You know, there was a customizing whisper.

• 48:24Speaker 01

  Yeah.

• 48:25Speaker 00

  This screen reminds me of a fallen competitor, which replicate. What's your postmortem on what happened?

• 48:34Speaker 01

  This is one thing we've kind of stayed away from is providing an API for models. Because I think providing model APIs is some of it ends up serving like a really hobbyist market, which is much less sticky. And we've always wanted to build for companies that are building sort of products and need sort of more flexibility that's not just an API.

• 48:57Speaker 00

  Which you can build an API for a model and this is clearly what it is. But you can, but what you're saying, you can wrap it into a more fully functioning backend that you that you run.

• 49:06Speaker 01

  Yeah, so actually, all for examples, it's not that spin up this model, here's an API token, use it. They're actually all code. And so the point is that this is an example.

• 49:16Speaker 00

  Starter code.

• 49:17Speaker 01

  Yeah. But you can you can tweak it however you want. And if you're like a company building a product like computational bio, whatnot, yeah.

• 49:25Speaker 00

  I guess I'm trying to tease out for listeners.

• 49:28Speaker 01

  Yeah.

• 49:28Speaker 00

  When does it stop becoming, oh, you're just an API call and you're just a wrapper on API to becoming what you call a product, right? Like, what is that layer? Like, what, like, you know, obviously more lines of code, but like beyond that, what is the substance that people add that qualifies it to be something more?

• 49:46Speaker 01

  I think there's a little bit of like a selection effect of like a lot of companies who do want to get deeper into that level are probably building something that's more differentiated. And I think an example is like we with LM and friends originally we worked with companies that were building their own post-training frameworks or they were ramp actually early in the day was training their own tokenizer and like swapping out the tokenizer in Llama and whatnot. I'm not saying that's that was successful in that case. A better example is like, like let's say Suno, because Suno does not use model for training.

• 50:26Speaker 00

  Mikey on the pod, yeah.

• 50:27Speaker 01

  But they they use model for all their inference and that's because they have like a custom, they have completely custom model architecture and that means that they have to be at the code level and tweak things that are not, you know, it's an API.

• 50:41Speaker 00

  It's interesting as well, like we had Ethan most recently on the XAI Grok team make a prediction that actually like the next tier in video gen is not a better video model, it's a better model or agent that orchestrates video models.

• 50:56Speaker 02

  Language model backbone that can use tools and write code.

• 51:00Speaker 00

  Like, yes, I can make my six-second video or my 10-second video from Grok, but actually I want my six-minute video and I'm not going there through normal video gen.

• 51:09Speaker 01

  Yeah, that's interesting. I actually, so we have GPU sandboxes and recently have seen a few companies doing sort of agents that do video manipulation or.

• 51:22Speaker 00

  Yeah, give it FFmpeg.

• 51:23Speaker 01

  Run FFmpeg.

• 51:24Speaker 00

  But like that's not enough. You need to give it Adobe.

• 51:27Speaker 01

  Yeah, I hadn't put it together with like, this would actually be a video production thing. And in my mind, these things were going more towards editing and.

• 51:36Speaker 00

  Yeah, yeah, I think about this a lot, obviously.

• 51:41Speaker 01

  Luma, Luma agent is a version of this for video production, but you know, it's a one-off.

• 51:46Speaker 00

  I was gonna get your quick takes on some other stuff that happens in recent news and see if you have anything interesting. GitPod, very like somewhat like, you know, different market, they're in like sort of the CICD market, but actually technically very impressive. I don't know if you've like taken a real look at them.

• 52:03Speaker 01

  Yeah, we've, people on our team have talked to the GitPod team and they've, they're technically very strong.

• 52:10Speaker 00

  Yeah.

• 52:10Speaker 01

  I actually, we're very bullish and model on the CI market as well because as there's more agents, coding agents, they're gonna run a lot more CI and the primitives there can be much better.

• 52:23Speaker 00

  I think there's a lot of wasted CI.

• 52:25Speaker 01

  Yeah.

• 52:25Speaker 00

  So is it just like, let's filter? Like, what is the highest order bit here in improving CI for agents?

• 52:32Speaker 01

  Well, there's a lot of wasted time in CI on like preparing your artifacts and like, you know, getting you to the, basically preparing your dependencies and whatnot. And obviously like built systems help with that, but like if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently?

• 52:55Speaker 00

  Oh, okay, okay, okay. Interesting. Yeah. I mean, another form of like, you know, on-demand compute?

• 53:02Speaker 01

  Yeah, exactly. It needs the same again platform.

• 53:06Speaker 00

  Yeah. So for those who don't know, GitPod rebranded to ONA. It was like this whole thing. I actually, I like sort of semi sounded the alarm in cognition. I was like, you should take these guys seriously because they're in front very good.

• 53:17Speaker 01

  Yeah.

• 53:18Speaker 00

  And but, you know, and then they joined OpenAI and presumably we'll see Codex Cloud from the ONA team, which I think would be very, very strong. To me, like, teams like that, that can set up the networking and like the secure boundaries for like, and your like agents to have their own cloud each effectively is what you're doing, kind of. And I'm just trying to draw the analogy or the differences. If you have studied them, like what is the philosophical difference, you know?

• 53:47Speaker 01

  My sense is maybe they didn't go after the right market at the right time because we I guess also got lucky with like agent to give cases really taking off and um dealing like a more of like a sandbox shaped thing than like uh uh mindset is yeah.

• 54:06Speaker 00

  Really sandbox is like CSD is sandbox.

• 54:09Speaker 01

  Yeah.

• 54:10Speaker 00

  It's just like build time sandboxes versus runtime sandboxes and actually it turned out runtime was better.

• 54:14Speaker 01

  Right, and the difference there is runtime sandboxes have a different configuration surface of like how you configure images, how you like attach like storage and.

• 54:25Speaker 00

  Yeah, it's fascinating. Other people, Astro, also OpenAI, also like Python tooling ecosystem people. Are you still sort of bullish building on top of Python? Also recently modular also got bought by Qualcomm. Just any of your takes there?

• 54:42Speaker 01

  Yeah, I mean, we we had Python as our first SDK language because that was the language that people did data and ML in. I actually now have go on TypeScript SDKs as well and our runtime is completely language. It isn't Rust, but it's it's not tied to Python by any means. We haven't seen anything with like inference and training stuff. People are still very Python and the interesting thing with like the agent stuff is people use our TypeScript SDK a lot more because they're not actually doing anything that needs ML. I don't think we'll have to go beyond that super soon because Python and TypeScript is still dominant.

• 55:19Speaker 00

  The last two languages in the world.

• 55:21Speaker 01

  Yeah.

• 55:21Speaker 00

  That's it?

• 55:22Speaker 01

  Well, English and prompting.

• 55:24Speaker 00

  English and prompting. I occasionally talk to people who try to build new languages. They're like, even what's it say, it's Brett Taylor, who's chairman of OpenAI, was like, we need a new language for LLM. So no one has come across one. And I keep looking, you know, Python and TypeScript are, you have a lot of data plus, but then also they are very imperfect as just as languages themselves. Then my close is, I think Model used to be a big bet on developer experience, and you've pivoted the team to agent experience. Is it like the way now, like, do you do, can entire companies and unicorns, multi-unicorns be built on just having better agent experience? Do you need something else?

• 56:05Speaker 01

  It's a big part of our identity. It's not just, you know, like the very tactical, how does an agent use the CLI? But it's also how easy is it to spin something up? Like, what is your iteration time when you want to spin up a new service? And you want to get something going in prod? In practice, that matters a lot to people. And I think it will continue to matter. Like, people are building stuff even faster. And if you give them ways to do it quickly and not have overhead, then.

• 56:36Speaker 00

  I think the debate for me has been, do you do anything differently? That is like very fundamentally different for developer experience versus agent experience? You seem to be on the side of they're like this, they're like.

• 56:48Speaker 01

  Yeah, we actually also have a blog post on that.

• 56:49Speaker 00

  Cosine similarity on like 0.9 or whatever.

• 56:53Speaker 01

  Yeah, I mean, pretty much it's the, the main shift for us has been, as I said, like we built this benchmark, Model Bench, to see where agents are lacking. And yeah, actually literally add surface areas to a product if they're reaching for something. Like, maybe this should be a CLI.

• 57:09Speaker 00

  They hallucinate, yeah, they hallucinate their own features.

• 57:11Speaker 01

  Yeah, and sometimes it makes sense. Like, if they're reaching for this thing, it's product feedback, like, give it to them. And then, yeah, actually moving, we used to only have like logs and metrics in our UI. Just moving all those things to CLI as well, so they're accessible in that form.

• 57:26Speaker 00

  Simple as that?

• 57:28Speaker 00

  Cool, thank you so much.

• 57:29Speaker 01

  Yeah, thank you.

• 57:30Speaker 00

  This is fun. It's a great update, and I can see why you guys have succeeded so much. It is really focused, but also really good execution.

• 57:39Speaker 01

  Thanks, I mean, we have a long way to go.

• 57:41Speaker 00

  All right, thank you.

• 57:42Speaker 01

  Cool.