Everything you need to run Mission Critical Inference (ft. DeepSeek v3 + SGLang)

Sponsorships and applications https://apply.ai.engineer/ for the AI Engineer Summit in NYC https://www.latent.space/p/2025-summit are live! (Speaker CFPs have closed https://x.com/swyx/status/1880427334148452398) If you are building AI agents or leading teams of AI Engineers, this will be the single highest-signal conference of the year for you. Right after Christmas, the Chinese Whale Bros ended 2024 by dropping the last big model launch of the year: DeepSeek v3 https://buttondown.com/ainews/archive/ainews-deepseek-v3-671b-finegrained-moe-trained/. Right now on LM Arena, DeepSeek v3 has a score of 1319, right under the full o1 model, Gemini 2, and 4o latest. This makes it the best open weights model in the world in January 2025. There has been a big recent trend in Chinese labs releasing very large open weights models, with TenCent releasing Hunyuan-Large in November https://buttondown.com/ainews/archive/ainews-tencents-hunyuan-large-claims-to-beat/ and Hailuo releasing MiniMax-Text https://www.reddit.com/r/LocalLLaMA/comments/1i1a88y/minimaxtext01_a_powerful_new_moe_language_model/ this week, both over 400B in size. However these extra-large language models are very difficult to serve. Baseten was the first of the Inference neocloud startups https://www.latent.space/p/gpu-bubble to get DeepSeek V3 online, because of their H200 clusters, their close collaboration with the DeepSeek team and early support of SGLang https://github.com/sgl-project/sglang, a relatively new VLLM alternative that is also used at frontier labs like X.ai. Each H200 has 141 GB of VRAM with 4.8 TB per second of bandwidth, meaning that you can use 8 H200's in a node to inference DeepSeek v3 in FP8, taking into account KV Cache needs. We have been close to Baseten since Sarah Guo introduced Amir Haghighat to swyx, and they supported the very first Latent Space Demo Day https://www.latent.space/p/jan-2023-update?open=false#%C2%A7generative-ai-hackathon-sf-feb in San Francisco, which was effectively the trial run for swyx and Alessio to work together! Since then, Philip Kiely https://x.com/philip_kiely/status/1872408515182207394 also led a well attended workshop on TensorRT LLM at the 2024 World's Fair. We worked with him to get two of their best representatives, Amir and Lead Model Performance Engineer Yineng Zhang, to discuss DeepSeek, SGLang, and everything they have learned running Mission Critical Inference workloads at scale for some of the largest AI products in the world. The Three Pillars of Mission Critical Inference We initially planned to focus the conversation on SGLang, but Amir and Yineng were quick to correct us that the choice of inference framework is only the simplest, first choice of 3 things you need for production inference at scale: “I think it takes three things, and each of them individually is necessary but not sufficient: * Performance at the model level: how fast are you running this one model running on a single GPU, let's say. The framework that you use there can, can matter. The techniques that you use there can matter. The MLA technique, for example, that Yineng mentioned, or the CUDA kernels that are being used. But there's also techniques being used at a higher level, things like speculative decoding with draft models or with Medusa heads. And these are implemented in the different frameworks, or you can even implement it yourself, but they're not necessarily tied to a single framework. But using speculative decoding gets you massive upside when it comes to being able to handle high throughput. But that's not enough. Invariably, that one model running on a single GPU, let's say, is going to get too much traffic that it cannot handle. * Horizontal scaling at the cluster/region level: And at that point, you need to horizontally scale it. That's not an ML problem. That's not a PyTorch problem. That's an infrastructure problem. How quickly do you go from, a single replica of that model to 5, to 10, to 100. And so that's the second, that's the second pillar that is necessary for running these machine critical inference workloads. And what does it take to do that? It takes, some people are like, Oh, You just need Kubernetes and Kubernetes has an autoscaler and that just works. That doesn't work for, for these kinds of mission critical inference workloads. And you end up catching yourself wanting to bit by bit to rebuild those infrastructure pieces from scratch. This has been our experience. * And then going even a layer beyond that, Kubernetes runs in a single. cluster. It's a single cluster. It's a single region tied to a single region. And when it comes to inference workloads and needing GPUs more and more, you know, we're seeing this that you cannot meet the demand inside of a single region. A single cloud's a single region. In other words, a single model might want to horizontally scale up to 200 replicas, each of which is, let's say, 2H100s or 4H100s or even a full node, you run into limits of the capacity inside of that one region. And what we had to build to get around that was the ability to have a single model have replicas across different regions. So, you know, there are models on Baseten today that have 50 replicas in GCP East and, 80 replicas in AWS West and Oracle in London, etc. * Developer experience for Compound AI Systems: The final one is wrapping the power of the first two pillars in a very good developer experience to be able to afford certain workflows like the ones that I mentioned, around multi step, multi model inference workloads https://www.baseten.co/blog/baseten-chains-explained/, because more and more we're seeing that the market is moving towards those that the needs are generally in these sort of more complex workflows. We think they said it very well. Show Notes * Amir Haghighat https://www.linkedin.com/in/amirhaghighat/, Co-Founder, Baseten * Yineng Zhang https://www.linkedin.com/in/zhyncs/, Lead Software Engineer, Model Performance, Baseten Full YouTube Episode Please like and subscribe https://youtu.be/KjH7Gl0_pq0! Timestamps * 00:00 Introduction and Latest AI Model Launch * 00:11 DeepSeek v3: Specifications and Achievements * 03:10 Latent Space Podcast: Special Guests Introduction * 04:12 DeepSeek v3: Technical Insights * 11:14 Quantization and Model Performance * 16:19 MOE Models: Trends and Challenges * 18:53 Baseten's Inference Service and Pricing * 31:13 Optimization for DeepSeek * 31:45 Three Pillars of Mission Critical Inference Workloads * 32:39 Scaling Beyond Single GPU * 33:09 Challenges with Kubernetes and Infrastructure * 33:40 Multi-Region Scaling Solutions * 35:34 SG Lang: A New Framework * 38:52 Key Techniques Behind SG Lang * 48:27 Speculative Decoding and Performance * 49:54 Future of Fine-Tuning and RLHF * 01:00:28 Baseten's V3 and Industry Trends Baseten’s previous TensorRT LLM workshop: This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Welcome back. Right after Christmas, the Chinese whalebros ended 2024 by dropping the last big model launch of the year: DeepSeek V3. This is a massive 671 billion parameter, fine-grained MOE model with 256 experts, trained with native FP8 mixed precision training, multi-head latent attention from DeepSeek V2, a new multi-token prediction objective, and 15 trillion tokens of data, including synthetic reasoning data distilled from DeepSeek R1. Right now, on the LM Arena leaderboard, DeepSeek V3 is rated the seventh-best model

• 0:41Speaker 00

  in the world with a score of 1319, right under the full O1 model Gemini 2 and for our latest and-above-O1 mini Grok 2, Gemini 1.5 Pro, and Claude 3.5 Sonnet. This makes it the best open weights model in the world in January 2025. There has been a big recent trend in Chinese labs, releasing very large open weights models, with Tencent releasing Hunyuan Large in November and Hailuo releasing Minimax Text this January, both over 400 billion size. However, these extra-large language models are very difficult to serve. Base10 was the first of the inference neo-cloud startups to get DeepSeek V3 online because

• 1:26Speaker 00

  of their H200 clusters, their close collaboration with the DeepSeek team, and early support of SG lang, a new VLLM alternative out of UC Berkeley that is also used at frontier labs like XAI. Each H200 has 141GB of VRAM with 4.8TB/s of bandwidth, meaning that you can use eight H200s in a node to inference DeepSeek V3 in FP8, taking into account KV cache needs. We have been close to Base10 since Sarah Guo introduced a mere haggy hat to SWICS, and they supported the very first Latent Space Demo Day in San Francisco, which was

• 2:08Speaker 00

  effectively the trial run for the podcast you're listening to right now. Since then, Philip Kiely has also led a well-attended workshop on TensorRT LLM at the 2024 World's Fair. We worked with him to get two of their best representatives, Amir and lead model performance engineer Yineng Zhang, to discuss DeepSeek, SG Lang and everything they have learned running mission-critical inference workloads at scale for some of the largest AI products in the world. Spoiler: Amir thinks there are three pillars of mission-critical inference workloads, and we spend quite some time discussing what you need for each of them. In other news, invites are now rolling out for the second AI engineer summit

• 2:53Speaker 00

  in New York City from February 20th to 22nd. We are bringing back the surprisingly successful AI leadership track from World's Fair, and the AI engineering track is now wholly focused on agents at work. If you are building agents in 2025, this is the single best conference of the year. We are curating all attendees, and we'll sell out after we announce speakers this coming week from DeepMind, Anthropic, OpenAI, Meta, Jane Street, Bloomberg, BlackRock, LinkedIn, and more. Look for more sponsor and attendee information at apply.ai-engineer and see you there. Watch out and take care.

• 3:40Speaker 01

  Hey, everyone. Welcome back to the Late in Space Podcast, our first recording of 2025. I'm Alessio, partner and CTO at Dacable Partners, and I'm joined by my co-host, Spix, founder of SmallAI. Hey, and today we are here with a special double guest episode with Amir. Oh, my God, I don't know your last hag yet.

• 4:00Speaker 02

  Close enough. That is good. I thought it was first try is good. That's really good.

• 4:05Speaker 01

  And Yineng Zhang from Base10, welcome. Thank you. Amir, we've met before. You're a co-founder of Base10, which is, you know, one of the leading sort of L M inference platforms. I don't know how you, what do you consider yourself? That sounds fine. And Yineng, you are lead software engineer on the model performance team, and you guys recently shipped DeepSeek V3 as one of the many models that you do host. You also are very involved in SG Lang, and that was actually One of the reasons we're discussing an episode with you even before DeepSeek V3 dropped as a Christmas present to everybody. So we can take this in a number of directions, but I think one thing we wanted to just get off the bat on was to start with DeepSeek and maybe and then we'll work our way backwards back to SG like, but DeepSeek is, you know, more kind of more recent.

• 4:50Speaker 01

  Why are people so interested? What's the history of like, I guess, like DeepSeek in general from your perspective?

• 4:56Speaker 03

  Yeah, because DeepSeek V3, I think, is currently considers the leading open source LLMs based on the benchmark results and the chat area results. And it's so big, you know, it's 671 billion parameter M O E. And I, I, I think it's a game changer for the open source AI. So everyone is interested in this model. Yeah.

• 5:23Speaker 01

  One of the interesting things is like they are bootstraps, like, you know, very private, small lab, like they have a lot, a lot less, a few resources than, than others. But it also, it's also interesting like that it's just open weights. Like for some reason, the Chinese labs are much better than the American labs. And sharing open weights and that's obviously beneficial for base 10. It is in your incentive to serve these models at, at all times, like, you know, like what, what are sort of the unique challenges that you face, like offering something this, this large?

• 5:54Speaker 03

  Yeah, I think because the model is very large. And if we use something like use H100, we cannot serve this model. Because you know, even we, we use the H100 eight, eight cards, it should be a 640 gigabytes memory. So DeepSeek V3 model, it has 671 billion weights. So even use the FP8 precision, you need, I think, 71 gigabytes for the weights. And you, you also need an extra memory for the KV cache. So it's not possible to run that on H100. That's why we, we, we chose H200 to run that model or use Marty Node to run that model. Yeah, it's very challenging.

• 6:39Speaker 03

  And I, I think another one is that DeepSeek V3, the, the weights is released. Was the FP eight precision. So if you want to write, you you should support that kernel. Because I I think the default is the BF sixteen and even larger. But if you want to run the FP eight, you need to support the quantization. So I think currently even the tense RTIM doesn't support the FP eight. So if we want to implement that feature, we we should do some feature development for that. The last challenging part is that, yeah, if you want to do some debugging or or some performance benchmarking, it's very hard. Why?

• 7:24Speaker 03

  Because the model is so large and the loading time is so long. Yeah, so it make it more complicated for the developer to do some debug.

• 7:36Speaker 01

  Is it complicated or just slow? I mean, you you've just only mentioned loading time, but like.

• 7:41Speaker 03

  Yeah, loading time is slow. Yeah, you are right.

• 7:43Speaker 01

  It's not complicated. It's just, you know, take just go for more coffee.

• 7:46Speaker 03

  Okay, okay.

• 7:46Speaker 01

  Okay.

• 7:48Alessio Fanelli?

  Can you maybe just give people a quick rundown of all the models you support on base ten? How it compares just on size, just to, you know, people here six hundred seventy one gigabytes. But like, is that a lot more than other models? And then you mentioned the BF sixteen FPA. What's kind of like the usual that you see? And do you see any variation based on model size or anything like that?

• 8:10Speaker 03

  I think at the base ten, something like Lama seventy B is more common. I think Lama three has released the four hundred zero five billion weights. But I think there are just a few users use that. So before the Deep Secret V three, I think we haven't encountered that issue for that so large weights. I think Deep Secret V three is the first one so big model that we should use H two hundred or use H one hundred multi nodes.

• 8:43Alessio Fanelli?

  And was that because of performance or why did people not use the four or five B Lama? I I think the. You know, what I hear from people is like, you know, the performance gains of like the four or five B at inference times are like not worth it, you know? So the 70B is kind of the sweet spot. Who are the people that use V3? Are people that were using maybe the Lama 70B model and just want better performance? Are people that are just experimenting? I think that's kind of the question that people always have. There's always a lot of excitement around open source models, but then maybe the question is like, what are they really good for?

• 9:15Speaker 02

  I can answer this just observationally. The interest that we have seen, and some of this is running in production, some of this just at the interest level, is generally not coming from folks who are trying to upgrade from a certain open source model to DeepSeek V3. It's we're seeing it generally speaking from folks who are coming from Claude and are doing so either because, and here's, I'm going to give you a list of reasons, and generally the reasons are a combination, a certain combination of these. In no particular order, either it is they're being rate limited, or the price is too high, or they have certain latency requirements or time to first token requirements for their

• 9:59Speaker 02

  use case that Claude cannot hit, or they want to have full control over the model as opposed to running it behind an API where the model underneath them can potentially change. And a couple of other reasons, but generally it's a combination of those.

• 10:17Alessio Fanelli?

  You mentioned the speed and some of these things. Do customers want to change the hardware? Also, like you're referring the H2 under is kind of like the default thing. Like, do people come to you and it's like, hey, I'd rather use a smaller system and can I have worse performance? Or how do you work with customers on that?

• 10:32Speaker 02

  Generally, people come with certain requirements around the latency, the throughput, the cost. Generally, they're not coming in saying, I want this particular GPU SKU, at least like as we go up market and we're talking to, you know, foundation model companies, for instance. The things that are top of mind for them are those are those requirements, not a particular GPU SKU. We're doing the different GPU GPU SKUs not because we wanna, you know, offer, oh, look, we have H200s, look at us, we have Meg the H100s, look at us. It's not that. It's really because those are the tools to achieve a certain kind of time to first token for certain types of models

• 11:16Speaker 02

  or certain kind of throughput and scale or a certain kind of price per million token or what have you per million images, depending on the modality. And that's the reason that, you know, we're talking GPU SKUs.

• 11:30Speaker 01

  I wanted to pick up a little bit on this FP8 thing. It seems like, you know, I think Noam Chazir started talking about sort of training natively quantized. And I think that's what DeepSeek seems to have done, at least they said in their paper. Is this a trend? Like, is the community settling on one form, one sort of numeric that everyone knows about? Tell us more about what you're seeing here in terms of like the training trends, those sort of model trends and the, I guess. So I think a lot of companies as well, like Together, they'll also release like quantized versions of the Llama models, right? Like for like turbo or light inference, or you know, like just based on different levels of speed. Like, do you do anything there in terms of quantizing the models that you serve?

• 12:13Speaker 02

  I'll let Yannick answer the sort of patterns around, you know, using FP8 in training. But I want to draw one distinction that I think gets to the latter part of your question, Sean, which is that unlike companies like, you know, you mentioned Together, or, you know, Fireworks or Replicate, Bostin doesn't provide a shared inference endpoint for the popular open source models. That is a product that we don't have on purpose. Those work really well, the shared open source, shared inference endpoints for open source models work really well for situations where the user is saying, hey, let me just call a certain popular model behind an API paid by the token. That is not our average customer or the median customer.

• 12:57Speaker 02

  Our customers generally have their own custom models, very custom workflows, strict requirements around latency and, you know, time to first token, and can't deal with Neighbour problems that like, oh, the the API is slow because some other customer has been calling it a lot. Other requirements around infrastructure flexibility, around around regions for for latency reasons, for compliance reasons. That's the side of the inference market that that we capture. And at base ten, when you deploy a model, whether it's your own custom weights or or or an open source model, you get dedicated inference, dedicated resources, dedicated inference. And so when it comes to the quantization question, where it matters is that we would never quantize the model behind,

• 13:42Speaker 02

  you know, the users back and say, look at us, there's a faster Llama seventy B that has been, you know, somehow quantized to to be faster and cheaper. Our customers are coming to us with those requirements that I mentioned. But in particular, when it comes to model quality, they have strict requirements. They would not be okay with us touching the weights, if you will. You know, we have done things like speculative decoding with a couple of different ways. But but all of those things, those methods guarantee the output is unchanged as opposed to quantization. So when it comes to quantization, we have built tooling that allows our users to quantize their models for the ones that we're working more hands on through our forward deployment engineering team.

• 14:24Speaker 02

  We're working with them on on evals as well to ensure that the quantized models are meeting their their requirements. However, this is all very much in conjunction with the engineers that are customers as opposed to us doing it behind the scenes.

• 14:38Speaker 03

  Oh, yeah, FPA training is very interesting. And I think the DeepSeek team is the first one to use FPA training for the large model. I think before that, maybe Lin Yi dot AI, sorry, zero one dot AI. They they use the E lightning, they use the FPA training. And others, I I think most of them use the BF sixteen training. And yeah, it it is the game changer. And for us, because the FP8 kernel should be implemented inference, it used the blockwise FP8. And currently, even you use something like Kuda or Kublas, you cannot support that.

• 15:20Speaker 03

  So you you should use something like Triton to implement the the kernel, or you should use something like a Catlas to implement that kernel. So I I think that's the challenging part.

• 15:31Speaker 01

  My theory is that this will pick up in terms of the models that people release, like increasingly, it will not be uh uh BF16. There was a bit of quantization. I was trying to look for the paper while you were speaking, but I couldn't find it. But there's a there's sort of this like ablation of quantizations paper that was out last year that showed that there is benefits to quantizing and sort of negatively trading all the way until like like six bit. And then like even smaller than that, it may be going too far. Yeah, I'm not sure if you you know what paper I'm talking about, but yeah, and there's there's an interesting trend for sure.

• 16:04Speaker 03

  Yeah, I I I think even they use the FP8 quantization, the benchmark result is very good, such as something like GSM8K. The score is nearly ninety four point six. It's so high, you know. I I think it's higher than every other open source L M, even the Lama four hundred zero five billion.

• 16:26Speaker 01

  So I'm gonna move on a little bit in terms of like one other notable detail. And then, you know, we don't have to speak too much about D C, because obviously, we don't know that much unless unless we work on the team. But like, you know, another trend that they have is the fine-grained M O E. I think that there is this question about whether or not M O Es will be more of a thing. Basically, like like this time last year, Mixtral was sort of kicking off a bit of an M O E trend with, you know, eight by seven B, eight by twenty two B. And then the rest of the year, not no M O Es, basically. So is this like discovery of fine green M O E is going to be a relevant trend for this year?

• 17:06Speaker 03

  Yeah, I I I think so, because as far as I know, some companies such as Baidu or Baidu Dance, their internal dominant A O M, they use the M O E architecture. And their ways, I think it's similar to the DeepSeek M O E model. So I I think after this new year, the M O E inference optimization will be very essential and yeah, yeah, so important.

• 17:32Speaker 01

  At the same time, like, why hasn't, you know, the the big labs done, why haven't the big labs done it, right? Like, I I think, well, Lama four or five B is dense. I think Grok is also a dense model, that you can correct me if I'm wrong. And yeah, it's it's it's just a weird counter trend, I think. This time last year, I was like kind of writing my recap and I was like, all right, M O E is like seem like they're going to be trending and then they did not trend anyway. So it's just just a note that I would flag out there, but I generally agree, like, it seems like fine green M O E is working out and I would definitely see, definitely want to see more people adopting it. I went to Jeff Dean's session in Europe's and he also mentioned as well that I think one of the Gemini, I think Gemini F one point five Pro is an M O E, which I don't think we knew before that.

• 18:16Speaker 03

  Yeah, yeah, so the reason why Lama open source M O E model, because I I think they they tried to train our M O E model, but they failed. So that that's why they didn't open source M O E model for Lama series.

• 18:30Speaker 01

  Why are the causes of failure? Like, why do M O E's fail? Like, I think this is another thing that people are talking about, right? Like, the the failures of three point five Opus, the failures of G P C five, like, you know, it's a it's a thing that people are sort of rumoring.

• 18:41Speaker 03

  Yeah, because if you want to train model for the training stuff, you need to some some benchmark or some, yeah, score. But for the M O E model, the benchmark score is even lower than the dance model. So in that in that way or in that case, they think of M O E model is worse than the dance model. So they didn't release that M O E model.

• 19:04Speaker 01

  Okay, well, then one more thing, I guess, maybe more commercially relevant. DeepSeek's A P I pricing is very competitive. How do you decide pricing in this kind of landscape with open models?

• 19:15Speaker 02

  Yeah, so it goes back to the use cases that we serve. And again, going back to the fact that we don't have shared inference endpoints for the different models. And so our pricing is never per token. Customers, like I said, generally come with their own custom models or open source models, but with strict requirements around certain things. Latency and time to first token requirements, or security and compliance requirements, or a particular scale that they're looking for without running into noisy neighbor problems, things like that. And the way that we price has always been based on consumption, based on consumption of resources. And that takes one of two shapes.

• 19:56Speaker 02

  One is the shape where things are running inside of our infrastructure. And we're running, by the way, on top of multiple different public clouds, many different regions within those. Then we charge them based on the hardware, the resources that they're using. The second shape that it takes is that our customer brings their own cloud. And we're seeing this more and more, where a customer has big committed resources inside of their AWS, VPC, or GCP, or what have you. And in that world, we also have a consumption model. Of course, the price is very different because they're using their own resources, but we are managing those resources for them.

• 20:38Speaker 02

  An example of that, that we're seeing more recently is the fact that we've had to build multi-cloud capabilities for us so that we can have a single model horizontally replicate across different regions and even different clouds. And more and more, as we go up market, our customers have their own cloud commits. They are also multi-cloud in order to be able to get good prices and good capacity. And it's unreasonable to expect every one of them to build the same multi-cloud capabilities that we have built. And so then they take advantage of what we have built and use all of the different cloud resources that they have as a whole holistic unit and have their models at inference time horizontally scale across those

• 21:23Speaker 02

  and even optionally overflow to our cloud when they start running out of committed resources. All of that has a consumption pricing model to it.

• 21:31Alessio Fanelli?

  Can we talk about what it takes to actually run your service? So we had episodes with, you know, replicated model of artworks. We always like to ask this question, obviously, since you're not the model maker, all the secret sauce is in how you actually run the model. I know you also have Trust, which is your more developer-led SDK. Can you maybe quickly run people through, how do you go from taking the DeepSeek V3 weights to like actually run it? What goes on behind the scenes? And then we can talk about SG Lang in-depth a little more.

• 22:01Speaker 02

  Yeah, totally. So we have, like you said, we have Trust, which is our open source model packaging and deployment library. Trust works with different frameworks underneath it, has very native and deep support for Tensor R T L M. Somewhat as an accident of history, we happen to have access to T R T L M before it was announced, contributed back to it. And we still do, pushed it to its limits and had to go beyond it in certain areas as well. So for example, you know, the Triton inference server, we've had to build our own version of that for for performance and reliability reasons. But we invested in it heavily because it tended to be, for the use cases that we were seeing from our customers,

• 22:45Speaker 02

  it tended to be the best framework to handle the latency and throughput requirements that we were seeing. In particular, when it comes to the kernels that they come with, I'm yet to see folks do better than what Nvidia can do when it comes to CUDA kernels. However, Trust is not tied to Tensor R T L M. For example, for the DeepSeek example that you mentioned, it's working with SG Lang, which is really cool to see. And we will be investing more and more in SG Lang, especially as the developer experience is just so much better than than Tensor R T L M. We've built a lot around Tensor R T L M, productized them too, to make it easier to work with. But still, SG Lang has been a joy to work with.

• 23:26Speaker 02

  Another trend that is really promising, and I learned this from the SG Lang folks, is that the Tensor R T L M folks have promised to modularize a lot of T R T L M so that other frameworks like Case G lang can grab certain parts of it and build on top of it. And so as a user, you don't have to go all in on one framework versus another. You can really pick and choose based on the requirements that that you have. And that's really been our approach as well. We have customers on on based in that are using tensor T L and we have ones that are using V L. And we have a growing number that are using this G lang two. It's not about really tying yourself to one versus another.

• 24:09Speaker 02

  It's about using the the best of of the bunch, depending on the requirements of the customer and for for their inference workloads.

• 24:18Alessio Fanelli?

  How did you think about designing the framework? So replicate also a cog, uh, which was kind of more tied to to Docker. What were maybe some of the design decisions that you had? And how do you think that that's changing, especially as the models change and like the runtimes change?

• 24:33Speaker 02

  Yeah, totally. So so we, you know, started trust, gosh, like four or five years ago. And at the time, the the sort of principle that we had in mind was let's make sure that easy things are easy, but hard things are possible. And so an example of easy things being easy is that, you know, think of it as a very simple, you know, you have a model. What do you what do you need to do to serve it? Well, you need to load it up and then you need to write the code for the for the inference path. And, you know, trust actually had, you know, hooks for these two things. And so you could just, you know, write two functions and voila, your your model was was was being served at least as a single unit. We can talk about the horizontal scaling part separately. That's a whole different topic.

• 25:16Speaker 02

  And so we did well when it comes to easy things being easy. I think we struggled with with the hard things being possible in the early days. Hard things, example of hard things are are cases where seeing where more and more our customers have their own custom models. Custom models that sometimes they've they've fine tuned, sometimes they've pre trained. You know, we we now have six or seven foundation model companies as customers who are sophisticated enough to pre train their their their own models. And they're trusting us with the inference layer. That's not a situation of, hey, here's two functions. Good luck. Have to have much deeper integration. With them. And so that's where we started rethinking some of the abstractions of of trust over time to allow for for those custom use cases.

• 26:00Speaker 02

  And and that has been successful. Another place where where we didn't think about at first, but became important, was seeing more and more use cases where the customer was saying, I can serve my models on base ten using trust. Fine, but my use case is not just call the model, get the response and run with it. I actually have a multi-step inference workload. So an example of that is the company Bland AI with their AI phone calls. To make an AI phone call happen, you need to transcribe what the human said, a couple of L L M calls to figure out what to say back, and then text to speech to to actually have the end to end work for working. Now, you can have these these three separate models, three separate deployments, but think about what happens is that you have to call the first model,

• 26:45Speaker 02

  wait for the response, call the second model, get wait for the response. All of that network back and forth is killing you. The latency is is becoming too high. That's not something that that we had designed for initially. And so that's when we came out with Trust Chains, which is the DevX for building these multi-step, multi-model inference workloads, but doing so in a very low latency way. So that instead of you orchestrating all of these calls and and incurring all of that network latency, you're actually making one call and these models are actually talking to each other. They're run independently on their own hardware, on their with their own auto scaling behavior. But the data from one to the next step is being actually streamed.

• 27:28Speaker 02

  And that way, going back to the AI phone call use case, you can get sub four hundred millisecond latency AI phone calls that actually feel very realistic.

• 27:36Alessio Fanelli?

  And those are all models hosted on base ten? Or do you also do chains?

• 27:40Speaker 02

  Those happen, those have to be models hosted on base. And if one of those steps is not hosted on base ten, then you're still incurring a massive latency on on the network side.

• 27:50Alessio Fanelli?

  Yeah. And then just to maybe tie this into SG lang, how do you kind of think about? The hidden magic, you know, should people know they use SG lang? Like, should people care? Like, especially for the people building the models, you know? Like, does it matter to them? They use a certain model runtime? Or do they not care? Everything just goes through the base end platform the same?

• 28:10Speaker 02

  Yeah. Should we talk about it? Yes, 100%. We want to be the transparent provider. I don't want to say, oh, just give us your model and voila, magic and trust our magic. I want that magic to be very transparent to our customers. That has worked really well for us. And you really need that, especially when you're onboarding, you know, foundation model companies. And, you know, they're not gonna, you know, just turn a blind eye on how things are run underneath the hood. When it comes to customers caring about what's happening underneath, they do. But more than caring about this framework versus that, they care about how the final output. In other words, is the quality the same? Or somehow something has changed underneath the hood?

• 28:55Speaker 02

  And the model isn't actually producing the same quality? How is the latency? And especially for certain use cases, what is the time to first token? And is that sustained? What is the P95 of that? What is the P99 of that? How well does it handle throughput? When you start getting a massive burst of traffic, does it still sustain those P95s and P99s? How do I make sure that the security of the data being sent into the model is guaranteed? How do I make sure compliance is guaranteed for HIPAA use cases? And how do I make sure that the data remains within a certain geo for compliance reasons? Or for latency reasons?

• 29:38Speaker 02

  And so those are the things that, those are the concerns that the customers are coming to us with. Let's so about, hey, here's my model. Make sure you run it with TRTLM or make sure we run with SG lang.

• 29:49Alessio Fanelli?

  Can you maybe give us an overview of all the different frameworks that people use? So you have SGLANG, TRTLLM, VLLM. Those are kind of like maybe the open source research ones. And then some of the other commercial companies are building some of their own stuff. But what's the state of the art today? Maybe like the top three most popular? And then we can talk about why SGLANG came to be and what makes it different? And some of the performance boosts that you get.

• 30:15Speaker 03

  Okay, yeah. I think for the common use case, maybe not not as a DeepSeek fee three. For the common use case, I think SGLANG's performance is better than FOM. And its usability is better than TansarTM. So when user care about the performance and the usability or the, I think they will choose SGLANG. And for the DeepSeek fee three case, because we do a lot of optimization in SGLANG. Something like DeepSeek fee two, they proposed Attention Parent named MLA, Multi-Latent Attention. And I think SGLANG is the only framework supported that. Maybe LightLM and TRTM also support, but FOM doesn't support.

• 30:58Speaker 03

  And also in DeepSeek, oh, sorry, in the SGLANG version 0.4, we also support the DP Attention for DeepSeek. And in the latest SGLANG release, we also support the Blockwise FP8 Kernel. And that Kernel was adopted and copied by FOM later. So I, I think we have done a lot of optimization for DeepSeek. That's why SGLANG is the recommended engine by the DeepSeek team.

• 31:30Speaker 02

  And maybe one thing to point out, and I think this is, this is important, is that the framework that you choose is part of the equation for running mission-critical inference workloads, but it's only a part of it. So maybe I can draw this out just based on the experience, based on what I've seen in the market as to what it takes to run mission-critical inference workloads in production. I think it takes three things, and each of them individually is necessary but not sufficient. One is performance at the model level. So in this case, how fast are you running this one model running on a single GPU? Let's say the framework that you use there can can matter. The techniques that you use there can matter.

• 32:14Speaker 02

  The MLA technique, for example, that that Eneg mentioned, or the CUDA kernels that are being used. But there's also techniques being used at a higher level. Things like speculative decoding with draft models or with Medusa heads. And and these are implemented in the different frameworks, or you can even implement it yourself. But they're not necessarily tied to a single framework. But using speculative decoding gets you massive upside when it comes to being able to handle high throughput. But that's not enough. Invariably, that one model running on a single GPU, let's say, is going to get too much traffic that it cannot handle. And at that point, you need to horizontally scale it. That's not an ML problem. That's not a PyTorch problem. That's an infrastructure problem.

• 33:00Speaker 02

  How quickly do you go from a single replica of that model to five, to 10, to 100? And so that's the second that's the second pillar that is necessary for running these mission critical inference workloads. And what does it take to do that? It takes, as some people are like, oh, you just need Kubernetes. And Kubernetes has an auto scaler and that that just works. That doesn't work for for these kinds of mission critical inference workloads. And you end up catching yourself wanting to bit by bit rebuild those infrastructure pieces from scratch. This has been our experience. And then going even a layer beyond that. Kubernetes runs in a single cluster. It's a single cluster.

• 33:44Speaker 02

  It's a single region tied to a single region. And when it comes to inference workloads and needing GPUs, more and more, you know, we're seeing this.

• 33:54Speaker 02

  That you cannot meet the demand inside of a single region, a single cloud's single region. In other words, a single model might want to horizontally scale up to 200 replicas, each of which is, let's say, two H100s or four H100s or even a full node. You run into limits of the capacity inside of that one region. And what we had to build to get around that was the ability to have a single model have replicas across different regions. So, you know, there are models on Base10 today that have 50 replicas in GCP East and 80 replicas in AWS West and Oracle in London, et cetera. And that was a big investment that we had to make.

• 34:39Speaker 02

  The final one is wrapping the power of the first two pillars in a very good developer experience, to be able to afford certain workflows like the ones that I mentioned around, you know, multi-step, you know, multi-model inference workflows, because more and more we're seeing that the market is moving towards those, that the needs are generally in these sort of more complex workflows. So these are the three pillars that it takes to run mission critical inference workflows. And the choice of the framework, the serving framework, is really a part of the first pillar. And that's something that I'm seeing the market of, like, people who are somewhat new to it, they're like, well, VLM equals equals production. That's what it takes to run inference workflows.

• 35:23Speaker 02

  And that's in practice, that is not true, and I wanted to call that out.

• 35:26Speaker 03

  I agree with Amir, because I think it's open source libraries such as VLM, SGLang, LattlM, or TensorRTM, they only provide a library, they don't provide a productive solution.

• 35:38Alessio Fanelli?

  Yeah, can we maybe talk about some of the SGLang unique things? So I read through the paper, sounds like some of the main use cases, like when you have very large batches, which makes sense for your use case, and also kind of longer, longer contexts. You know, what was the decision by creating the framework, which I think is like Around one year old, I think the paper came out December twenty twenty three, something like that, so it's still

• 36:01Speaker 03

  Yeah, yeah, yeah.

• 36:02Alessio Fanelli?

  Still fairly new compared to some of the the other ones. And then maybe what were some things that you had to change, you know, a- as you built it or any any fun stories?

• 36:10Speaker 03

  Yeah, yeah, yeah. I think last year, oh, not not last year, sorry. At twenty twenty three, maybe August, at that time, Liangming and Yin want to create the S J L A N, maybe for the front end. Something like L L M program, they want to solve that problem. And at twenty twenty four, January, they support something like Redis Cash. It's a prefix caching technology. I think S J L A N is the first framework that support prefix cache. And at February, they also support a concentrated decoding and support some jump forward. So at that time, it's a no for the language generator, not the inference backend.

• 36:53Speaker 03

  And at twenty twenty four, July or Ju- June or July, we want to make S J L A N a fully functionality. L M inference engine is just equal with equivalent with the F M or with the Tensor T L M. So at that time, we published a blog compared with other frameworks. And its performance is is amazing, yeah. At that time, I I think its performance is maybe three times, it's rather put it, three times than F M. So after that, F M also do some refactor to make it faster. And at September and December, we continue to release new versions for S J L A N.

• 37:36Speaker 03

  Yeah, we we support some deep seek optimization such as M L A optimization, D P attention optimization. And we also support the Sierra overhead C P U schedule. Also, we we support something like S J L A N router for the cache aware load balance. Yeah, we we deliver so many features, we just build on the ship. And I think why Liangming and Yin want to create a new framework rather than use the existing solutions such as F M or Tensor T. Because at that time, you know, at that time for the F M, I think it's easy to use, but its performance maybe it's not good. Some design, I I think. It's not okay. Yeah, maybe the the code is a little messy.

• 38:21Speaker 03

  And if you want to extend some new feature on top of that, it's a little hard. And Tensart IM, I I think it's blazing fast. Its performance is so good, but it's not easy to do some secondary development. If you want to add some new feature, it's a little hard. So just think about, oh, how can we create a new framework? It can achieve the good performance. Also, it's easy to develop, to maintain. So that's why they they create this SGLAN project.

• 38:52Alessio Fanelli?

  Let's run through maybe the three main techniques behind SGLAN. So the first one is a Raddix Attention, which focuses on KV cache. And when you think about a model that is, you know, as large as DeepSeek V3 especially, like having better KV cache reutilization is great. Can you just talk a bit about that performance impact?

• 39:11Speaker 03

  Yeah, Raddix Cache, I I think it it's the technology of the prefix caching. And it's a special case for something like block size is one. You know, for VIM or for other frameworks, they use something block size thirty two. And SGLAN use the block size one. I think if you use the block size one, you can make the cache heat rate higher than other frameworks. I think that's the mainly benefits.

• 39:39Alessio Fanelli?

  And for your case specifically, how does that change when you have like a base ten type use case where you don't do not have a share endpoint versus like, you know, is this less helpful for GPU clouds to do one model for like many people that have like very different use cases versus like when you have just one endpoint for one customer? I'm sure they have a system prompt that a lot of models share and things like that. Anything you want to mention there?

• 40:04Speaker 02

  Yeah, we've seen this be massively helpful in for the reason that you mentioned. There is a certain sort of finite number of prompts or at least prompt prefixes that are being used per customer. And What we've seen is that previous caching and the different techniques to make that better has been massively helpful. However, we still had to build on top of that. The example there is that you have a model with dozens of replicas, each of which has its own state of KV cache. And your request comes in. And what we used to do back in the day was that that request would be randomly assigned to one of these replicas.

• 40:49Speaker 02

  But the better way to do it is that knowing the state of KV cache in these different replicas, trying to decide which one it should go to. One of the parameters that you need to consider, there are other parameters to consider around the size of the queue at each of the replicas and the location of each replica is depending on how geo where you want to be. But adding that additional consideration around KV cache aware load balancing was something that we saw improve latency quite a bit for our customers.

• 41:23Alessio Fanelli?

  And then the second part, which was maybe the harder one to understand as a practitioner, which is this idea of like turning some of the decoding process into a finite state machine instead of a more open ended, when you're using, especially with structured outputs. Can you maybe explain what that means? And I would love to learn too. So maybe this is an opportunity for everybody to better understand how you think about going from a normal kind of like token by token decoding to having a more, I wouldn't say pre-compiled, but like pre-understanding of what the paths are gonna be.

• 41:57Speaker 03

  I think SGLAN support concentrated decoding and it also support jump forward. And we use something like outline or the X grammar to do the, something like change the convert as a schema from JSON to the FSM, the state machine. And we can use the state machine to control the output. Something like the output maybe to the JSON mode or something like, yeah, it should be obey some rule. So in that case, because the output should obey some rule. So you can skip some token, something like you should decode four times, but you should obey that rule. So, or you can get that token in advance.

• 42:40Speaker 03

  You can just use one preview to replace the four decoding, for example. So that's why you can jump forward.

• 42:48Speaker 01

  I guess the question is like, why doesn't everybody do that? When I was reading, I was like, this just sounds better, especially both for accuracy. You know, you're kind of constraining for structure output as well. You can do faster decoding. Are there downsides to it as well?

• 43:02Speaker 03

  I think maintain jump forward is a little hard. Yeah. At later, we we support something like CPU overlap. I think in the overlap model, we even make it compatible with the jump forwarding. Because just if you want to maintain the jump forward with other features, it will be more complicated. So I think we we only use it default setting. We disable it by default. But if you want to enable it, you can just use some arguments to enable that. But it's a little hard to maintain, especially compatible with other optimization features.

• 43:37Speaker 01

  Just as a side note, you mentioned X Grammar, which I had never heard of about. And I looked up the GitHub repo. It's actually from MLC, which we talked to TQ.

• 43:46Speaker 03

  Yeah, yeah, yeah.

• 43:46Speaker 01

  I think a while ago. Any comparisons between X Grammar and outlines? Is there a trend in this in this world or is it mostly settled science?

• 43:55Speaker 03

  To be honest, I I prefer X Grammar, you know.

• 43:59Speaker 01

  Okay, yeah, tell us.

• 44:00Speaker 03

  Yeah, MLC AI is founded by by Tian Tianqi. Both Tianqi and X Grammar's other, Yi Xin Dong. They were the students graduated from Shanghai JiaoTong University. And the creator of X G Lang, Liang Ming and Ying, they also graduated from Shanghai JiaoTong University.

• 44:20Speaker 01

  Oh, my God. Is it the Berkeley of China?

• 44:22Speaker 03

  Yeah, you're right. And I I think X Grammar's performance is better than the outlines. And also in the Tensor RTM, the latest release, Tensor RTM also integrate X Grammar as the backend for the concentrated coding.

• 44:41Speaker 01

  Okay, this is new to us. We had Remy from Outlines speak at my past conference, but I did not even, I wasn't even aware of X Grammar being a thing. But yeah, I mean, structured output is something that a lot of people care about. We had OpenAI talked about their structured output implementation. And there's a lot of interest in making sure that there are no trade offs. I think there's a little bit of fud around how maybe the models are dumber when you use structured output instead of the sort of base, sort of next token generation. But I don't think it's significant that much.

• 45:13Speaker 03

  Yeah.

• 45:14Speaker 01

  We can talk about the last one, which I don't know if it's as relevant for Base10, which is the third technique of SG language is API speculative execution, which seems to be only for API-only models.

• 45:25Speaker 03

  Oh, yeah, I think it's the front-end feature, it's not the back-end. Yeah, something like you have some control flow for the LM task. Yeah, such as you have the one request to get a result and you just continue to another call. And for this case, you can use the SG lang front-end language to describe the control flow. It will make it easier to control that pattern.

• 45:52Speaker 01

  Okay, awesome. Tracing this human path, I'm pretty sure I know the answer, but is there a reason that big projects like Grok, you know, like XAI also use SG lang?

• 46:01Speaker 03

  Yeah, yeah, yeah, right.

• 46:03Speaker 01

  Is it just the same people?

• 46:05Speaker 03

  Yeah, yeah, yeah, right. Li Ming and Yin are the XAI's member of technical staff.

• 46:11Speaker 01

  I mean, it makes sense. I wonder if, you know, what's the impetus for SG lang to kind of break containment? It seems like VLM obviously has the, it's one year older, it has more community pool. I wonder how this will shake out. I don't really know. But, you know, you said it's a library, you said VLM's library of SG lang is much, much more comprehensive. I mean, do people care?

• 46:37Speaker 01

  Maybe it's like when you're serving models at scale, then you start really prioritizing the sort of performance that SGLAN offers.

• 46:44Speaker 03

  I think if you care about the performance, maybe TensorRTM is the best solution for now, especially for the latency sensitive scenery, TensorRTM doing well. But if you also want to implement some feature by yourself or do some optimization by yourself, you want to customize the framework, I think SGLAN is a good option. And VIM, I think TM's community support is very nice because it was used by so many users and it has so many GitHub stars. And you know, SGLAN, when I participated in the SGLAN team at July, it has only 2,000 stars and right now it has more than 7,000 stars.

• 47:28Speaker 03

  Yeah, I think it also grows so fast.

• 47:31Speaker 01

  Anything that people should look forward to that's on the roadmap for SGLAN that people should be aware about?

• 47:37Speaker 03

  Yeah, yeah, yeah. And we post the roadmap in the issue and we pinned that issue. Also, we have bi-weekly meeting to think with the community about our progress, our plan, which feature do we want to implement in this quarter, something like that. And we also co-host some meetups, something like the first meetup we co-host with MLCLM and Flash Infer. And we also, yeah, participated at some hackathon, something like Camo AI Hackathon. We do the presentation about SGLAN, yeah.

• 48:12Speaker 01

  I just saw it now. It sounds like actually there's, you know, you mentioned Eagle, mentioned Medusa. I think Amir mentioned Medusa, but Eagle is also part of that cabal, this speculative coding techniques. It looks like not yet supported.

• 48:24Speaker 03

  Yeah, we already supported. And I think in the open source implementation, something like a VIM, SGLAN and other frameworks. I I think it's the sorta performance. And currently, even use the tense RTM, it always support Eagle one, not Eagle two.

• 48:41Speaker 02

  Yeah. And one thing to note about speculative decoding, different versions of it is that the framework supporting it is one thing. But you will have to do the job of the training of the draft models or the additional heads. And a lot of the benefits will come from how good you are at the training aspect. And in terms of the data that you use to train the the draft model, to essentially distill the the target model or mimic its its behavior so that you can have a very high rate of acceptance.

• 49:18Speaker 02

  It's the kind of throughput improvement that you get are ultimately dependent on how good of a job you do at training the the draft model in the case of the draft target model mechanism. So that's another thing that's like, hey, does the framework support it? Or you can just turn on speculative decoding with with a flag. That's not the case. There's more that that goes to it.

• 49:42Speaker 01

  One more side thing on training. I also noticed that with OpenAI offering fine tuning for O1 and all these things. I think people are also very interested in sort of RL trainers is what they what you have here. It looks like you're supporting Hugging Face TRL and OpenRLHF. Do you think that this will become something that a lot of people are demanding? Like the sort of general feel of RL for LLMs relatively abandoned, I think, up till like the end of last year, basically.

• 50:09Speaker 03

  Yeah, I I I think so.

• 50:11Speaker 01

  I don't know. It's like, it's one of those things where like, maybe people have to wait for a base model that is that has some layer looping or some other sort of friendly architecture for reasoning instead of just pure RL on LLMs. Because I think so far, people have not really exploited RLHF as much in the wild. I mean, correct me if I'm wrong.

• 50:34Speaker 02

  Yeah, so I can I can give you some examples of when we've seen at work. Again, this is generally done by our customers before they come to us for inference. But there's examples like in the healthcare world, fine tuning models for understanding medical jargon. Like for Whisper, for instance, you know, a version of Whisper that I can actually understand medical jargon. So that's not that's non L M use case. In the L M use case, staying in the healthcare space, models that can do medical document extractions and do a very good job compared to even, you know, state of the arts because of the data that the company had gathered through human in the loop. Are those gonna go away?

• 51:19Speaker 02

  The need for those are gonna go away because there's a model that can do reasoning and do a very good job at it? I don't know. My intuition says yes. Will it be cost effective? That's the question that that I have. And in the short term, no. In the long term, maybe. But I haven't seen the need for the more traditional fine tuning actually go down. In fact, we see that quite a bit right now in the market. The question for us is, hey, do we want to address that market, knowing that the entire market might go away one day? And my general answer to that is, let's solve today's problems.

• 51:57Speaker 02

  Even if they're not around in two years, you will learn a lot along the way by onboarding customers that have today's problems. And you learn from them about tomorrow's problems and you will build ahead for them.

• 52:11Speaker 01

  What, so hang on, why do you think fine tuning may go away?

• 52:14Speaker 02

  Because, like you said, there's gonna be models with complex reasoning capabilities that can actually figure it out in a, you know, few shot kind of way without needing a large dataset to fine tune the model with. That's what some people are saying.

• 52:32Speaker 01

  I really have a trouble believing that that'll be the case. Much more believe that it's just easier to change your prompts rather than actually do full fine tunes or even parameter efficient fine tunes.

• 52:45Speaker 02

  For sure.

• 52:46Speaker 01

  Is there anything else that we haven't touched on that you wish the people ask you more about because it is something that is very interesting from your point of view of what you're seeing among your community?

• 52:56Speaker 03

  Yeah, when we released the DeepSeek Feasory support, we have some community user, something like a Cusol. Do you know Cusol? I think it's very popular.

• 53:07Speaker 01

  Of course, of course. I use them every day. When I type code dot inside of my IDE, my terminal, it actually opens Cusol instead of VS Code. I feel very bad for VS Code.

• 53:19Speaker 03

  And when we released the DeepSeek Feasory support, some employee from the Cusol team was very interested in our implementation and asked, reached out and asked some question from us. So I think as SGLAN grows faster and the features optimization, we iterate so fast. And I think there will be more users from different companies, from different teams to use it.

• 53:48Speaker 02

  Honestly, I would go back to what I emphasized earlier, which was that I wish more people asked about what it takes to run mission critical inference workloads. Because I see this in the market sometimes that they're like, well, I can just use VLM and that puts my model behind an API and that is that is production. But really it takes three pillars that all need to be there. One is performance at the model level. That is where frameworks that we talked about today really help you with, but you still have to guide them. Like when it comes to speculative decoding, yes, they support it, but who's going to train or fine tune the draft model or the Medusa heads?

• 54:33Speaker 02

  Or who's going to ensure that the reliability of the VLM server that you know, you see it in production, like, you know, there are crashes, you know, how do you, how do you recover from those without affecting production traffic? But, but by itself, that's not enough, because invariably, that one model running on a set of hardware is going to get too much traffic that it cannot handle. And at that point, you need to horizontally scale it. And that's not an ML problem. That's not a PyTorch problem. That is an infrastructure problem to ensure that you can horizontally scale up your model extremely fast to meet your, you know, P90, P99 latency requirements.

• 55:15Speaker 02

  And to ensure that you're not running out of capacity in a single region that that model lives, you end up having to scale that model across different regions and across different clouds even to ensure that that model is not being starved of resources in the one place that, that it lives. So that's, that's an area of investment that we started investing in some, some time ago. And like really paid off this past year. And the third pillar is enablement of workflows. Workflows such as the sort of AI phone call example that I told you about, the, the ones that require multi-step, multi-model inference, but in a very low latency way.

• 55:58Speaker 02

  That's the third pillar that really allows the developers to be able to use the power of the first two pillars and then be able to combine them when especially you need multiple models for, for your workflows and doing so in a, in a reliable way, repeatable way and a low latency way. And that is, those are the three pillars, honestly, that, that we have been investing a lot on. You know, some of which we started investing in three years ago and it really started paying off, you know, a year ago. So it, it, it takes, it takes quite a bit of build to get it to the point where you're truly running folks, customers, mission-critical inference workflows. What do I mean by mission-critical inference workflows? Inference where

• 56:42Speaker 02

  If inference is slow or down, the the main product of our customer is slow or down. So they really care about it. They have strict requirements around latency, around being able to support large throughput, about being able to do so in a way that, you know, other customers' usage doesn't affect the SLAs that they are getting, dealing with noisy neighbor problems. And inference done in a compliance way, whether it's HIPAA or or or certain SOC requirements. And also inference done in a geo where kind of way, both for compliance reasons and also for latency reasons, where when you where you forward the traffic has an impact on latency in situations where 50 milliseconds really matter,

• 57:27Speaker 02

  100 milliseconds really matter, and we're seeing more and more of those use cases.

• 57:31Speaker 01

  Well, one way I would recommend doing that is kind of a it's kind of a manifesto type of thing. And I'm sure you know the Heroku's tall factor app.

• 57:39Speaker 02

  I've seen that. Yes, yes. That's a good idea, actually. Yeah.

• 57:42Speaker 01

  Yeah, maybe even separate, put out a separate property than Base10 and just go like, here's what we think, you know, mission critical a application should be and, you know, have some thought leadership there and and flesh it out and, you know, see if the the market takes it on as a as a mission. And obviously, you will be best prepared to to serve that market as well. I've also seen this done very well with enterprise ready.io. I think it used to be done by, I think it's called gravitational or replicated, one of those.

• 58:09Alessio Fanelli?

  From a replicated.

• 58:10Speaker 01

  Yeah, yeah. These kinds of things, when you have a list of requirements, when like you're like, look, everybody needs this. Okay, like write them up and then, you know, put a little bit of marketing on it, spit it out from the main company brand. And that tends to work very well.

• 58:22Speaker 02

  Yeah, good idea.

• 58:24Speaker 01

  Cool. Well, thanks. Thanks so much for your time. I think this is a really good dive into both Base10 and NSG Lang and a little bit of DeepSeek V3, which people are very interested in. I'm trying to talk to them as well, because obviously they're they're a fascinating lab and but, you know, I I think you guys, you guys are doing a lot to make it accessible for everyone. So thank you so much.

• 58:43Alessio Fanelli?

  And as a, just to get based on some street cred, you know, they were one of the first sponsors for latent space events. And Amir brought 100 croissants at our latent space hackathon in 2023. So yeah, just want to bring that up. I always tell, I saw Phil and at WBS Reinvent, and I told them that was one of the the first events that we really did and one of the turning points of this industry as far as community goes in my mind. You know, everybody, everybody, everybody was there. The croissants. No, no, not the croissants, the event itself. Yeah, entire company is launched. Yeah, you were, I mean, you know, like Natter from Brev was there and did the with Joseph from Roboflow. They did like the prompt battle thing, like Harrison was a judge and it's like Jerry from Amindex was there.

• 59:27Alessio Fanelli?

  Like, kind of like everybody that is kind of now breaking out. If you look at the the graph that Jensen put on on the screen at CES with some of the companies that work with, a lot of them were at that event. So, yeah, thanks for staying, stay involved with us, Amir, and I'm sure we'll, we'll do more together. And thank you, guys. Many more years to come, for sure. Thank you for taking the time today. Good to see you both. Alas, see you, Sean.