#239 Tuhin Srivatsa: How Baseten is Disrupting AI Deployment & Scaling in 2025

This episode is sponsored by Thuma. Thuma is a modern design company that specializes in timeless home essentials that are mindfully made with premium materials and intentional details. To get $100 towards your first bed purchase, go to http://thuma.co/eyeonai ————————————————————————————————————————— AI deployment is broken—can it be fixed? In this episode, Tuhin Srivatsa, CEO & Co-Founder of Baseten, reveals how his company is DISRUPTING AI infrastructure, making it easier, faster, and more cost-effective to deploy and scale AI models in production. As enterprises increasingly turn to open-source AI models and grapple with the high costs and complexity of scaling, Baseten offers a game-changing solution that eliminates bottlenecks and simplifies the process. Discover how Baseten is taking on AWS SageMaker, OpenAI, and cloud-based AI deployment platforms to reshape the future of AI model deployment. What You'll Learn in This Episode: Why AI deployment & scaling is one of the biggest challenges in 2025 How Baseten enables enterprises to run AI models faster & more efficiently The shift from closed-source to open-source AI models—and why it matters The hidden costs of AI inference & how to optimize for performance Why most AI models fail in production and how to prevent it The future of AI infrastructure: What comes next for scalable AI Whether you're a machine learning engineer, AI researcher, startup founder, or enterprise leader, this episode is packed with actionable insights to help you scale AI models without the headaches. Don't miss this conversation on the next era of AI deployment! #AI #ArtificialIntelligence #MachineLearning #Baseten #AIDeployment #AIScaling #Inference #MLInfrastructure #TechPodcast Stay Updated: Craig Smith Twitter: https://twitter.com/craigss Eye on A. I. Twitter: https://twitter.com/EyeOn_AI ————————————————————————————————————————— (00:00) Tuhin Srivatsa's Journey in AI & Baseten (01:50) What is AI Infrastructure & Why It Matters (03:30) How Baseten Optimizes AI Model Deployment (05:19) Why Most AI Deployments Fail (And How to Fix It) (09:17) The Future of Open-Source AI Models in Enterprise (11:01) How Baseten Automates AI Scaling & Inference (14:12) Why AI Developers Struggle with Cloud-Based AI Tools (18:47) The Real Cost of AI Inference (And How to Reduce It) (20:44) Why AI Scaling is the Biggest Challenge in 2025 (26:55) Can AI Run on Non-NVIDIA Chips? (The Hardware Debate) (31:23) The Future of AI Model Deployment & Inference (37:05) How AI Agents & Reasoning Models Are Changing the Game (40:39) The Truth About AI Hype vs. Reality (45:04) How to Get Started with Baseten (45:48) The Future of AI Infrastructure

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You're a company that is realizing that AI is a big part of your strategy, that you've been using OpenAI or Anthropic to get started. Now, all of a sudden, you see a lot of, you see a lot of adoption, and you're like, wow, this is cool, everyone's using AI. And based on that, you say, okay, I really need to scale this up. As you scale, you start to hit some limits. So one, you want to run this side of your VPC, which is very expensive. Two, you're now beholden to OpenAI, you don't have much transparency in terms of, you know, where your data is going. What does this model look like, and so on and so forth. What deeply provides an open source model to be able to do that? And so you can run this model faster, cheaper, better, with a bit more control. And all of a sudden, become somewhat irresponsible not to consider something like that. Hi, my name is Tu Han, I'm the founder and CEO of Base10.

• 0:42Speaker 00

  You know, my, my background's in machine learning engineering for the past fifteen years. You know, prior to that, I, I studied electrical engineering at USC. I, I think, you know, I, I've been really focused on bringing machine learning products into production and, you know, making sure that models get used in good ways, and that kind of left, led to the genesis of Base10. Base10 is a machine learning infrastructure company, we're about five years old, based in San Francisco and New York. And we, we focus on the ability to run, giving customers the ability to run their models in production, performantly, reliably, and securely.

• 1:24Speaker 00

  And what that really means is that, you know, you come with your own model or the model you want to use. We kind of give you all the tools to be able to run that as fast as possible, as scalable as possible. And kind of come with all the batteries included, so you don't have to deal with issues. So, you know, it's a combination of workflow, it's a combination of compute, and then a bunch of optimizations to allow that, you know, you can serve your customers as well.

• 1:50Craig S. Smith?

  Yeah, and, and when you say AI infrastructure, what exactly is, is Base10 a, a cloud? Is it a, a, a private cloud or, or?

• 2:07Speaker 00

  Yeah. Yeah, a hundred percent. So, you know, I I would divide, I would divide infrastructure up into a couple of different problems. So there's like the training problem and the inference problem. Um, they're they're the the two broad categories. And look, there's a bunch of subcategories alongside this. We're wholly focused on infrastructure, on the inference piece, which is running those models. What do you, what do you need to run models? So you kind of need like two or three things, right? So you need the ability to acquire compute. Um, because these models require GPUs. Um, and so we allow you to either come to us and we will provide a compute, or you can run it on your own compute.

• 2:47Speaker 00

  And that brings up to the second piece, which is, you know, we give you a lot of software to manage the mo, the running of those models on that compute. So you have kind of like compute on one side, APIs that run models on the other side. The thing that maps those two things, that's what Base Ten is. Uh, and then, you know, when you move to production, when you move to production and when you work to move to real workflows, um, you have all these needs. So think about, you know, when you run a web service, you know, you obviously have, you know, where does the, where's the web service run? You have your servers. And then you have all the observability and C I C D. Um, and, um, alerting and and whatnot. That sits on top of that. And Base Ten provides all that um, kind of glue as well, if that makes sense.

• 3:30Craig S. Smith?

  Yeah, and it's a, it's a, a software layer that then orchestrates uh, uh, compute resources from whatever cloud it's connecting to. Uh, and and when you say the tools to, uh, uh, you know, manage and operate, uh, your, your, uh, your model, what what does that mean? What kind of tools?

• 3:57Speaker 00

  Yeah. Uh, so all all sorts of things. So like, you know, for, you know, to start with, it's like, hey, you know, how does, how do you manage different versions of this model? So, you know, we give you, you know, version management, then there's like the C I C D part, which is like, hey, how do you roll out models? Um, then you have, so that's one piece, which is like, hey, you know, A B testing, production rollout, all all those things, blue green, um, deployments, we give you, we give you access to tooling for that. But then there's like the observability piece, which is like, hey, my model's running. How well is it running? And so we have a whole observability suite that sits within Base10. And then, you know, obviously the management of compute as well. So one thing we provide is the ability to run your model across multiple different clouds.

• 4:41Speaker 00

  So that's one workload that's sitting across a bunch of different clouds. And, you know, it's like, how do you orchestrate that? And we give you kind of the software layer to say, hey, I want this model to run in this place and overflow to that place. And this is the amount of replicas I want running here. And how, you know, what are the scaling settings for that? So we give you an abstraction on top of that as well. And then, you know, we go all the way down to the performance level, which is like, so I've talked a bit about the software and the workflow piece. I've talked a bit about the scaling piece. Then there's the real performance piece, which is how well does a model run on the individual GPU? And then we give you a bunch of software to optimize that itself.

• 5:20Craig S. Smith?

  Yeah.

• 5:22Craig S. Smith?

  And why would someone use Base10 as opposed to whatever else they would use? What would they use without Base10?

• 5:33Speaker 00

  Um, yeah, so, you know, it's funny, like, you know, we spend a lot of time, you know, selling up market and like talking to customers who are either in the enterprise or we're selling to the enterprise. And what we find is that actually for that set of requirements, there's not that many tools that a customer has at their disposal. Like, you know, this market, the really cool thing about AI is that everything is so new. The really, the really bad thing about AI is that everything is so new. And so what that means is that the set of tools required to run models in production is pretty underdeveloped. So most of our customers just end up building them themselves. So what that means is usually, you know, going to their cloud of choice.

• 6:15Speaker 00

  So whether that's AWS or GCP or Azure, and trying to piece together all the, all the different services that sit on top of that to construct something like Base10. And honestly, that's where we see.

• 6:29Speaker 00

  That we have the biggest value add where i- if you are now sitting inside AWS and trying to or GCP and trying to connect, you know, Vertex to the Observability Suite, to the Compute Suite, uh, all of a sudden you're fighting these internal organizations uh, at GCP without really knowing it, or AWS without really knowing it. And you know, what Base Ten does is kind of, you know, um, packs us all those things together and provides one comprehensive end-to-end solution. And that's really where we see our value. So like, I'd say like, most of our customers that we talk to are trying to cobble it together themselves using a vast number of resources um, at their disposal.

• 7:08Craig S. Smith?

  Yeah, and the, and the interface uh, for Base Ten is that uh, all the tools are, are, are together in one place and yeah.

• 7:19Speaker 00

  One consolidated experience, yeah.

• 7:21Craig S. Smith?

  Yeah, and then uh, uh, do you, do you actually, can you, could, could you share your, your, uh, the screen and just walk us through uh, uh, is that okay?

• 7:34Speaker 00

  Uh, that is okay. I, I'm not ready for that, but I can do that if you give me a couple.

• 7:37Craig S. Smith?

  Yeah, okay, at another time then, yeah. Uh, and uh, so for someone to use Base Ten, uh,

• 7:47Craig S. Smith?

  walk us through a use case.

• 7:49Speaker 00

  Yeah, um, so why, why don't we talk about something that's very top of mind, DeepSeek? Yeah. Okay, so Deep, DeepSeek is an open source model that was released, yeah, uh, well, DeepSeek is an organization that released a bunch of open source models. Um, you know, in, at the end of December and a few, a couple of weeks ago, respectively, a, a base model could be three and a, and a reasoning model called R1. Now, you, you're, you're a company that is, you know, realizing that AI is a big part of the strategy, uh, and you've been using OpenAI or Anthropic um, to, to get started. Now, you know, all of a sudden you see a lot of, you see a lot of adoption, and you're like, wow, this is cool, everyone's using AI and um, based on that, you say.

• 8:34Speaker 00

  Okay, I really need to scale this up. As you scale, you start to hit some limits. So one, you want to run this side of your VPC, which is very expensive. Two, you're now beholden to OpenAI. You don't have much transparency in terms of where your data is going. Where your data is going, what does this model look like? And so on and so forth. What DeepSeek provides an open source model to be able to do that. And so you can run this model faster, cheaper, better. It's with a bit more control. And all of a sudden, become somewhat irresponsible not to consider something like that. So what Base10 does, okay, now go DeepSeek.

• 9:14Speaker 00

  Inside my VPC.

• 9:17Speaker 00

  This is a 671 billion parameter model, which is very big. So for a number of reasons, this is very hard to run. You know, it requires a lot of compute. So you can't just run it on, you can't run it on your laptop. So you need to go and buy and compute. You, if for something like DeepSeek, you actually need a whole lot of GPUs. So it's not just, it's just not like one H100. You need either 16 H100s or eight H200s. So all of a sudden, you need to, that's, so let's talk about the H100 case. Cause not many people have H200s today. So you have two nodes of H100s that you need to run. So multi-node inference is now a problem where you need to shard this model and its weights across 16 different GPUs. And then you have it running on the GPU.

• 10:02Speaker 00

  And then like as your product scales, you need that to scale up and down. So how, how now, let's map that to Base10 now. So you come to Base10. You, you know, you, it's as simple as writing a simple Python file that pulls down the weights from Hugging Face. And then you write an inference path in the predict path. It's Python file, honestly, probably 20 lines of Python code. We have a lot of boilerplate, so you don't have to do this too much either. You come that you go to your CLI and you run trust push. Um. Trust push. Um. And all of a sudden, this this this model with all its scaffolding at the compute resources under under the hood gets spun up um on base ten.

• 10:44Speaker 00

  What you get with that is a visual interface for um you know, being able to manage everything around that. But then you also get an API endpoint. You also get an API endpoint um to run that. So all of a sudden, you have this model running, but then you're also getting all the scalability. So now, when your model, when you when you start to get traffic, um, you know, cool, it's all working. But then your your traffic doubles. So what do you do? Well, you know, in in the old world, you have to figure out how to scale that up. Base ten automatically scales for you. And so all of a sudden, it's running a twice as many replica and then scales back down where you don't need that traffic. Um. You know, some you hit a hitch, you know, and you're like, what what's going to run? You you log into base ten, you hit the metrics tab, and all of a sudden, you have all the metrics on your on your service.

• 11:29Craig S. Smith?

  Okay, the um, and then the model resides on on the virtual machines in whatever cloud you're using.

• 11:41Speaker 00

  Yeah.

• 11:41Craig S. Smith?

  And and base ten is is a layer on top of that to to manage the model. Is that right? You're not providing your.

• 11:51Speaker 00

  So we we we can provide compute, but they're not our compute. You know, we sit on top of um all the clouds you might think about. So like Azure, OpenAI, Azure, AWS, GCP, Crisso, Oracle. So we can provide compute and we have, you know, we're putting great pricing and all that stuff if necessary. Um. We like to think of ourselves as a software layer and we're not compute providers. You can get that if you want, but you know, what we're seeing from most enterprises today is that everyone wants to have hosted. And somewhat of it's somewhat of a nonstarter. So even, you know, enterprises that are using OpenAI and Anthropic um at scale, they're using it deployed within their VPCs, um and so we can sit on top of your VPC.

• 12:32Craig S. Smith?

  Yeah, uh, you know, I'm familiar with SageMaker at Amazon. It's it's some of these uh, some of those functionalities sounds like SageMaker. How how um, do you do you differentiate or am I completely off base?

• 12:49Speaker 00

  No, that that's right. I think, you know, say SageMaker, you know, was built for the old world when there were small models. Then there were small models and they were easy to learn in memory. But again, like SageMaker, like, yes, like i- in in the um, in the best case scenario, SageMaker would give you a lot of this functionality. What we actually see from our customers is that SageMaker is that, you know, you're you're fighting within fighting with software to make it work. And really, like, you know, we, the thing we haven't touched on right now is the developer experience that sits on top of this. Which is that, you know, we try to make it easier. So we've heard from many customers that, you know, hey, it could take me anywhere from forty to sixty hours to deploy a model in SageMaker. You know, our goal based hand is, you know, ten minutes to deploy a model.

• 13:34Speaker 00

  And, you know, that that that's how we think about our differentiation, which is, you know, for for for AI teams today. Your number one, your number one differentiated factor and is your time to market and how quickly you can go to market. Um, you know, your tool shouldn't get in the way. The tool should be an enabler for that. And we're really trying to speed up that that those iteration cycles um, between you and your customers. So you can, you know, roll out pretty advanced AI initiatives um, as opposed to, you know, spending your time fighting with SageMaker um, or Vertex, um, you know, a- amongst others. Does that make sense? It's it's it's just like, it's like the um, you know, this is like, I really like the analogy of something like a data dog.

• 14:20Speaker 00

  Um, where, look, Amazon and GCP have a bunch of observability suites. You know, I they, you know, there's probably two dozen observability products in each of those. Data Dog is a really great business because what it does is it kind of links all those tools, turns it into one. I'm a provider that end to end developer workflow, so you don't have to fight with those tools.

• 14:43Craig S. Smith?

  Yeah, and and base ten is itself is not open source, but you employ a lot of open source tools.

• 14:52Speaker 00

  Yes, that's right. And so like one one thing is like, you know, when you're writing code in base ten, you're just writing Python code. There's nothing really proprietary there. And and so like our our core technology is closed source. But you know, we have, we have trust, which is a way to package your models, which is an open source technology. That, you know, again, it's on ramp to base ten. But if tomorrow you say, hey, I actually want even more control and I don't want to be beholden to base ten, you know, you can pull your models and we, you know, we we bank on ourselves to like keep you with the value we're providing. But we we don't want to have you stuck. And and honestly, like the closed source nature of base ten is for no other reason except speed.

• 15:34Speaker 00

  You know, we can, you know, it just allows us to move faster and move with the market. And with, you know, with open source stuff, you know, we're we're built on the shoulder of a bunch of open source giants and, you know, we we could do a lot of work with open source, but for for us, it's more about how do we enable our customers to move as fast as possible. Yeah. If that if that makes sense.

• 15:53Craig S. Smith?

  Yeah. And and you're serving, I mean, you it's it's for the deployment primarily of open source large models, right?

• 16:03Speaker 00

  Yeah, hundred percent. Open source and custom models, I would say. So, you know, like, for example, we do work with foundation model companies that have their own models and we are the serving layer for them. And and and and really like the the narrative there, or the way I would think about that as a customer is that you know, your your core differentiation is in the application layer for the most part, or the model, like the infrastructure is pretty much undifferentiated.

• 16:29Speaker 00

  You know, you can either go the OpenAI or Anthropic round and build out, you know, a three or four dozen inference team to run these models. Or you can ideally just use someone like us and kind of abstract that problem away, allow you to move faster on the the experiences that differentiate you.

• 16:45Craig S. Smith?

  Yeah, yeah. When you say

• 16:49Craig S. Smith?

  Abstracted way, I mean, if if you're if you're using one of the big foundation models, like OpenAI, is is Base10, you are you, do you, do you create a private instance of that model that then you can fine tune and?

• 17:13Speaker 00

  Exactly.

• 17:13Craig S. Smith?

  Yeah. Or or are you just hitting that model through an API? And Base10 is operating at the application layer?

• 17:22Speaker 00

  Yeah, so it's a bit of both, right? So you could you could kind of choose like, you know, there are definitely customers that deploy a model and use this as API. And like, it's an API management product, but you know, a lot of our customers go a lot deeper and like, you know, you're able to tune the parameters and around which, because one thing you have to remember is that

• 17:42Speaker 00

  all customers, it's not like a one size fits all solution for all customers. If that makes sense, like, sure, the the core of serving technology

• 17:53Speaker 00

  is not going to be that differentiated between our customers, but what will be differentiated is how could how they configure that serving technology? If that makes sense. And so like, you know, people have different traffic patterns, so they need different scaling stuff. They they have different quantization methods, so they, you know, they care about like the different, the different

• 18:14Speaker 00

  like quality of the model and like some people care about quality because I'm talking about speed. And so that's the control we're giving you while abstracting out the core service layer, if if you would like that.

• 18:25Craig S. Smith?

  Yeah, the you're also more cost efficient, aren't you, than

• 18:37Craig S. Smith?

  than SageMaker or or it's another

• 18:43Craig S. Smith?

  system like that? Can you talk about the pricing?

• 18:47Speaker 00

  Yeah, um, look, there, there's a couple of different ways we do pricing, depending on like where you're running the compute. Um, what we find is that, you know, when you're running on base ten, because we have this elastic compute model. Where, you know, um, you know, what our, our goal is to go and negotiate compute on, on behalf of our large customer base and then amortize that over our customers. Um, it's almost like a collective where the union, where the union of unions of startups, uh, uh, in a way. But what that allows us to do is it allows you to, let, let's take a customer, you know, typical web traffic patterns kind of like this, right? Where it's like, you know, in, in, in the day, you get a lot of stuff at night, it's pretty quiet.

• 19:28Speaker 00

  The default right now, if you run your own compute with a stage maker is to just have it on all the time.

• 19:33Speaker 00

  And what that means is that, you know, you, you, the gaps in traffic you're paying for. Base ten actually allows you to scale with your traffic. So you, you know, you end up actually just saving a lot of money with that auto scaling. Uh, behavior, but then we do a lot of performance tuning as well. So we can, you know, we, we employ stuff like. Um, distillation and spec lip decoding to be able to run your models faster, which again results in. More computer, like better computerization and cost efficiency. And then on terms of, um. Uh, in terms of actually how we think about, um. It doesn't actually how we think about pricing the product. Look like we, we want to be charged. We want to be seen as a software layer, not a compute markup layer. Does that make sense?

• 20:15Craig S. Smith?

  Yeah.

• 20:15Speaker 00

  And, and so I, for all intents and purposes, you could think of it that, you know, we, we pass through the value of computer that we charge you a software layer. On top of that, and like ideally, you know, the way we think about that is especially in the enterprise is that we are saving you at employing a team. Of, you know, anywhere from three to thirty people to do this, and we just want to have some, um. Portion of that at our, as I think it, it ends up with a lot of cost savings for our customers. And it's kind of cost savings on three fronts. Uh, if that makes sense.

• 20:43Craig S. Smith?

  Uh, yeah, and, and this, uh, it seems that you're in the right place at the right time because open source is exploding, right?

• 20:52Speaker 00

  Yeah, yeah, yeah, 100%.

• 20:54Craig S. Smith?

  And, and so are you seeing a spike in demand as people turn to open source models? I mean, again, not that they can't use proprietary models, uh, with Base10, but it, it, it seems geared toward, uh, you know, open source deploying, open source models. Create an oasis with Thuma, a modern design company that specializes in furniture and home goods. By stripping away everything but the essential, Thuma makes elevated beds with premium materials and intentional details. I'm in the process of reorganizing my house and I'm giving Thuma a serious look for help

• 21:38Craig S. Smith?

  in renovating and redesigning. Thuma combines the perfect balance of form, craftsmanship, and functionality. With over 17,000 five-star reviews, the Thuma bed collection is proof that simplicity is the truest form of sophistication. Using the technique of Japanese joinery, pieces are crafted from solid wood and precision cut for a silent, stable foundation. With clean lines, subtle curves, and minimalist style, the Thuma bed collection is available in four signature finishes to match any design aesthetic.

• 22:23Craig S. Smith?

  Headboard upgrades are available for customization as desired. To get $100 toward your first bed purchase, go to thuma, that's T-H-U-M-A dot C-O slash ionai. Ionai all run together, E-Y-O-N-A-I. So for $100 off your first purchase, go to thuma.co slash ionai. That's T-H-U-M-A dot C-O slash ionai to receive $100 off your first bed purchase.

• 23:06Speaker 00

  A hundred percent, you know, like I, I think the, the narrative, the narrative, the high level narrative has been over the last. You know, let, let's call the start of time is the chat G P T moment, you know, that feels like a decade ago. It's crazy to think that was Yeah. Two years, two years ago, but you know, the narrative has been over the last two years that. You know, there's two, there's been two prevailing narratives. One has been. Look, close source is the future. Anything behind that's not a frontier model, it doesn't matter. Um, and you know, that, that's largely been pushed by the large labs who'll be developing these closed models. There's been the second.

• 23:47Speaker 00

  Other narrative, which is, hey, open source models are incredibly important to the future. These are very powerful models. Um, you're having more, having more things out in the open, um, is going to make a big deal. Um. We definitely have conditioned our company on the existence of both, but what's happened over the last two years is that. There's been a convergence of the quality of open source models, uh, with closed source models. Uh, and what that means is that as a enterprise, you now have a lot more options into deploying these models. Uh, and what models to use. Um, and again, like we're talking about anywhere from like a thirty to forty percent decrease baseline in costs. To at times a eight to ten X decrease in costs.

• 24:29Speaker 00

  Uh, and, and with that in mind, and with the security and privacy concerns where these models run. And they're not, not being beholden to one provider as your model provider. And what that means, the enterprise have more option and frankly, it's, it'll be, it's somewhat, um.

• 24:47Speaker 00

  Irresponsible for you not to be considering how to, how to, uh. Factor open source models, um, into your strategy. And so, look, the business has grown. Remarkably, I'd say like eighteen, if I looked at eighteen, eighteen months ago to now, we're talking about, you know, a two hundred X increase in revenue and usage over that time period. You know, I'd love to tell you that we are special and you know, we have, but I'll tell you where the right place, right time, and that's really it. But I think especially over the last two weeks or three weeks as you know, this barrier of it can open source ever be as good as closed source has been breached.

• 25:28Speaker 00

  So you know, we've, we've probably heard from, I want to say three dozen enterprises saying, hey, we need to be briefed on the strategy here going forward. Everyone is trying to figure out what this, what does this mean? And that, that, that to me is very exciting and that, that's where I, I can only see that increasing and accelerating over the next 12 months.

• 25:48Craig S. Smith?

  Yeah, and you, you, you're focused on inference.

• 25:56Craig S. Smith?

  Do you offer other chips than Nvidia?

• 26:02Speaker 00

  Yeah.

• 26:04Craig S. Smith?

  Because I, I've had Rodrigo Leong of San Benova on and Andrew Feldman of Cerebras. And I haven't had the Grok, Grok Q, I haven't had him on yet. But, but these guys are blindingly faster than GPUs for inference. So, yeah.

• 26:30Speaker 00

  Yeah, it's a really fascinating question. It's a, it's a really fascinating market. It's a local. Firstly, we are agnostic. We're cloud agnostic. We're chip agnostic. That, that's the reality. You know, we, we, we, we run, we have run on TPUs, we've run on Tranium, we've, you know, we've looked at, we've looked at AMD chips. We've, we've spent time with this. That being said, like, that's great. And like, that is the, the, the narrative going forward.

• 26:55Speaker 00

  Any work we have done on anything except Nvidia chips has been somewhat painful, somewhat painful. And you know, we just don't see, you know, CUDA is, is old technology that has, you know, be, you know, I was, I was, when I was downloading video game drivers 20 years ago, sitting in my, in my bedroom, my childhood bedroom, I was using CUDA drivers to get FIFA running. And so, you know, the, this is old technology that is a lot, a lot.

• 27:23Speaker 00

  Easier to work with now. If I, if I go back to the original premise of Base Ten, which was like, if you're, if you're a, if you're a company trying to move as fast as possible, um, you don't want to be sitting there fiddling with kernels, um, you know, the day before a big release. Uh, and, and Nvidia just provides that and like they're, from what we have seen, they're hands down better than those other chips or faster and easier to use than those other chips I mentioned. Um, in terms of the kind of the new, the new chip, the new chip companies, it's so exciting. We're very excited about those and seeing where they go. I still would say, you know, there's a, you know, I don't know if any of them have proved or shown the unit economics to show that that is actually just a, you know,

• 28:06Speaker 00

  that is actually sustainable at any major scale. Um, and you know, I, I think that will change, but I think today, um, they are fast, but the, the abstraction hasn't built, been built on them, nor do customers have, um, clarity around, um, you know, what, what are the unit economics?

• 28:26Craig S. Smith?

  Yeah.

• 28:27Speaker 00

  Here.

• 28:28Craig S. Smith?

  Yeah. Um, okay. So how does, uh, how does somebody use Base Ten?

• 28:33Speaker 00

  Yeah. Um, and, and so it's pretty straightforward. You know, you sign up, you, you can either sign up and go through our self-serve flow, or you know, you, does it get in touch, but it, we'd love to help. Uh, what we believe is that, um,

• 28:50Speaker 00

  you know, we, we really believe in like accelerating time to value for our customers. Um, and so like, we almost take like a boutique-like approach to working with customers. So when we work with enterprises, we'll actually give you a team that will help you get up and running really, really fast. And so what that means is that we'll, you, we'll talk about the use case, you'll identify the model, we'll work with you to optimize the model for your use case and we'll deploy it within Base Ten. And so there's kind of two separate approaches to using Base Ten. There's the more self-serve option. Uh, and there's the more high-touch option and you know, we're, we're happy to support folks, uh, whatever journey, um, you, you want to go down. But there's a, the, the trust is a really great entry point that I talked about earlier. It's an open source packaging library that you can use.

• 29:33Speaker 00

  Um, but you can also just sign up and get started. That's a model library that you can deploy. You know, dozens of pre-existing models. I mean, it creates an API for you and you're almost ready to go instantly.

• 29:45Craig S. Smith?

  Yeah. And on the models that you've deployed or you see people deploying, you know, you look at the hugging face now, there's.

• 29:54Speaker 00

  Yeah.

• 29:55Craig S. Smith?

  I don't know how many there are, but there's.

• 29:56Speaker 00

  Hundreds of thousands, I'd guess.

• 29:58Craig S. Smith?

  Hundreds of thousands, yeah, of open source models. So is it pretty concentrated on like the top three or four? And what are those right now?

• 30:12Speaker 00

  Yeah. Yeah. I mean, I, I, I think like, the truth is, is that, you know, just with any, anything in life, everything just follows a power law. And what I mean by that is that, you know, the, the top 10% model of models account for 90% of the usage. And, you know, for us, there's, you know, those models are the llama, the llama family, the Mistral family of models. Quen, which is from the Alibaba, DeepSeek has, DeepSeek and all those distillations are a big deal. That's on the language model side, on the, on the image model side, Flux is stable diffusion models are very, very popular.

• 30:52Speaker 00

  And then on the audio side, Whisper is still, you know, or variants of Whisper, which is the text, text, speech to text model. Is very, very, very, very popular.

• 31:06Craig S. Smith?

  Yeah, yeah. For your growth, I mean, who is, who is your competition? Is there competition?

• 31:18Craig S. Smith?

  And how do you see your, your company growing?

• 31:23Speaker 00

  Yeah, I mean, I mean, look like, I'll be honest with you, like, I, I, I am, you know, somewhat convinced that this is like one of the largest value creation opportunities of our lifetime. These models have to run somewhere. To me, I think it could be one of the most important companies in this space.

• 31:41Speaker 00

  The scope of the opportunity also means that, you know, I think other people see us as competition. You know, our core differentiation from those customers is kind of across the, the three pillars that I described, both pillars that I described. So, uh, and I'll go over them again, which is infrastructure, like I think we have some of the best infrastructure that's been built here that's scalable, runs wherever. You know, you name it, like any cloud, any chip, uh, scales, scales kind of forever.

• 32:11Speaker 00

  There's a massive differentiation there, um, and we've built a lot of software there, um, in, in, interesting things there. The second piece is performance, which is like, you know, we, we think performance is necessary, but not sufficient to win. These models have to run fast. We provide you with a suite of tools to run them as fast as possible. The third one is developer experience, you know, we, we think, you know, where we, we differentiate pretty aggressively with most of our customers in terms of ease of use. And the last piece, which I, you know, cannot understand is our forward deployed engineering team, which is like, we, we want to work with our customers. We enjoy working with our customers and we don't think of it as like a, a cost of goods sold. We think of it as like a value creation opportunity and that is a massive differentiation.

• 32:54Speaker 00

  So, you know, again, when you, when you go and sign up with one of our customers and when you are competitors and when you go and sign up with us, I think the thing you hear time and time again is, you know, these people felt like an extension of my team, uh, as opposed to like, you know, customer support.

• 33:11Craig S. Smith?

  Yeah. Uh, and, and when somebody deploys a model using base 10, then, then you, um,

• 33:21Craig S. Smith?

  I mean, you help them deploy, but then are, are you there for the long haul with? You said that, you know, people can, uh, port their models. Uh elsewhere, but but yeah, I'll talk about that.

• 33:36Speaker 00

  A hundred percent like, you know, we we have like a sub ten minute response time to any time a customer needs anything, you know, we we, you know, we we want to scale with customers forever. One of the big reasons for, you know, the self hosted offering we have is that we believe that it helps align incentives between us and our customers like I didn't, I think. If we were, i- if you had to run a base in cloud and the markup was all in the base in cloud, uh, I think that'd be pretty, that's pretty, you know, I I it'd be hard for me to found them at scale as a buyer. And so we want, you know, started like our our journey, hopefully it looks like you start on the base ten cloud. You know, we we sit across a bunch of other clouds, you don't have to think about it, you hit some substantial scale, we're happy to.

• 34:20Speaker 00

  Port it over to your cloud and, you know, change the change the nature of the deal. So, you know, we can be supportive for the long for the long haul. Um, you know, we we are, you know, we're pretty proud that, you know, in the the five years of our existence, you know, we basically have never lost customers. Um, you know, the, you know, obviously it's not zero, but it's pretty close to zero and just for customers who have left us.

• 34:43Craig S. Smith?

  Yeah, and and it's not only uh cloud, you can um um this can work on premise.

• 34:50Speaker 00

  Hundred percent.

• 34:51Craig S. Smith?

  Yeah, yeah, and and you see what is everything primarily on cloud right now, or is there? I mean, as people get their hands on these open source models uh in their certain industries where things have to be on from. Uh, do you see that as is a trend?

• 35:11Speaker 00

  Yeah.

• 35:12Craig S. Smith?

  Yeah, you know.

• 35:13Speaker 00

  Hundred percent. I think there's many trends here, which is like, you know, on your cloud, on our cloud, on your cloud, on your VPC, on your on your FedRAMP VPC. And then in our data centers where we're in like for our for our customers in their data centers. And I think, you know, this is like. I I think this is like one of like the most interesting trends in the and I'm sure Craig, you you you have thoughts on this, which is like the reverse migration from, you know, it was kind of like everything on prem to everything in in our cloud to, hey, how about your cloud? And I think this is the future of software, the B O I O C approach.

• 35:52Craig S. Smith?

  Yeah, yeah. Bring your own cloud. Yeah. Interesting. The uh uh and and how how are you scaling? I mean, can you talk about uh the number of uh models deployed or the number of customers? How it's growing? It just seems again that you're in a in a unique uh position, given what's happening in the industry.

• 36:18Speaker 00

  Yeah, um, you know, I I think there'd be other people would be very upset at me if I disclose exactly what I can tell you. You know, we have, we have thousands of customers and

• 36:25Craig S. Smith?

  Yeah.

• 36:25Speaker 00

  you know, tens of thousands of models deployed and the, you know, the, again, like, the growth is, the growth is astounding to some degree, you know, like again, like we're we are, you know, this market, I I think what's amazing is like we've grown so much over the last eighteen months, but what's very clear to me is that over the next five to ten years, um, the like, I don't think we've scratched this, we're just scratching the surface right now of what's possible. You know, most enterprises have not come online yet. And the opportunity in front of us is a lot more exciting than where we're from, if that makes sense.

• 36:57Craig S. Smith?

  Yeah, uh, and beyond uh inference, uh, people are are now focused on building agents.

• 37:05Speaker 00

  Yeah.

• 37:06Craig S. Smith?

  Uh, how does that uh uh work with?

• 37:10Speaker 00

  Yeah.

• 37:11Craig S. Smith?

  X ten?

• 37:12Speaker 00

  Um, yeah, well, you know, it it is very funny. I think there's two, there's two new approaches, two new changes in the in the industry over the last six months, which I know like twelve months, which um, actually, I think are pretty conducive for us. Uh, one is that agentic applications and the other one is reasoning models. And both of those actually just require more inference. So you know, an agent is just a, you know, a chained number of inference calls.

• 37:41Craig S. Smith?

  Yeah.

• 37:41Speaker 00

  It's like, you know, and you know, in a lot of ways, like all these performance issues become a lot more important when you get to agentic models, agentic applications, because you're not just making one call. At times you're making hundreds of calls

• 37:56Speaker 00

  to the model. And so agentic models drive more inference. And I think like the optimization piece gets a lot more important with that. And then the second piece is reasoning models, which is, you know, scaling test type computer and or just inference, scaling inference. Again, more inference. And so again, like these are great macro trends for us. But I think they're also like very hard things to deal with as a customer. And I think, you know, we really do want to help people through this journey, through this journey, if that makes sense.

• 38:24Craig S. Smith?

  Yeah. Well, let's talk about the agents first. Does, does the application, the agent application, does that sit outside of base ten and then, and then the, it's, it's calling or going through base ten.

• 38:41Speaker 00

  Exactly.

• 38:41Craig S. Smith?

  To get the inference. Yeah.

• 38:43Speaker 00

  Exactly. The application is our customers, you know, we just make it easy to do the inference itself. So like, all that, like, look like we have a product called Chained, which allows you to build a lot of these compound AI systems, which, you know, we mentioned become AI agents, but I think higher level than that, you know, we don't want to get intertwined with our customers' application logic. That is theirs. To us, like those application, those agents have a bunch of inference calls in them. And, you know, we want to be that delegated inference layer for them.

• 39:12Craig S. Smith?

  Yeah, yeah. And just sort of backing up a little bit, where, where do you see this going? I mean, you know, these reasoning models are getting increasingly powerful. The, the agents are becoming increasingly powerful. People are now talking about society of agents or

• 39:33Speaker 00

  Yeah.

• 39:34Craig S. Smith?

  The only humans only interact with at one end or the other.

• 39:40Craig S. Smith?

  I mean, how, just can you talk generally about how you see AI developing?

• 39:45Speaker 00

  Yeah.

• 39:46Craig S. Smith?

  Going forward?

• 39:47Speaker 00

  Yeah, I I think I think that's right. Like, if I'm being really honest, like, uh, I get Craig, you you you're you're in this much as I am from the high from like the the macro level, which is like, I mean, every six months, I'm sure every prediction you've made over the last three years has been has been proven drastically wrong. And I think, you know, that that's the truth here. It's like, look, we're we're here riding the wave. Uh, I I think what's more important is that what we're seeing is that inference becomes a lot more important. Reasoning models, again, these models get a lot stronger. Uh, um, and the inference time compute gets a lot more important. Um, but what's more important, to be honest, is like,

• 40:27Speaker 00

  what I think I like separates out it. It's how this AI hype from past bubbles is like the value is so real. You know, we're fundamentally redefining like, you know, when people ask me about, is this a bubble? You're like, probably a short term, yes, but long term, no, right? It's like, you know, yes, we're probably over like overvaluing a lot of things over the last next few months and, um, but it does seem like over the next 10 years, um, everything will fundamentally change. And, you know, AI is going to be a big part of that. And, you know, inference is going to be a really big part of that. And whether that's through scaling inference time compute, whether that's through reasoning model, whether whether that's through agentic applications

• 41:10Speaker 00

  um, or compound AI systems, inference has a big part in that. Um, but you know, any, a- again, like, I I you would have, you'd have, you'd have to be crazy to make a prediction, um, in this market, how things mature over six and six and 18 month, six to 18 month timelines. But I think over five to 10 years, what we can, what I can tell you is that, I'd be very surprised if, you know, like 10 times or 100 times as many enterprises aren't using AI um, in their in their call flows than they are today.

• 41:42Craig S. Smith?

  Yeah, I I want to go back to these new chips. Um, and and you were, yeah, uh, Nvidia has has a lock because of CUDA. Uh, but on the inference side.

• 41:58Craig S. Smith?

  Yeah, I guess you're you're deploying models, but uh cerebrus and Samba Nova and Grok, they all have their own clouds of, you know, pending a time when the hyperscalers sort of install their chips if that happens. Uh, and and the for for an application that's looking for inference, it can hit those clouds through an API. So they don't have to fiddle with uh with

• 42:33Speaker 00

  Auto scaling and what? Yeah, exactly.

• 42:36Craig S. Smith?

  Right. But so

• 42:40Craig S. Smith?

  how how does that relate to Base10? I mean, can it? Yeah.

• 42:45Speaker 00

  I I had a really good question. So like, look, like, at some point, you want more customization. And like, you know, what you don't want in your production application is being throttled by Grok's own scaling limits or cerebrus's own scaling limits. I think that's where Base10 becomes really important. It's like, hey, you know, you get started with the fastest, easiest way to run these models. Yeah. And that might be through a OpenAI, it might be through a Grok, it might be through a cerebrus. Um, you know, you get a lot of tokens per second, it'll be fast. But all of a sudden, you know, you, when you are running your model and you just scale that 100x, you know, you really need a lot more control and and options in terms of where that compute lives and a bunch of workflow tools that sit on top of that. Uh, and that's what that's where

• 43:29Speaker 00

  I think that transition from off the shelf fast APIs to something like Base10 happens. When you have control, when you need developer workflow, it's like, imagine, imagine giving your your web application to a third party and have it just be a black box. You're not going to do that. You need observer, you need control. You don't want to be beholden to someone else. And that's really where I see that transition point. You know, we say that to customers all the time. It's like, hey, sounds like you do the API right now and not a Base10. Yeah, you don't have that scale yet, or you're not at that maturity with something like Base10 makes a lot of sense. Um, let's be design partners and help you through this transition as you get through it. Um, but does that answer your question in terms of like that trade off? Like when you need, when you need something off

• 44:12Speaker 00

  the shelf, and when you need something that's a bit more custom bespoke and give you more flexibility and and the knobs to turn?

• 44:19Craig S. Smith?

  What is uh scale to zero functionality?

• 44:23Speaker 00

  Yeah, it goes back to auto scaling, which is like kind of what I said to you, which is like, uh, a lot of folks, they don't need GPUs all the time. So we give you the serverless type approach where you can uh, basically use, use the GPUs when you need to or compute when you need to. And when the models aren't being used, it just turns off. So you don't get in charge. Now, this only works if you have really, really fast cold starts, which is like, when I need it, you know, it it spins up very, very quickly. And, you know, we've done a lot of work on that to make that as fast as possible. So that if, you know, at times, you know, the wall time of a mo- depending on the size of model can be sub ten seconds from the time where you're not paying to its consuming resources running at scale.

• 45:04Craig S. Smith?

  For people to to to use space ten, they just go to the website?

• 45:09Speaker 00

  Sign up. You could you could sign up to talk to, talk to us. Honestly, if you send me email at two in at base ten dot co, I'd love to help. You know, and I I'm happy to push you towards the right people and get you started. Um, but you know, the, once you go to our website, it'll be very, very easy to figure out how to get started. And if you fail at that, and it's not not as easy as I'm I'm selling it to be, send me an email and I'll I'll figure you out.

• 45:33Craig S. Smith?

  Yeah, and it's base ten dot com. Is that right?

• 45:36Speaker 00

  Yeah, I I we have we have base ten dot com and base ten dot co. Um, so yeah, so, you know, we we acquired the dot com later than we wanted at that point. We already have the yeah.

• 45:48Craig S. Smith?

  Is there something that I haven't covered that that you think listeners should know?

• 45:53Speaker 00

  No, no, this is this is very comprehensive, like, you know, de- depending on who you are, look like open source models are going to be a big part of the future. And we'd love to figure out how to help you use them in your applications. And, you know, that's, you know, we're we're hap- happy to help as thought partners, as a software providers. Um, and yeah, hit us up. But otherwise, Craig, this was fantastic. It was very comprehensive and thank you for having me.