Self-hosting & scaling models

We’re excited to have Tuhin join us on the show once again to talk about self-hosting open access models. Tuhin’s company Baseten specializes in model deployment and monitoring at any scale, and it was a privilege to talk with him about the trends he is seeing in both tooling and usage of open access models. We were able to touch on the common use cases for integrating self-hosted models and how the boom in generative AI has influenced that ecosystem. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Tuhin Srivastava – GitHub https://github.com/tuhins, X https://x.com/tuhinoneChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Baseten https://www.baseten.co/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 0:58Chris Benson?

  Doing very well today, Daniel. How's it going?

• 1:00Daniel Whitenack?

  Oh, it's going great. I spent the afternoon in sort of a brainstorming session with a couple of our team members here at Prediction Guard, and it was, it was a ton of fun. So talking about a lot of prompt engineering things and how different models perform and that sort of thing. So it was a good time.

• 1:18Chris Benson?

  I'm glad you're doing that because you know what? I just want things that just work, you know? I don't have to think about it. I'm glad you're thinking about it. I think we might have someone else to talk to who knows how to make things that just work.

• 1:30Daniel Whitenack?

  Yeah, yeah. Well, a lot of the models that we're running sort of just work for us in terms of inference because we're, we're hosting some of our models in Base10. And we've got Tuan joining us from Base10 today. How are you doing, Tuan?

• 1:46Speaker 03

  Hi, Dan. Hi, Curtis and Nestor. Nice to see you guys again. Thanks for the kind words, Dan.

• 1:50Daniel Whitenack?

  Yeah, yeah, for sure. Well, it's exciting to have you actually back on the show because it was, I believe, I looked it up, it was like May, June 2021, when we recorded and released the last episode with you. So how are you doing? What's, what's new? And how is Base10? How's the ride been?

• 2:12Speaker 03

  Yeah, it's been crazy. I feel like the last, I think it was like May 2021.

• 2:18Chris Benson?

  That's like a millennium in AI time, you know?

• 2:20Speaker 03

  Oh, my God. If it actually does feel like it was a different job ago.

• 2:25Daniel Whitenack?

  Yes.

• 2:25Speaker 03

  It feels like the job before the last job.

• 2:30Speaker 03

  If that makes sense. No, I think, being crazy, I think the last two years for everyone here have probably been a bit of a whirlwind. And you guys are pretty on top of current things in machine learning and AI. And I think, I imagine, just like for you guys, it's hard to keep up at times with what's going on. I think, you know.

• 2:49Chris Benson?

  We only do one show a week and, you know, it's getting, we almost need a daily show. There's so much content now.

• 2:54Daniel Whitenack?

  It's not enough. Yeah. Don't give our listeners ideas because I don't know. I don't know if I can do a daily show. But it is a lot. And I think, so I'm looking back previous at our, at our episode. And last time we talked about sort of the easiest way to create ML apps. That was kind of part of how the conversation was framed. And I know just from working with you and, and talking with you as friends, that a lot has changed and you've seen some things within how people are deploying machine learning, AI systems that now Base10 is really focused on.

• 3:33Daniel Whitenack?

  Could you give us kind of like the high level view of Base10 and the type of problem, the type of solution that you're offering?

• 3:41Speaker 03

  Yeah, yeah, yeah. I think, I think it's just worth probably pointing out, you know, before I go into Base10 specific things like, what are the, like, the key things that changed since we last talked? I think, you know.

• 3:51Daniel Whitenack?

  Sure.

• 3:52Speaker 03

  If you think of like the year of like 2012 to 2020, you know, data scientists were the ones doing a lot of machine learning. I think, you know, that's changed for a number of reasons. I have a lot of thoughts on that, but probably the bigger changes are, you know, the emergence of good open source models. And I know you do a lot of work with that and, you know, like we've seen hugging faces of community evolve into like really, really vibrant place where if you want to get a sense of how fast things are moving, it doesn't take long to, you know, take screenshots of. Hugging face every Monday morning and see how the trending is changing and you'll see that things are pretty different every week.

• 4:27Chris Benson?

  I don't know if it was Daniel that said this or another friend of mine, but the analogy was that, you know, hugging face has become to AI, kind of what GitHub has always been for software developers of the last decade or so. It's just, you know, it's the place to go to find it. Anyway, I didn't mean to interrupt you there.

• 4:42Daniel Whitenack?

  Yeah. And the good about it, but also the confusing too. It's like, you know, you have like random person clones model or copies model and uploads random version of that model that like maybe works.

• 5:00Speaker 03

  Yeah, a hundred percent. I feel like the game you have to play with hugging face is like, but does it run? You know, does the model, but does the model run? But, you know, I think open source emerged and I, and I think like stuff like Whisper showing up and, you know, some of these OCR type replacement models showing up, you know, they're probably the more interesting ones to me, not because what they do, but because they end up just solving a lot of open problems. You know, if you think about transcription as a problem of think about nuance and like how long they were working for that, you know, like literally 20 or 30 years of, of work, just kind of, all right, that's the sole problem. Now let's move on. Oh, we've, we've actually solved multi-language with the same model too.

• 5:43Chris Benson?

  Business models come and go, don't they?

• 5:44Speaker 03

  Yeah, yeah, it's wild. And I think like the last piece is just around, you know, like the chat GPT moment for AI, interesting for a number of reasons. I personally think as someone who built infrastructure, that it's most interesting because if you want to call that the iPhone moment of AI, I think it's a bit different to that because it's so early in the journey. It's like if the iPhone showed up and we were all using 5110 from Nokia, the world would be very, very different. And I think because consumers and developers, their first taste of machine learning and AI was through chat GPT and GPT APIs. The stakes are just, you know, it it's just harder to build something good. Like, you know, people don't want to use a model that takes 12 seconds to run.

• 6:30Speaker 03

  You know, like, high-speed production inference is, you know, taken for granted when you're using um this model. And then when you kind of combine that with, okay, open source models need to be run somewhere. All right. We we personally think that's like, okay, well, there's a massive infrastructure opportunity that, um, well, may maybe not even opportunity, that just say, a fact that a whole new stack will be built to support uh, models to be able to power these end user experiences. And I think like, that's kind of the core insight. Kind of going into base 10 and talking a bit about base 10 around like what's changed is that, you know, we we kind of, two years ago, when we were talking about data scientists, we want to talk about engineers. I think that's pretty key to our story, which is that,

• 7:14Speaker 03

  I think we came to the realization that every engineer needs to grapple with machine learning now, as opposed to maybe a smaller market that was data scientist. I think going from smaller models that run in memory to larger models is another big, you know, focus change we've had. I think there's a bunch of language stuff and N L T K stuff. And, you know, you were doing all that work, Daniel. But for the most part, everyone's using, you know, psychic one. And that and psychic one models for the most part run in memory.

• 7:41Chris Benson?

  On CPUs.

• 7:42Speaker 03

  On CPUs.

• 7:43Chris Benson?

  Yeah. If you think just in that time period, the the amount of maturation, you know, that's occurred in this industry, it still bought you something a second ago, which I it kind of hit me. And that is most people out there in the general public, you know, are really just getting into this, you know, with chat G P T and stuff. And we've come a long road already. I just, I'm listening to you and it's amazing how far we've come in such a short time.

• 8:06Speaker 03

  Yeah, it's insane. Hundred percent. And I think that, you know, going from small models to large models as well, it's just like, that change kind of happened pretty quickly. Yeah. I saw one who did a bunch of work with small models, like, they have their time and place, but they're just not that fun anymore.

• 8:22Speaker 03

  Like, something that runs in memory just doesn't give you the same feeling as. And then I think the last one is just so much of the stuff that was happening with machine learning outside of Fang, I'd say, was like Fang had some production use cases around like ad serving and search and and whatnot. But outside of that, it was mostly just internal workflows. You'd go and work on fraud and content moderation and recommendation systems. And I think, you know, going from, hey, every product is going to have some sort of machine learning in it. Um. Every existing product will will definitely and ninety percent of new products will be built with the with a new pillar. That is machine learning AI, which wasn't the case, I think, two years ago, which is just crazy to think about.

• 9:03Chris Benson?

  Yeah, it's completely changed in that time.

• 9:05Daniel Whitenack?

  I think one thing that was brought into focus for me while you were talking was that as soon as you make this leap to kind of larger models. And you make the leap from some closed API that's very fast to maybe running your own model. There's two things that become like immediately clear. One is the infrastructure challenge around that, which I think is. The workflow around that and the model hosting, you know, base ten, of course, is an expert in that. And the other side of that is like the product. Sort of concerns around running these things, which I feel like, you know, we always have great conversations to and because like you're on the one side of that.

• 9:49Daniel Whitenack?

  And I'm probably on the other side because, yeah, like when you're using chat GPT or even the open AI API, they have layers of. You know, protections on the prompts and like on the output, they have, you know, filters to make sure they're not. Responding in certain ways. And there's all these product concerns that people don't think about. And then they take like a llama two model or something. They run it and then there's like, oh, this doesn't respond like this is not a product, right? And so like the infrastructure is a piece of that, the ability to iterate very quickly with models of a variety of types. I think is part of that infrastructure challenge. How do you see that infrastructure piece of this kind of playing out?

• 10:30Speaker 03

  Yeah, ma- maybe just before I say go into that, when you're saying that reminded me of like, I bought a drone in twenty fourteen. Um. And I was so excited, like DJI Phantom three, I was so pumped and, you know, I flew it around a bit. And they had basically had autopilot mode where you didn't have to do anything. Um. Like it kind of just stabilized itself. And then there was this button that said manual mode and it has like all sorts of warnings. And I remember saying, like, being like, oh, how hard can it be?

• 10:55Chris Benson?

  Right. Oh.

• 10:57Speaker 03

  I really like joking, like, I was in a safe space, but I put it into manual mode while I was up in the air and it just fell to the ground very, very fast. I don't know, Chris, do you have some experience with this?

• 11:08Chris Benson?

  Well, I'm, I'm in aerospace professionally, so I know a little bit about that. And yes.

• 11:13Daniel Whitenack?

  Chris, weren't you like the TV host of the drone racing competition?

• 11:18Chris Benson?

  I was.

• 11:19Daniel Whitenack?

  Or something like this?

• 11:21Chris Benson?

  Yeah, a few years ago, uh, yeah, I was, I was one of the hosts of the first drone racing league. They had a championship series and they were using, instead of you as a human, they were navigating obstacle courses and stuff like that. And what I learned through that experience is when you have the, how much autonomy is required to make even small things fly well.

• 11:44Speaker 03

  Yeah.

• 11:44Chris Benson?

  And yeah, so I, I sympathize with you for going on manual there. Oh, no, don't do it. Don't do it.

• 11:49Speaker 03

  And the analogy I think holds here, it's like the close, the close API, it seems so great. And then you, you see something like a Llama two or a Mistral and you're like, okay, I'll just rip and replace this. And it's like, nope, that's all going to work for a number of reasons. And I think that's kind of how we see about, like, think about infrastructure based sandwiches that running models in production is very, very difficult. It's difficult for a number of reasons. So I can decide, but, but like from like a user requirements perspective, like latency and throughput paramount, um, costs are what something you want to optimize. Data privacy is, you know, a whole nother beast. Um, security comes into play. Then orchestrating this across a bunch of different hardware, orchestrating this across clouds becomes a problem.

• 12:32Speaker 03

  Benchmarking these things isn't easy. And that's even before you go into all the evals and the, you know, the kind of like the guardrails you want to put around this thing to get it running. And I know some of the stuff that you think a lot about, Dan, so.

• 12:45Chris Benson?

  Just as an extension of what you're saying, and Daniel, you mentioned it at first, but if you could also talk a little bit about what the difference is about just hosting, like, like just having the model hosted and kind of the idea around what you have to put around it as a, to make it a product. Cause I don't think most people talk about, I don't hear a lot of conversations about that. And it's a big set of gotchas on what to do, you know, and kind of what's involved in that. What's your thinking around that when when people are looking at doing that?

• 13:10Speaker 03

  I could talk well about the first one and I have thoughts about the second. Dan's going to be the expert around the latter piece of that. But to get a model running in production, there's actually a ton of work you need to do from an infrastructure perspective. But and this is before we talk about the workflow stuff. You know, you need to figure out how to containerize this thing and get this image running like, you know, as we alluded to earlier, like just taking your model or parking fits and expecting it to run is not a thing. You know, there's a bunch of requirements that these models have. There's quantization of inside the code. There is um different base images that they might need based on Torch and PyTorch and um Python versions. Uh and then, you know, you can really find yourself in a bit of a pickle just trying to dockerize a model.

• 13:51Speaker 03

  So first thing is you need to figure out how to get this in some sort of containerized form so you can run it elsewhere. Once you have that, the truth is that you need to spin up some sort of service that can deal with variable traffic. Um and the reason why that is is a traffic, you know, these things tend to be expensive. They tend to be bound by compute. So if you get smashed with a bunch of requests, it's not like you can just have one model and it will queue out. There will timeout, the whole thing will slow down, your product won't work. So you need to figure out how to scale up and down with traffic. And then you need to figure out all the security concerns that come with all that. That's just on the serving layer. Now you need to start thinking about the workflow layer that sits on top of that. And I think, you know, version management is, you know, non-existent for the mother. Hooking it up into C I C D, it's to really treat this like a

• 14:34Speaker 03

  a service or a microservice or how's putting your model behind an API, you need observability um and logging. Another whole um set of features. And what you realize is that taking a model and getting it working in production behind a reliable, secure, performant API and maybe cost efficient. As someone who's done this myself, um and someone who's built a company to try to abstract this way, it it is easily for one model, couple of people have had counts work for a couple of quarters if you're lucky at scale. And I think that is the most efficient organizations that can hire people with Kubernetes uh Kubernetes experience to be able to do this.

• 15:14Speaker 03

  And so that's the type of things that we try to abstract away from our users where it's like, you know, you figure out the Python code. We'll figure out everything else and we'll give this model this first class treatment. Um so that you can version around, that you can log around it, that you can observe it um and you can call it. But you don't need to think so much about that. You get that. That's now to the point where you have something behind an API and ready to consume. Now, there's a bunch of stuff that needs to happen to make sure that, you know, it doesn't start saying random stuff that you protect against hallucinations. It's not just ingesting PII all the time. Dan can probably just talk really quickly about that as well, I'm sure.

• 15:53Daniel Whitenack?

  Yeah, I think part of the reason why I'm always excited to talk to Tuan and his team at Base10 is because they are experts in this. All of those layers that we just talked about, I was actually on a call with someone the other day and we were talking about like spinning up some microservices or something. And I think my comment was like, I just really don't want to care about Kubernetes because I don't want to like wake up lying in a ditch crying in the fetal position. Like, that's how I view that like whole world. So props to you and your team for dealing with that side of things. I think that's what's allowed us then on like the prediction guard side in a lot of ways to like bring up a model quickly.

• 16:37Daniel Whitenack?

  And then have the time to think about some of these other things. Tuan, I don't know if you can comment on, like, I have my own perspective from trying to run models for my company. But it would be interesting to hear the perspective of different personas that are coming into Base10. Like, are they people that are sort of application developers that are, you know, not infrastructure people? Are they like data scientists? Like, what are the, what are the types of people that are coming to Base10?

• 17:16Daniel Whitenack?

  And I, maybe along with that, like, as you mentioned, closed APIs are getting used a lot. But still, people are coming over to think about like hosting their own models. One question would be like, why? Like, who are these people and why?

• 17:33Speaker 03

  Totally. I'll answer the second one first. No, I think I'll add them together is that it is more and more just engineers. I'd say like, I don't know if there's any, there's any distinction now between like, I'd say it's less and less data scientists, traditional data scientists. It's more and more people with some ML exposure, product engineers, infrastructure engineers who have tried to build it themselves and have really felt the pain. I think from a product engineering perspective, like why want people want to use? Open source APIs, I think cost is one big thing is that OpenAI has to stack up over time, I think, or Anthropic. I think the other one is data privacy and security is

• 18:14Speaker 03

  that you don't want to just be piping over all your data to OpenAI today, and especially when you start to talk about B2B use cases in enterprises. And I think there's like, probably the more interesting one is that there are just like a long tail of people working on weird models. People are fine tuning models, fine tuning OpenAI models is not great. You get a bit more control with that manual mode with open source models. And so it's kind of like the long tail of use cases, I'd say, are coming more and more. And these can be engineers, they can be machine learning engineers, it can be honestly like a lot of audio models, like different modalities that there's not that much exposure to with closed APIs and a lot of custom stuff as well.

• 18:55Chris Benson?

  You mentioned, you know, like shipping data over to OpenAI, and I have talked to gazillions of people who have that as a constraint in their businesses, you know, because the attorneys for the business are like, nope, you don't want to send, you know, your proprietary information and stuff over that. I guess you would not have that issue at all with Base10, would you? I mean, that's, that kind of goes away altogether when you're hosting in that way, right?

• 19:19Speaker 03

  Yeah, you have ownership of your data. We don't log any of that data. You're treating the model as just like a, like a map of input, the output and nothing else.

• 19:27Chris Benson?

  Yeah, that would really solve a lot of people's problems by taking an approach like that.

• 19:32Speaker 03

  Yeah. And I think the second piece there is that once you adopt the base and approach, you can then start to think about like self hosting, deploying it within your own VPCs. Like, you know, we have customers that deploy Base10 within their own AWS account and data never leaves their boundaries or their accepted boundaries.

• 19:50Daniel Whitenack?

  Yeah. And we've kind of, I think you framed the concerns that you're looking at with Base10 very well, these sort of infrastructure scaling concerns of hosting your own model. Could you maybe take a step back and just describe like? If I go into Base10, like, how have you architected the approach like to, "I'm an application developer, I want to run, you know, some random fine tune of Llama 2 that I've created somehow. What is it like for me? What does that look like with the way that you've structured this? And what's some of the thinking behind that in terms of the workflow? And how you want it to be for people so that they can treat, I guess,

• 20:34Daniel Whitenack?

  that model as a first-class thing that is a first-class asset in terms of what they're monitoring and logging, that sort of thing?

• 20:42Speaker 03

  Yeah, for sure. I'll go with just to make it easy and, you know, try to take away as much of the complexity, but, you know, maybe more importantly, it's still that you'd have a bit of control. You know, I think Base10 started as like a one-line-to-deploy model. It's like, you know, we don't believe that anymore. You know, either we think that, you know, actually having a little bit of structure around it gives you actually, um, a bit of structure upfront, gives you a lot more flexibility on a bit down the line. So we have an open source library called Trust, which is basically a, it's abstraction that if you write your model and you get kind of everything free. And so basically, you need to write two things. You need to write one Python class with a load function and a predict function. And this is vanilla Python code. It can sit within your monorerebo.

• 21:27Speaker 03

  You can specify requirements as you want. There's nothing Base10 about these files. You know, you could run them outside of Base10. I think that's very important for us as well. But once you write that load and predict function, it does two things. One, it tells us, "Hey, what are you trying to do here?" And, you know, we can load that up. And when we deploy a model, we load that function. When we infer, we run the predict function. But more importantly, within those functions, we allow you to compile stuff down. We allow you to kind of do the tricks that you need to do so that, you know, you still have that control. And, you know, within that, you can write preprocessing and post-processing functions that allow you to, you know, maybe like strip out some data, log something, mod to something. But really, it's still giving that control at the product and application level, while still abstracting out the thing we want with Trust.

• 22:13Speaker 03

  Once you have a Trust developed and you can go to trustabase10.co and check out a bunch of these, um, it's a pretty simple abstraction. And you can just push that up and we kind of give you all the workflow and version management around that deploy trust.

• 22:27Daniel Whitenack?

  Yeah. Could you speak then to like, that's like the prep kind of that goes into, "Oh, I've got my weird model. I'm writing this Python class. I'm going to deploy it on somewhere." And I know that like, one thing that I think is really cool, how you've made Base10 trust is, like you mentioned, it is open source. And so you can run, trust things and deploy in a variety of ways. One of those being like Base10's hosted infrastructure, which is of course easy, but it's also like generally a great, great sort of framework to package your models. But let's say that you do kind of go the Base10 route.

• 23:08Daniel Whitenack?

  You deploy this through the Base10 client to Base10. Could you kind of compare and contrast like, let's say I just tried to run my model in a fast API, API in a EC2 instance or ECS or like whatever that is in my cloud. What is going to be different about what I look at when I kind of look at my model in Base10 versus running this API somewhere else? And how does that make a meaningful difference? Or what are you trying to do in terms of making a meaningful difference for the day to day for people?

• 23:46Speaker 03

  Yeah, well, I think what you're doing is that, so besides, you know, you can run that model on Fast API. Great. You got this model. You give it an import. It gives you an output. Fantastic. Let's carry on. But it's like the depth of features and the creation of workflow, which are really important here. And so like, the depth of features is that, hey, if you can do that with Fast API, great. You're still going to have to set up auto scaling. You're still going to have to set up observability. You're going to have to set up logging and whatnot, hardware management. But I think the workflow is probably more important, to be honest, which is I do. We're creating a defined way for you to publish new versions of this. If you want to A/B test two models, you can have two models running at the same time. You know, it's really that the removal of boilerplate and the addition of some workflow.

• 24:29Speaker 03

  So that, you know, when you were deploying this in production and you need to roll back a version, you don't need to go and scramble to find that Fast API file that you're using before. We've all been there before. And so it's that creation of workflow is probably, I think, what a lot of our customers probably use us for. Besides, you know, I think the production grade inference is a given, but a lot of the differentiation comes from that workflow.

• 24:53Chris Benson?

  Just to totally boil it down, you're saving them a lot of work right there.

• 24:57Speaker 03

  Yeah, 100% yeah.

• 24:59Chris Benson?

  There's a lot of kind of grunge work. It reminds me of Dan liking to do his data massaging that I'm always teasing him about. But joking aside, you're basically saving us all sorts of work so we can get into production faster, get it up and running and, and know that it's production grade all the way through with a minimal amount of effort and know that it's just there.

• 25:19Speaker 03

  100% like we're working with this customer right now. This was only that they were a, this is a pretty late stage startup, raised hundreds of millions of dollars, AI native product. They've got a team of four AI infra people to manage this. And they've been working on this for about two years. You know, we were able to replicate and get a more performant API up and running in two days.

• 25:42Chris Benson?

  Wow.

• 25:43Speaker 03

  I think that is kind of what we are trying to, it's the performance, the workflow, it's the maintainability, but it's also just the speed to prod.

• 25:51Daniel Whitenack?

  I don't know how many of your users do this, but the fact that this might be sort of revealing about me as a person and also reveal some utility of Base10, but like I can literally, like last night I'm sitting on my couch and I can log into Base10 on my phone. And that just changed the auto scaling from like two replicas to like five max replicas and the timeout. And like all those things of like the auto scaling of my Lama 2 fine tune from my couch, like in between Halo games. So that was, that was.

• 26:28Speaker 03

  That's terrifying.

• 26:30Daniel Whitenack?

  I don't know what that reveals about me as a person, but certainly like that ease of use, I think is really interesting. It, it's like that, uh, that proverb, when you like try to solve a problem and then you're like, I'll, I'll solve this with RegX. And then you just have another problem to solve. It's like, kind of like that. It's like you want to deploy your model. And then you want to deploy it with Kubernetes and then you have like a whole nother problem to, to solve and like solve the auto scaling stuff and, and all that.

• 27:01Speaker 03

  And then like, I think on top of that, it's just all the SRE work that you have to do for that service. Like, you know, what happens when it goes down? You know, what happens when you need to migrate something over? Uh, what happens when there's a new GPU you want to use? There's just so much. I feel like we've really turned the corner and I think stuff like AI and ML that really has helped here because people want to move fast is like I feel like we've put the build versus buy debate a little bit to rest for a bit. Where we just don't hear it as much. It's like, hey, we we want to build it ourselves. Like, people are just like, we want managed solutions.

• 27:35Chris Benson?

  Yeah, you got to go fast these days because if you don't, somebody else is going to get there first and and you're not going to have a business.

• 27:42Speaker 03

  And the market is remarkably talent constrained. Like, yeah, again, like Dan, you're saying this and this makes me happy because like just like, so you know, like Dan's background is in data platform. And dealing and dealing with all of these things, you know, like it is.

• 27:54Chris Benson?

  I've cried myself to sleep.

• 27:56Speaker 03

  In fatal position.

• 27:59Speaker 03

  And so really, it's just an ease of use and like it's ease of use and the ability to scale with you like and that's probably like the two things which we try to bring to our our customers. And I think just even outside of base ten is probably the the biggest opportunity, I'd say in machine learning infrastructure right now is think of all the user stories that are important now that weren't important 12 months ago and maybe just take a long, a slightly longer term view than what can I build around open AI APIs, which is like stopped a lot of attention. These are all places where base sand is thinking about going or like, you know, we we will partner with people are doing. If you think about people, the emails layer, think about people of the fine tuning layer, you think of people at the training layer, at the observability layer, the logging layer, it's like

• 28:42Speaker 03

  the entire new stack here to be built and that's a massive opportunity. It was a really interesting company to look at, to be honest, because they were training their models, they were helping people train them on the remarkably early. But like the value is very, very clear to a pretty sophisticated buyer in data breaks. And so to any folks building tools around here, like so much value to be added. It's greenfield.

• 29:05Daniel Whitenack?

  I have another question to ask you because you are living in kind of that world of model deployments and the various ways that people are doing this. People are fine tuning their own models or just using open models. I'm wondering how you see the trends going. So you've already talked a little bit about open models being available. Small to big models and how people are hosting them. There's tons of people that are also exploring this area around running models kind of at the edge or in various environments or on laptops. And also people that are exploring kind of, you already mentioned quantization and running

• 29:48Daniel Whitenack?

  models on CPUs potentially instead of GPUs. I'm curious for, as like someone who hosts like a lot of models in the world, like what are you seeing in terms of this trend? Like, cause you hear about these topics, but I don't really have a good sense of, like, obviously people are exploring those things, but are those people just like extra loud on the internet? Or like certainly there's use cases, Chris knows them well for running certain things at, at the edge. But for many people out there that are like maybe building a SaaS platform or something, you know, that's less relevant, although maybe the costing, you, you mentioned cost

• 30:31Daniel Whitenack?

  optimization as well around like that sort of thing. So yeah, how are you seeing as someone, I guess my question is, is someone who hosts a lot of models for a lot of people, what, what are you seeing as people's concerns, both in terms of that costing, optimizing models and like deployment targets, I guess?

• 30:51Speaker 03

  Totally. I think what we're seeing is this is remarkably early, to be honest, like there are opportunities, like people are deploying stuff on edge, but I think there's not enough of them today. Just kind of think about like a generalized opportunity there. And so I think, you know, there's a company called OctoML that started with edge deployment and then think they were just like, let's move because that's where the opportunity today is. I think all the stuff that's happening around running these models on less and less hardware or optimizing them in X way or Y way. It's remarkably intriguing, right? It's pretty crazy that we can get a model that we can barely run on, you know, the biggest GPU we can find and suppose I've figured out how to compile it down or rerun it with C++ kernels and all of a sudden it runs anywhere.

• 31:36Speaker 03

  That's pretty fantastic, but I, I do think that, you know, we're still in the research phase there. We're in the experimental research phase and like, you know, we've seen a lot of people deploy those models. Yes, we've seen a lot of people play with those models, see a lot of interest in those models, but I can't really think of too many examples of people running those models in production just yet. But it seems inevitable that over time.

• 31:57Chris Benson?

  Oh, it will happen.

• 31:58Speaker 03

  That is the arc that we are on.

• 32:00Chris Benson?

  Yeah.

• 32:00Speaker 03

  These models are getting smaller and smaller. Yeah.

• 32:02Chris Benson?

  When I'm not on the podcast, I'm in a world where it's all about things moving around in time and space. And a lot of those things will have AI capability on board going forward. So that's, I agree with you. There's a lot of research going on and no one, it's not a solved problem. And when, you know, there's not a set of best practices yet.

• 32:21Speaker 03

  Yeah.

• 32:21Chris Benson?

  If you will, but it's an area that is majorly ripe. I'll be really curious to see if, if you guys or another company is able to leverage all the expertise you've built up and experience you built up in the cloud and kind of move out into those areas.

• 32:37Speaker 03

  Pretty around device. Yeah.

• 32:38Chris Benson?

  Yeah, there's millions of devices out there just waiting for you.

• 32:41Speaker 03

  Yeah. I'd say like the, the challenge there is just around, you know, how, I'm guessing Lockheed's devices are a bit of a snowflake and you know, we can't, you know, you can't build for one type of device and just go and apply that to the next device there. I feel like there's probably some generalization that needs to happen at the OS layer before we can do that. But I am also completely uneducated on edge stuff. So you probably have a lot more to say there than I do.

• 33:08Daniel Whitenack?

  Well, and it's interesting too. I sort of asked this in a leading way because one of the people that we're talking to, I'll kind of genericize this, but they run some equipment at the edge in manufacturing and they have like a hub at the edge, which is air gapped, which doesn't talk to the internet. But their whole like next generation of things is going to be internet connected. And when I was talking to them about like doing some things with large language models in that environment, you know, essentially where we got was, hey, well, It's gonna be more hassle for you to figure out some of these like model optimization things and all of that than to just like set up an API in Base 10 or something like

• 33:53Daniel Whitenack?

  that and and just connect it out if that's where you're headed anyway. And it's not like a military security concern like so one of these situations. So yeah, I think we're probably see both and but for a lot of people, it's like kind of how I've categorized these things in my mind is yeah, some people will want to run Kubernetes in their own infrastructure and they have the expertise to do that. And if that's you, then like, great, you're one of maybe a few people on the planet. I don't know.

• 34:23Speaker 03

  Good on you.

• 34:24Daniel Whitenack?

  Good, good on you. And similarly, like with Chris, if you're running a lot of models on the edge, which I know certain people are and in certain industries, it's really important. That's great. And that expertise will be there. But I think for the, I don't know, my sense is that for a lot of majority of people, like separating out that infrastructure concern of model hosting is really, really a useful way to think about things. So I don't know, we'll see. I'm always bad at predicting the future, but.

• 34:52Speaker 03

  We talked to this one customer, they were saying that for some reason, their CEO bought a bunch of GPUs and they literally have machines in the office. They're like, oh, well, we're gonna do the, I think Amazon has like the Kubernetes anywhere or something where you can basically.

• 35:09Daniel Whitenack?

  Like a hybrid sort of thing?

• 35:11Speaker 03

  Yeah, I've, you know, we ran away from that opportunity, as suffice it to say. But you know, there are, I think these people are thinking about these problems. I don't know if there is a, a solution here just yet.

• 35:24Daniel Whitenack?

  Yeah. Well, as you kind of think about, so obviously, and I know just from our discussions, like you're helping a lot of people and really significant use cases in the space with already with what you're doing with the infrastructure side of model hosting. But as you look to kind of the next, I can't even say like the next. Years because things move so quickly, but as you look towards the future and like, what is not yet solved on the infrastructure side with model hosting? And what are like you and Base10 really excited to dig into? What comes to mind and what are you thinking about?

• 36:05Speaker 03

  Yeah, I think within the containers, like that layer is gets very interesting. I think, you know, BLM, which I'm sure you've played around with, TGI, which, you know, they're both great. I still, I, you know, I think they're still far from ready for prime time just yet. BLM, TGI and TRT L M that Nvidia just put out. I think there's going to be more and more of these frameworks and supporting these frameworks. I think it's going to be very, very key for us and what we're really excited about. So we're going deeper at that layer. So you can kind of bring your own framework on your container and really benefit from that. So, you know, we're going to have first class support for TRT L M pretty soon. And we already do it for TGI and BLM. And I think that side is pretty interesting.

• 36:49Speaker 03

  I think, you know, we have a big launch coming up. I'm happy to talk about right now, actually, but around multi cluster. So that's basically being able to one, use your own compute to bring your compute to Base10. So the, the control plane sits on Base10 and the workload plane considered GTP, Azure, AWS, or some combination of the three. And then we'll keep adding clouds to that. And so I think that's very, very exciting, especially in the enterprise that because, you know, people want self hosted.

• 37:19Daniel Whitenack?

  That's huge.

• 37:20Chris Benson?

  It is. It's really nice.

• 37:22Speaker 03

  That's really big. And then kind of just beyond serving. And I think we're really excited that, you know, we've already had one for a, and we learned a lot and, you know, we, we, we end up retiring, but we're going to, we're going to get into fine tuning at some point. I think like that's, we keep seeing like, just like I said about kind of like the edge device stuff and the compilation stuff is that. Mind tuning is still an art. I'm a little bearish on like APIs that say, give me your data, I'll give you a model. Like, I think you need more controls. As someone who built an API that said, give us your data, we'll give you a model with Blueprint. You know, I think you need more control. You need control of your base models. You need more control over even the fine tuning scripts to customize that model. We'll start to think about that very soon. You know, we're already doing a bunch of work with customers

• 38:07Speaker 03

  to make sure that we're marching in the right direction. So I'm very excited about that, which is that, you know, over time, Base10 becomes this place where you can run your models great, but then you can also start to collect data sets around your model. Just imagine if you could just give your model to Base10 and then, you know, you either opt in and we basically write all your input and output data to S3.

• 38:27Chris Benson?

  That's beautiful.

• 38:28Daniel Whitenack?

  Yeah, like essentially a level of caching for model inputs, outputs.

• 38:33Speaker 03

  Exactly.

• 38:33Chris Benson?

  Yeah, the multi-cloud thing really will be big for enterprise, by the way, just to that point, because I think most enterprises across many industries are recognizing that their future is a multi-cloud world. It's no longer tied to one. And if you have Base10 able to do that hosting and run a control plane, and you can deploy into any of the cloud clusters that you happen to be, and maybe different parts of the company emphasize one or the other, then that takes a lot of challenge that they're currently facing out of that. So it's pretty sweet.

• 39:03Speaker 03

  I mean, also it's opened up opportunities, right? Especially in a GPT you can train world.

• 39:06Chris Benson?

  Yeah.

• 39:06Speaker 03

  You can get them from wherever you want. And then, you know, I think once you have data sets as well, then fine tuning just becomes obvious, which is like, okay, now I can I fine tune this and you're like, or maybe it's even like, hey, hook up your OpenAI endpoint using Base10 so we can collect that data set. And then we can create that fine tuned Mistral or Llama 2 to keep you the model you want. So I think there's a lot of interesting things along that whole stuff. And as I said to you guys earlier, there's so much opportunity here for people building in the tooling layer. Brand AI and ML. It's very exciting overall.

• 39:38Daniel Whitenack?

  Yeah, well, we appreciate you taking time out of doing great work in that layer to talk to us and share with our listeners. This is a great conversation and hopefully we have you on the show in less than three years from now. But if not, at least three years from now, hopefully sooner. So thank you for joining us again and giving us an update and some insights around this and really appreciate what you all are doing and appreciate you taking time.

• 40:09Speaker 03

  Of course. Thank you. Thank you so much for spending time on the show. I know it's hard to be honest.

• 40:22Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.