Serverless GPUs

We’ve been hearing about “serverless” CPUs for some time, but it’s taken a while to get to serverless GPUs. In this episode, Erik from Banana explains why its taken so long, and he helps us understand how these new workflows are unlocking state-of-the-art AI for application developers. Forget about servers, but don’t forget to listen to this one! Sponsors: Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with extended episodes, make the ads disappear, and increment your audio quality with higher bitrate mp3s. Let’s do this https://changelog.com/++! Featuring: Erik Dunteman – Website https://www.erikdunteman.com, GitHub https://github.com/erikdunteman, LinkedIn https://www.linkedin.com/in/edunteman, X https://x.com/erikduntemanChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Banana https://www.banana.dev/ - Scale your machine learning inference and training on serverless GPUs. Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 0:57Chris Benson?

  I'm doing fine. It's been interesting times, though it's not what we're going to be talking about today. Been watching the showdown between, you know, Google and Microsoft over ChatGPT and Bard, and you know, things are happening as we're recording this, so.

• 1:15Daniel Whitenack?

  I thought maybe you would have been too distracted with the new Harry Potter game.

• 1:19Chris Benson?

  I, well, there is that. Yes, we all have our secret little things that we do to keep entertained.

• 1:27Daniel Whitenack?

  Yeah, yeah. Well, I forget one of our recent guests brought up that quote of like, "You don't need to do machine learning like Google." And you were talking about like Google and Bard and all of these things. And when you think about those things, you think about like, oh, these like data centers full of GPUs and these huge supercomputers that they've got at their disposal to do things, which isn't the type of GPU infrastructure that most practitioners have access to. And that happens to be maybe the topic of what we'll get into today a little bit.

• 2:03Chris Benson?

  Excellent.

• 2:04Daniel Whitenack?

  With Eric Dunham, and founder of Banana, serverless GPUs. Welcome, Eric.

• 2:09Speaker 03

  Thank you. That was a beautiful lead in. Definitely want to help people get the Google level infrastructure without that level of effort. So glad to be here.

• 2:17Daniel Whitenack?

  Awesome. Yeah, well, we're really excited to have you. I have to say, I did spin up a model in Banana leading up to this conversation. So I'm pretty excited to talk about it. But before we get into the specifics of all the cool things that you're doing, I know that our listeners, like I say, they're probably very familiar with GPUs and like why they're important to AI, machine learning, modeling. But maybe they've just heard of serverless as like this cloud thing that is like a thing that people do in the cloud. They do serverless things and they've never thought about like serverless GPUs. Could you just like step back for a second and describe like,

• 3:00Daniel Whitenack?

  first off, like for people that might need it just a very brief intro, what do you mean when you say serverless? And then kind of take us into like serverless GPUs? Is that a new thing? Has that existed before? Curious to hear your perspective.

• 3:17Speaker 03

  So I love your specific phrasing. What do you mean when you say serverless? Because serverless is one of those terms that nobody has really pinned down exactly what it defines. Our working definition is this idea that when you need capacity, when you need servers to handle your requests, when you're in periods of spikes, in surges of use, you have more servers. When you have less use, you have fewer servers. And when you have no use, you have zero servers. And the idea of this is to make it so that you as an engineering team and as a product, don't need to think about your compute as a fixed cost. It allows you to essentially view it as pretty much per request phase.

• 3:58Speaker 03

  You go, funny enough, serverless really does mean servers running under the hood. But the less is that you just don't need to think about it. You think about it less. Happy to dive into what the details of that mean in regards to GPUs. But serverless has been around for about 10, 15 years. I don't know my exact timelines, but it's been a concept within CPU-based compute, serving things like websites, backends. And people have been wanting this to exist for GPUs for a long time and nobody's really cracked it. And that's the challenge we've been working on.

• 4:34Daniel Whitenack?

  I know that, like, you talked about websites, about like backends, that sort of thing. Just in general, when we're talking about serverless GPUs, in your mind, is the use case that you have mostly on like the inference side? Or on the like training side of what practitioners are doing? Or is there a little bit of both?

• 4:57Speaker 03

  The vast majority, at least from what we've seen, is on inference. And I think inference is where the value of serverless comes in the most. There's other tools for training where it's not as latency constrained, where you could use other infrastructure orchestration tools. But for specifically inference, serverless is one of the keys to the kingdom. If you could really do serverless well. So we just as a team have chosen to focus mainly on inference, real-time inference. So if there's a user at the other end waiting for a response, we're the ones responsible for making that response happen quickly.

• 5:32Daniel Whitenack?

  Gotcha. And why has it taken so long to get to serverless GPUs versus serverless CPUs?

• 5:40Speaker 03

  One of the biggest problems in serverless is what's called the cold boot time. Cold boot, as in you don't have servers running. A request comes in, that request coming in triggers a server scale up, going from zero to one, and then one to many. And the time it takes in order to get resources provisioned and ready to handle requests in CPUs can take, you know, a couple of seconds on a platform like AWS Lambda could take multiple seconds, maybe 10 seconds for a cold boot. And that's just simply spinning up the environment, spinning up a container or a micro VM, whatever they're running, and getting an HTTP server ready to handle that particular

• 6:22Speaker 03

  call or set of calls for the user before then shutting down. So Cold Boot has been a big blocker, and it's primarily the initialization time of the application before handling jobs. On GPUs and machine learning, exponentially harder. Reason being, we're running 20 gigabyte models. Those models can't be taking up RAM before a call comes in because that is not serverless. Then you're just running an always-on replica. So the Cold Boot problem is deeply exaggerated when you get to GPUs because not only do you need to provision the GPUs and the environment or the container, you need to load that model from disk onto CPU, onto

• 7:04Speaker 03

  GPU. That process could take 10 minutes for some models. And it's just been a pretty huge blocker for most GPU use cases. So for that reason, this product hasn't existed before.

• 7:17Chris Benson?

  Definitely not trying to delve into the secret sauce, if you will, but can you kind of lay the landscape of how you even start to think about that problem? Like, what are the some of the different ways that you might address? And maybe different orgs, you know, as you develop competition over time, probably different people will take different approaches. Like, how do you even think about that landscape? Because that seems like a daunting task. You know, when you talk about 10 minutes to get it moved over and stuff, that's huge. Like, how do you even start to approach the problem?

• 7:48Speaker 03

  So this is definitely one of our most prized pieces of IP, our Cold Boot tech. So can't dive too deep into the details.

• 7:55Chris Benson?

  No worries. Whatever works.

• 7:57Speaker 03

  What is publicly known, you got to think about getting, well, firstly, constraint. Constraint, you cannot take up GPU RAM. If you have a 40 gigabyte A100 machine, if you put a model into that RAM. That portion of the RAM or like that machine entirely, if you're not virtualizing it is just like it's taken. You are paying for it. It is dead space. If you're not using it, that's massive GPU burn without any utilization. So constraint model can't sit in RAM.

• 8:30Daniel Whitenack?

  Okay.

• 8:30Speaker 03

  At least GPU RAM. So when we go about the cold boot problem, what we're really thinking about is how do we get the model specifically, the weights as close to RAM as possible without actually occupying resources? Or, you know, more precious compute resources like 40 gigs of limited RAM. That's hard. But if you have a terabyte of storage on the machine, you could at least have local caching, the model. So like you could take that up passively between calls without incurring, you know, sacrificing that piece of hardware. Because you could fit so many more models onto the disk.

• 9:05Daniel Whitenack?

  Gotcha.

• 9:06Speaker 03

  And then you could start thinking about like, you know, how do you start pre-caching this on the CPU if the CPU has enough RAM? I'm not saying that's something we do, but these are like the framework in which you would start thinking about it is how do we get that RAM is or that model is close to GPU RAM without actually taking up GPU RAM? Because in the end, GPU RAM is, that's where the cost goes. Because once you use that, that machine is tied up. It's not usable for anything else.

• 9:30Daniel Whitenack?

  In your experience, I mean, I know you've been likely talking to tons of different clients, different use cases like that are really kind of thinking about how how their workflows could adapt to the serverless workflow. I'm just thinking about my own workflows. Like we're running a lot of models, but none of our models on my team are like receiving thousands of inferences per second or something like that. Like it is very much like in this zone where we kind of get a burst of activity and then we're kind of down, you know, for a bit, not getting that much, and then maybe another burst that we need to process.

• 10:10Daniel Whitenack?

  So in that case, like I would probably be willing in my own use cases to put up with somewhat like longer of a cold start like response for the model when it comes up. And then subsequent ones during that burst being much faster. What have you noticed with clients? Like what is the tolerance there? Like where are you trying to get? And where do you think is like reasonable for most workflows, I guess?

• 10:39Speaker 03

  I don't have a perfect answer for you on this in that ideally cold boots are zero.

• 10:45Daniel Whitenack?

  Yes, that that's true, I guess.

• 10:48Speaker 03

  On Banana, on a serverless platform in general, unfortunately, you do have to start thinking about the servers because you want to avoid cold boots when avoidable. In the case of Banana, if you have a model, it's undergone a cold boot, it's handled the first call, it's ready to go. We have it configured to hang around for 10 seconds, just in case more calls come in. And that 10 seconds is completely configurable by the user. So if no calls come in, we consider it, okay, we've gone through the surge, we could scale down. That particular replica scales itself down. If calls start coming in again, cold boots are incurred again. Only if the existing replication you have can't handle that throughput and start scaling up more.

• 11:32Speaker 03

  So because we give users the ability to like fine tune their auto scaler in sense, or fine tune maybe.

• 11:40Daniel Whitenack?

  Configure.

• 11:41Speaker 03

  Yeah, configure. You can configure the auto scaler. So we have some users who choose to run always on replicas with a minimum replica count. So at any given time, maybe you have a baseline of two GPUs running, but you could surge to 20 if you need. So we have some users doing that. We have some users who have gone away from the default 10 seconds idle time to go longer because they know like they would rather pay for those GPUs to be up and handle any traffic that may come in than have more frequent cold boots. Reason I give the context about Banana is I've been really surprised by how few users increase their idle time.

• 12:23Speaker 03

  Right now, or at least the majority of the customers we're serving are more price sensitive than latency sensitive, at least given the general trade off we give them in that they could configure the idle timeout and through that tune how much they pay versus how much they wait. But most users would rather have machines shut down and then incur that cold start time. And that's a great thing for us because that allows us to chip away at this cold start problem and give users an exclusively better experience of. The faster your cold starts are, the more willing users are to take those cold starts because it's less impactful on their inferences. And the less idle time you could run on your GPUs following calls before they start shutting down because it's not as risky, you know?

• 13:25Speaker 00

  Hello friends, this is Jared here to tell you about Changelog++. Over the years, many of our most diehard listeners have asked us for ways they can support our work here at Changelog. We didn't have an answer for them for a long time, but finally, we created Changelog++, a membership you can join to directly support our work. As a thank you, we save you some time with an ad-free feed, sprinkle in bonuses like extended episodes, and give you first access to the new stuff we dream up. Learn all about it at changelog.com/plus. You'll also find the link in your chapter data and show notes. Once again, that's changelog.com/plus.

• 14:09Speaker 00

  Check it out, we'd love to have you with us.

• 14:21Chris Benson?

  So as you were kind of describing that, that was really, it's a very interesting mesh of skills, it seems, to do what you're doing there because you obviously have to have a pretty good understanding of deep learning in general and the kind of the AI space and the performance characteristics around that. But you also have to go very, very deep in terms of network engineering and architectural considerations and such like that. It also kind of brings different cultures together. For instance, in terms of like the choices of languages and stuff distinctly, like, do you tend to go with one? Language for everything for simplicity sake?

• 15:04Chris Benson?

  Or do you tend to go with different languages that are catered towards specific use cases? By way of example, like Python for deep learning, specific things and like Rust, you know, or something C++ for infrastructure thing? How do you, or do you stick with one like Python for everything? Because that way you have a simpler setup to govern. What, how do you take that strategy wise?

• 15:26Speaker 03

  So the obvious language for hosting ML model inference is Python. It's almost a requisite as in all of our users are running in it. So therefore the framework that we give users to build off of, which is essentially boilerplate for a server. That's written in Python. We don't need to maintain that too much. It's an extremely simple HTTP wrapper and the vast majority of our work on the pipeline, the infrastructure side is all done in Go. So we're probably 95% Go. We have some TypeScript for our web app, some Next.js that we're running. And then when you get deep into like the runtime, we work on C++ and CUDA as well.

• 16:09Speaker 03

  But that's a small subset of our engineering team works at that level. The majority of us write pipelines and networks within Go.

• 16:17Chris Benson?

  I gotta say, it's kind of funny that you bring that up. Daniel and I love Go. We actually met in the Go community because we're both Go. We were like, at the time, kind of the two AI oriented people in the Go community. So it's just a little bit ironic to hear that.

• 16:31Speaker 03

  That's awesome. I've been so disappointed in Python. I mean, Python's amazing language. It's where I learned my first bit of serious general purpose programming was Python. But I'm saddened to know that the language you chose for GPU programming, basically, is a language that like has a global interpreter lock, like Yuki.

• 16:55Speaker 03

  It does not have great multi-processing built in. I wish Go were the choice there. It doesn't seem like it's going to happen, but I'm huge on Go. I think it's a great language to write in and I could go on for a long time about this. In fact, one of the reasons I learned about the change log network was listening to the GoTime podcast, so.

• 17:13Chris Benson?

  Yeah, for sure. Shout out.

• 17:16Speaker 03

  Shout out to that other podcast.

• 17:17Daniel Whitenack?

  Yeah, definitely, definitely. It's cool to hear about like, I guess, the setup of how you thought about this problem and how you've even structured the team and that sort of thing. I'm wondering at this point, if you could kind of just give us a sense for like, if I'm a data scientist, if I'm a, or even just a software engineer trying to integrate a model into, you know, my stack, like, what does the workflow as of now look like for me with Banana? Like, what do I do to get a model up and going? And maybe just a couple of examples of that to give people a sense. It's a bit hard on audio podcasts, but I'm sure you've done similar things in the past, so.

• 17:57Speaker 03

  Yeah, well, I'd love to give a visual demo, but going through audio wise, generally, the process looks like this. A lot of people are building off of standard models. So say a stable diffusion or a whisper, at least for like this current, this hype wave of all these new exciting open source models coming out.

• 18:16Daniel Whitenack?

  Until next week.

• 18:17Speaker 03

  Yeah, until next week, and then the next one comes out. Thankfully, we have these one click templates that you could use on Banana. So in a single click, you could go from an open source model that somebody has published on Banana and bring that into your own account and start using it yourself. So within a few seconds, you could have a functioning endpoint for popular models that have been put up by the community. And then we see naturally the step beyond that, moving from you effectively have an API. You don't really know what's running behind the scenes. You could fork that code, you start working on it yourself and customizing it for your own use case. So if you're doing some fine tuning, if quite honestly, you want to go away from the standard

• 19:00Speaker 03

  or like the big model templates and roll it yourself, just have whatever deep net that you've built. That's where you start getting into sort of the local dev iteration cycle. And this is where I shout out a previous guest, Brev Natter over there at Brev. We recommend users go and have an interactive GPU environment so that you could load your model, test it against some inference payload, shut it down, iterate. If you're doing something like a stable diffusion, you want to make sure that the image transformations server side are happening correctly. That's where you iterate. You're doing all of this within the Banana framework. We have an HTTP framework.

• 19:42Speaker 03

  You could find open source online. That's generally the building point for most users. So you're modifying a function within that, that is the inference function. Takes in some JSON, runs the model, returns some JSON. Do that iteratively until you have your customized model that works to the API you're hoping for. And then you push that to GitHub. And then from there, you could go into Banana, you could select that repo and we have a CI pipeline built in. So when you select that repo, we build the model, we deploy it every time you push to main, we rebuild and redeploy. So we generally recommend users to, if they're shipping new fine-tuned versions, it's usually them updating, say, a link to an S3.

• 20:27Speaker 03

  Then in the build pipeline, we bundle that model into the container itself and get that deployed to the GPUs.

• 20:32Chris Benson?

  So kind of curious, and this is sort of a follow-up largely because of the medium we're in. Since we're audio only and we don't have the ability to to show the process that you're describing. Just for clarity, like your typical customer slash user, what skills would they typically have to productively use Banana? You know, what are those necessary minimum skills for them to be able to really engage productively and move through things?

• 21:00Speaker 03

  A lot of our users are quite surprisingly full-stack engineers and not deep experience data people and ML people. So as long as you can wrap your head around using frameworks or abstractions like hugging face, for example. If you could use a pipeline like that, pull it locally, that's something you could deploy into Banana. So some Python expertise in order to write the code in the first place, it's an HTTP server. So you write that, you wrap it around, say, a hugging face model. You don't need to fine tune it. You could use the standard models and then learn fine tuning later. And ideally, you do have some knowledge of Docker. Ultimately, what is deployed to Banana is a Docker file. If you build within our template, generally, you don't need to do things that are too custom unless you choose to.

• 21:45Speaker 03

  But a little bit of knowledge of Docker helps. So Python hugging face Docker, that's effectively all you need in order to get something deployed onto Banana.

• 21:54Daniel Whitenack?

  I'm just on the site now and kind of looking through some of your community templates, which are pretty cool. I mean, you have all sorts of things, CodeGen, T5, Santa Coder, all sorts of things with a sort of one-click deploy button to get them up and going. One question I had, just like when I deploy, because it looks like based on your docs, I can call it with like the model ID from Python, for example. So I could like integrate this directly in a Python app. Can I also call it sort of like as a REST endpoint or something like that? Or is the primary use case a client integration?

• 22:34Speaker 03

  We do have public documentation for the REST endpoint, so.

• 22:38Daniel Whitenack?

  Awesome.

• 22:38Speaker 03

  It's not officially supported. We try to encourage people to go through our official SDKs, which at this point are Python TypeScript go in Rust. That said, anyone who wants to go directly into the REST endpoint, there's documentation to do so. We like being able to boil it down to a simple banana dot run function where you just give a model key. You give whatever JSON in you want your server to process and then you receive the JSON out from that. But our goal is to be able to give people access to the levels of extraction that they choose to run in. For example, because we have a public REST endpoint, people have integrated Banana into their Swift applications or into their Ruby applications. So it's an HTTP call in the end.

• 23:22Speaker 03

  People could unwrap our APIs and go at it directly, feel free.

• 23:27Daniel Whitenack?

  Yeah, I guess that leads right into my next question, which is, does anything stand out in terms of like how people are using the serverless workflow? That maybe surprised you based on what you're seeing?

• 23:39Speaker 03

  I've been amazed at the quantity of fine tunes that are deployed through Banana. If you look at the analytics of people deploying from our one-click templates versus people deploying from custom repos.

• 23:52Speaker 03

  Eighty percent are custom repos. And that means that people are coming to serverless because they have a unique API that they need to run somewhere and that they can't simply run with a standard API provider, or even an API provider with fine tuning features. Like, they want to be able to own the API themselves, own the application logic themselves, to fine tune themselves, and just Dockerize that up and send it on to Banana. So, vast majority of our users are doing custom workloads, which to me was surprising. We, little Banana lore, we previously started as an ML, as an API company. The idea of showing up, click the model you want and you get an API for that. And there's a lot of pull there, especially right now with the hype.

• 24:35Speaker 03

  There's so many people who want to integrate AI into their applications without touching the AI at all. So it has been surprising for us, seeing how many people are running custom code on us. It's been validating of the idea that the platform approach versus the API approach has been the way to go.

• 24:50Chris Benson?

  Could you kind of walk us through what a typical one might look like? You know, where someone's doing that kind of custom thing? Just to give us a sense of what it is that you're seeing, whether it's, you know, fictional but realistic, or a real case example, whatever works for you.

• 25:05Speaker 03

  So one thing users are doing, just as a very basic example of, if latency is an extremely sensitive thing for them, and cold boots are particularly painful, what they'll do is they'll engineer a conditional, like a Boolean, in the JSON that they send in. That's called the warmup. So they'll do like, warmup equals true, and make it so that server side, they actually don't perform any heavy computation. It's just intended as a warmup call. So if architecturally, they need servers running, like fully warmed up by the time the actual inference starts running, they engineer this into their endpoint. Another thing as well, if people want to run fine tunes or run multiple models side by side,

• 25:47Speaker 03

  and start doing some model chaining, we see people building that into Banana as well. And then... Lastly, are just basically state of the art moves so fast right now that the second stable to Fusion launch, for example, suddenly there's in painting and in painting is the next thing that came out a week later and that's some random code people found in a GitHub and they integrated themselves. So customization on that sense allows users to stay as far ahead as they possibly can if it's necessary for their use case.

• 26:20Daniel Whitenack?

  Could you highlight something you have in your mind as maybe like a workflow that would not be appropriate for the sort of serverless GPU infrastructure? So I think this like, like you say, fine tune models, inferencing, like, you know, using these state of the art templates? Is there something where you would say, hey, like, maybe that's not fitting for the serverless use case?

• 26:47Speaker 03

  Yeah. So in inference land, if you have completely steady traffic all the time, don't use serverless. You'll get unnecessary cold boots and it just slows down your inference and you're paying effectively the same. So that's the inference side. Training side, we'd like to think that you could currently train on banana, though I often find that training's a more interactive experience, or at least in like the initial prototyping phase. Once you have pipelines built in to say automatically collect data and batch train, that actually does work on banana because you could just fire that data as the payload, train the model server side, upload it to S3, return the call and then the replica shuts down. But most

• 27:31Speaker 03

  training jobs or most like exploratory training jobs, I would not recommend doing on serverless in part just due to the like the observability that you need to see, the tracing, setting up things like, you know, this is outdated tech, but TensorBoard, those more visualization tools. Also, keep in mind, I'm not a training expert. So perhaps there's there's space in the training that people would see value in serverless, but generally I'd recommend avoiding serverless. And then lastly, if you have any jobs that are batched, as in you know exactly when they're going to happen. It's a bit easier to automate your own infrastructure and build it yourself to do that. Ideally, we make serverless so good that you don't need to think about that.

• 28:16Speaker 03

  But I think in the current state of serverless, a lot of batch processing jobs, if you're, say, running an indexer across an internal database and you don't need to have it running all the time. That's where running a serverless may be a bit too much lift in order to port it into serverless versus just doing it yourself.

• 28:34Chris Benson?

  I was looking, I'm also looking through your website while we're talking and I'm in the docs and I kind of hit the SDK area, which you kind of talked about a little bit ago with the different SDKs in Python, Node, Go, REST. Did you mention Rust earlier or did I mishear that as REST? I may have misheard something.

• 28:53Speaker 03

  I did mention Rust. I actually don't know if we have it documented. We launched it two days ago, if I recall.

• 28:58Chris Benson?

  Gotcha. Well, so the thing that got me thinking here, that's very leading edge. It's very like out there. I'm kind of getting the sense that your, your customers are adopting more, you know, forward leaning languages in general for what they're doing. And that's why they're, you know, they're leaning forward into this new concept of serverless GPUs. Is that, is that consistent with what you're seeing? Are you really kind of targeting the types of software developers that are kind of early adopters paving the way versus somebody that's maybe in some of the older, more enterprise-y languages, maybe not quite as, you know, risk-taking and such?

• 29:42Speaker 03

  That's very much in line with what we've been seeing. We find that a lot of our users are adamant in Vercel users as an example. So they're in Next.js, they've chosen a relatively modern framework to build their front-end apps in. And they make the same decisions for their backend. They're often TypeScript-forward. If they want to do systems level, they'll do Rust or Go. For these reasons, we've chosen to offer these official SDKs.

• 30:07Daniel Whitenack?

  Yeah, that's really interesting. I, one of my questions in kind of thinking about this is like the different use cases that you could have, the different industries that are rapidly adopting AI, integrating it in their, in their software stacks, like everybody's adopting AI, right? But like, it's certainly making a lot of strides in certain areas and certain industries like, you know, let's say healthcare or something like that, have very unique constraints around even like their own inference data leaving to go like to some hosted model somewhere that's not in their own infrastructure. But in other words, when I go to banana, I see like, all I have to care about is like

• 30:51Daniel Whitenack?

  deploying a model, there's my model ID, right? I can think about like the timeout and all of that. It's all very functional, right? And I don't even have to like give a thought for where that's running. I could see like the opposite end of that is like certain industries would probably be a little bit uncomfortable with that, but there's a whole lot of developers that are just wanting to like, you know, bootstrap these like amazing AI powered things like very rapidly. There's so many things coming to market like that. So I guess that would be fitting in that way. Do you have any plans in the future for like banana serverless, but like connect my AWS

• 31:31Daniel Whitenack?

  infrastructure or something like that to run in the banana way or something like that?

• 31:37Speaker 03

  Short answer, yes. Long answer.

• 31:40Daniel Whitenack?

  It's complicated.

• 31:41Speaker 03

  It's gonna be a long time. Yeah, it's very complicated. And one of the things that we see with serverless is the fact that we have economies of scale sharing everyone as tenants within our cloud, because that allows us to do more efficient bin packing and make it so that when you're not using a server, like when the server container is shut down, you're not charged. If you're running on your own cloud, you still need to have the underlying resources running. We're a venture scale business. We want to hit that million dollar annual revenue, ideally, or sorry, not million dollar, a hundred million dollar annual revenue, ideally more. And I think getting into that, we're eventually going to have to start thinking about how do more traditional enterprises integrate this. Though choosing our niche right now, we see significant pull that could get us to one

• 32:25Speaker 03

  $10 million annual just from these like new teams who aren't bound by such constraints of needing to run in their own cloud. So, long answer, restated, we'll get to it eventually. And I'm sure it's like, it will be a necessary part of the product, but it loses out on a lot of the magic that we're currently providing. So we'd rather just focus on these new and upcoming startups that are running on us.

• 32:47Daniel Whitenack?

  Yeah, that makes a lot of sense, I think. It does make me wonder, like, because you are creating so much magic for the users, and a lot of that, like you're saying, like, thinking about, like, what GPUs are you spinning up? Like, how are you bidding on these? Like, where are you, like, how are you allocating them? Have you learned any sort of, like, general, like, you can get GPUs from a lot of places. There's a lot of different kind of scales of pricing. There's a lot of different ways to run GPUs in the cloud. Have you found any just sort of, like, good practices or things that you've found to

• 33:28Daniel Whitenack?

  be useful, just generally in terms of thinking about, like, using GPUs in the cloud?

• 33:34Speaker 03

  Yeah.

• 33:34Daniel Whitenack?

  That you'd love to pass on to listeners?

• 33:37Speaker 03

  So, we use this phrase called skate ahead at the puck. It's a phrase from hockey where, don't go to where the puck is, go to where it's going. So, applying that to auto scaling. Auto scaling really has two components. You're auto scaling the underlying nodes, the hardware that's running the GPUs, like that's, you know, running the Kubernetes cluster, whatever your deployment target is. And then secondly, you're auto scaling the deployments themselves, going from replication of zero to one to many, within the confines of whatever nodes you have set up. So you're effectively auto scaling two things, Kubernetes pods and the nodes themselves. So my recommendation are, if people are building things like this in-house, what

• 34:22Speaker 03

  they should absolutely do is use a platform that has an automation API. For the underlying VMs, right now GPU cloud is sort of the Wild West. There's a lot of new players. Traditional hyperscaler clouds like Google Cloud, AWS, Azure. They have the automation, but the GPU prices are not as competitive as you could get on some of these newer clouds. So my biggest recommendation for people building mature systems would be to like choose a provider that you get ideally guaranteed access to GPUs, which allows you to scale your GPUs up ahead of the demand of whatever workloads you're running within your cluster.

• 35:03Speaker 03

  And then it doesn't have to be homogeneous, the workloads deployed. Just as long as you maintain GPU capacity to handle those, you should be good. But because you're auto scaling, like the applications within Kubernetes allows you to have a little more lead time for like super slow scale ups on the GPUs.

• 35:22Chris Benson?

  This has been a super instructive conversation. I'm learning a lot. I want to extend your analogy one question further because you're talking about skating ahead of the puck. Not skating to the puck, but where it's going to go. You are pioneering this field. You are out there on the front. You are leaning forward and you were supporting other people in other organizations that are trying to lean forward as well. So I'm going to ask you, where is the puck going? You know, short term, middle, middle, long term? How do you see the future for those who are not in your industry, but are are going to be supported by you? Tell us the vision. What's it going to?

• 36:02Speaker 03

  Fine tunes are going to be huge. I think there is two camps for where AI is going to be going. There's the the one model should rule them all camp, which is there's going to be some mega model that does everything. And then there's the other camp, which is what we're leaning into, which is the best model for you as a user is a model that's trained on data from you, specifically you. And we see customers deploying fine tunes on us, not just for their use case, but for their end user. Imagine you are building a writing assistant app. How do you fine tune for every single one of your end users and deploy that? And make it so that that user has a unique model, is essentially a companion, almost a clone of them.

• 36:45Speaker 03

  And where the puck is going is where we, every human on earth, just like they have a phone in their pocket, they're going to have a fleet of models fine-tuned just on them. And that's one thing we're excited about with serverless is in order to do that viably, you got to have serverless. Can't have it running all the time. So very excited in this sense. If you're not looking into user level fine tunes, I think it's a very interesting space to be in because it gets you so much further than any application level stuff you could do to make the experience better.

• 37:13Daniel Whitenack?

  That's awesome. Yeah, I think that's a super exciting way to close out the conversation. This is a really exciting time to be in the space, both in terms of what's possible with fine tuning and those sorts of technologies, but also like new infrastructure coming up, like what you're, what you're building. So thanks so much for taking time to chat with us, Eric. It's been a real pleasure.

• 37:35Speaker 03

  This is awesome. Appreciate it, guys.

• 37:45Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a long time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.