End-to-end cloud compute for AI/ML

We’ve all experienced pain moving from local development, to testing, and then on to production. This cycle can be long and tedious, especially as AI models and datasets are integrated. Modal is trying to make this loop of development as seamless as possible for AI practitioners, and their platform is pretty incredible! Erik from Modal joins us in this episode to help us understand how we can run or deploy machine learning models, massively parallel compute jobs, task queues, web apps, and much more, without our own infrastructure. Sponsors: Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Erik Bernhardsson – Website https://erikbern.com, GitHub https://github.com/erikbern, X https://x.com/bernhardssonChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Modal https://modal.com/Episode 142 discussing Erik’s “building a data team” article https://changelog.com/practicalai/142Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International. And I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 0:58Chris Benson?

  I'm doing very well. How are you today, Daniel?

• 1:01Daniel Whitenack?

  I'm actually doing amazing. So I'm not in my normal location. I'm down in Orlando, Florida. So one thing is it's sunny outside and I can like be outside without suffering. But also, while I'm in like in-person meetings here with some of our collaborators and partners, and they wanted me to do a demo today. So I got up early this morning at like 6:00 a.m. before hotel breakfast, and I threw together a quick demo. And I used Modal for that. And there's literally someone that stood up out of their like seat and clapped after the demo.

• 1:40Daniel Whitenack?

  So our guest today is Eric Bernadson with Modal. And so basically, Eric is making me look good in all respects. And and I'm pretty excited to talk more about Modal and share it with everyone today. Welcome, Eric.

• 1:54Speaker 03

  Hi, hi. Thanks for having me. I'm excited to talk about Modal or anything else.

• 1:59Daniel Whitenack?

  Yeah, yeah, I think, Chris, do you remember like quite a while ago, I don't remember when this was, maybe Eric, you remember, I think you wrote a blog post about building data teams or something like that? I forget exactly what it was, but I remember Chris and I talking about it on the podcast. I'll see if I can find it back in your blog.

• 2:19Speaker 03

  Yeah, that was in the summer of 2021.

• 2:22Daniel Whitenack?

  Yeah, yeah. So we should have had you on the show then, but I'm glad that we get to have you on the show now. So, so you describe Modal as an end-to-end stack for cloud compute. So I guess one big question, maybe to start things out is cloud compute isn't new, but it definitely can be complicated depending on what you're trying to do. Like, what got you starting to think about like the set of problems that you're addressing with Modal? Like, what, what got you going down this path?

• 2:58Speaker 03

  Yeah, it's kind of a longer story, maybe, but I've been working with data for 15 years or maybe more. Most of my career, I was at Spotify for seven years. Built the music recommendation system there and open source a vector database called Anoyn and workloads scheduler called Luigi. But they're like, kind of everything from like deep learning to like business intelligence to large-scale, big data type, like Hadoop stuff. And then I was a CTO at a company called Better for six years, managed data teams, but also managed other teams. And so as I was thinking about starting a company, I kept coming back to data. And my starting point was really just like, it's hard to work with data. And I feel like data teams don't have the tools they need.

• 3:41Speaker 03

  And initially, I was super agnostic as to like what to build. I kind of frankly wanted to rebuild everything, which is not particularly realistic.

• 3:50Daniel Whitenack?

  Maybe in a lifetime.

• 3:51Speaker 03

  Aspirational, megalomaniac, maybe. But what I realized was that, at least like, you know, if you want to rethink a lot of the data stack, a good place to start is at the bottom. So I almost like sometimes joke that I'm like, kind of like grudgingly had to start, you know, it's like a spite startup, like, you know, and doing this like work at now at like the most lowest level, which is to, you know, solve the compute problem of like, I have code and I want to deploy it in the cloud. And like, why is that so hard? I don't know, but you know, it's a big problem for many data teams is, you know, just a problem of like, you know. Taking code and scaling it out, or scheduling it, or running it on GPUs, or, you know, setting up web endpoints, or whatever it is,

• 4:34Speaker 03

  and really focusing on that problem as like, you know, building this core foundational layer. That's very abstract and, you know, very general purpose. So that's also why our website, I think, is a little bit confusing the first time. But in particular, what I would say we'd be focusing on the last six months is online inference. So it's a lot of machine learning, AI models focusing on that use case as a sort of initial starting point. But but Modal always, always, to me, had this promise of, you know, running almost anything. It's almost like a Kubernetes in the cloud.

• 5:05Daniel Whitenack?

  Yeah, and one of the interesting things to me, I think, where like, maybe it took me a second for this to like sync in, but once it did, it like was a really encouraging thing for me was like, I have my code locally and I know how to run it locally. But then like you have this sort of concept of like these decorators within Python code that kind of take your code and like you run it like Python, you know, main.py, whatever. But actually something like the moment I realized, okay, well, now this function is actually not running locally.

• 5:46Daniel Whitenack?

  Like I just did some sort of like batch inference or something with this script and didn't like, you know, my fans aren't going on my laptop because this is actually running somewhere else. Could you describe like, I mean, there's a lot of ways that you could have gone about this sort of lower level of the problems that data teams face. There's a really fundamental piece of this, which is like the local to cloud or local to deployment like cycle. And with Modal, that seems very, very quick. Like, how did you zero in on that kind of workflow?

• 6:26Speaker 03

  We built something that architecture looks something like AWS Lambda, right? Like, you know, it's like a function as a service. We take code and execute it in a serverless way in the cloud. The starting point, like, the reason why I ended up going down this rabbit hole of doing that, you know, with the whole, like, you know, serverless, you know, runtime, is really kind of thinking about, like, developer productivity and developer happiness. And, and my sort of philosophical observation as CTO for many years is that developers, developer productivity, I think, you know, is very often, like, well-understood in terms of feedback loops. And so, in particular, there's a sort of, you know, as you write code, there's like, almost like a nested set of for loops. It's the innermost loop of, like, you write some code, then it's a syntax error, then you fix it, and then you run it, maybe you have some unit tests.

• 7:09Speaker 03

  But then, like, there's like these, like, outer loops, they're often like, okay, let's, like, deploy this to the cloud, or like, you know, let's, like, run this on a massive data sets. And that's when the iteration speed gets, like, very, very slow. And so, you look at data teams, like, they're often, like, particularly exposed to these feedback loops, because they have to run a large data sets, or they always have to run things in production. Like, you can't really run, like, things on, like, synthetic data as a data team. Like, you have to kind of deploy it into production, or like, run it on a real data set. And so, it really frustrated a lot of data teams, just like that, that sort of, like, very slow iteration speed. It's like, I write some code, now I have to, like, you know, create a container, push it to the cloud, then go and click on an interface, or like, you know, merge, you know, some pull requests or whatever, then, like, you know, my container fails, now I have to go and look at logs or whatever.

• 7:53Speaker 03

  So I started thinking about, like, what if we bring the infrastructure into that, like, innermost loop? Like, the, the loop of, like, okay, you just write code, and then you, like, immediately run it, but it actually runs in the cloud. Uh, and, you know, in order to do that, we realized we can't do this with Kubernetes, we can't do this using Lambda. Like, we basically have to build our own infrastructure that takes code and can launch containers, maybe hundreds of containers in the cloud in a few seconds. Uh, so we went very deep, you know, down that rabbit hole and, and built basically our own container runtime, our own file system, our own container builder. Uh, luckily, I'm not, you know, afraid to go deep and, like, solve, like, kind of tricky, you know, container problems and, and dealing with Linux and file systems.

• 8:35Speaker 03

  Um, but that's like a lot of what we had to build in the last two years, is that... Is that foundational level like that runtime? But the benefit is like now we have this, you know, super nice developer experience. We can just take code locally. You can spawn 100 containers in the cloud in a few seconds, like running the latest code in the latest container.

• 8:52Chris Benson?

  Sounds fascinating. I'm really interested in it, but I want to ask you to step back for a second with a follow up and bridge a gap of understanding for me. You were saying like can't do it with Kubernetes, can't do it with AWS Lambda. And I believe you, but I don't know why. And I'm imagining that maybe a few of our listeners don't know why either. Could you kind of tell us what it is like? Cause a lot of them, their companies are in one of those big three providers. And to kind of show them, you kind of demonstrated with the user experience quite well a moment ago. But could you talk a little bit about like what was falling down in those kind of more mainstream, big three kind of approach, Google, AWS and Azure, so that we kind of understand that? Cause you made a statement, I'm with you on that, but just bridging it.

• 9:38Speaker 03

  Yeah, first of all, like I'm like the world's biggest AWS fan, right? Like, you know, we run everything on AWS and like I love it for the capabilities it brings me as a developer to run things at scale. Developer experience in AWS has never been particularly good. And so, you know, like I've been banging my head, you know, for years against like, you know, AWS documentation and in the end I usually figure it out, but it was a pretty jarring experience. I think in particular, the problem with both Kubernetes and AWS, you know, or like Lambda or EC2, etc. That we saw like, you know, either, you know, for users to use it directly or for us to build on top of that is, it's just the iteration speed. Like, so if, you know, for instance in Kubernetes, let's say you want to run something in Kubernetes in production, going from code locally.

• 10:21Speaker 03

  Well, now you have to first build a container, then you have to do some sort of Docker push as a registry, right? Then you have to kick off a Kubernetes job. Then, you know, you have to go and look at the logs of that Kubernetes job. And by the way, kicking off a Kubernetes job, like that often like entails like, you know, the Kubernetes worker like pulling down their Docker image. And so we were like looking under the hood and trying to understand how like Docker works and Docker. You know, it's an amazing piece of technology like, you know, for the sort of, you know, the new way of thinking that it brings to the table around like, you know, uh, insulated containers, but it's quite inefficient in starting containers like, you know, most containers end up having lots of data that's never actually read, like there's like, you know, thousands of like time zone files of like, you know,

• 11:06Speaker 03

  locale information about, you know, time zones in Uzbekistan or whatever, like you're never going to read those, right?

• 11:11Chris Benson?

  Unless you're in Uzbekistan. Sorry, just getting that in there.

• 11:15Speaker 03

  Yeah, and this, you know, whatever, you know, or uninhabited islands, like there's like time zone information about uninhabited islands in, in like, you know, the standard Linux distribution, like, okay, like, great, but like, get them out of my Docker container. Uh, but the other thing is also Docker is quite inefficient and like it has this like layer thing, but like other than that, you know, it doesn't really deduplicate information. And so what we realized is that, what if we like rethink how, you know, those containers get pushed and pulled and we end up building our own file system, we need to duplicate the content by computing a checksum of every file. Uh, it's actually sort of similar to how Lambda works, uh, but, but Lambda is also not fast enough in the sense that like, if you publish a new Lambda, it still takes about a minute for you to be able to run it. Lambda also has other limitations. It doesn't support GPU, it doesn't support long running jobs, et cetera.

• 12:00Speaker 03

  Uh, so those are all the reasons like, well, we ended up deciding we can't build this on top of Kubernetes or Lambda or any existing solution. Uh, also not Docker. Uh, we ended up using lower level primitives instead and building a lot of it ourselves.

• 12:13Daniel Whitenack?

  And are there specific things about, and, and I sort of like, in my own experience and using Modal have experienced this, but from your perspective, I would be interested to hear, like, you talked about kind of moving towards this use case, like the use cases around machine learning, around AI as being kind of like very well suited to this workflow. Of those types of workflows have any sort of like added benefits and or challenges that maybe like, you know, running a web scraper or something like, like some other sort of like use case, which is related to data, but maybe not involving sort of like serialized model files and inference and GPUs, like what are those,

• 12:57Daniel Whitenack?

  those things about these machine learning or AI workflows where you think like, either there's specific challenges that people have that are solved by the, this kind of quick cycle workflow um, versus just kind of like other data related workflows?

• 13:13Speaker 03

  Yeah, I think we focused a lot on online inference recently. So basically, you know, let's say you have a model. It could either be some off-the-shelf model from Pugnface or some fine-tuned model that you have yourself. And you want to deploy that. And in particular, if that model uses GPU, the set of vendors that support that is somewhat limited. And the other reason why it's is also cost. You know, traditionally, if you go through the Kubernetes or EC2 route, if you want to deploy a model inference endpoint, you have to spin up an instance that sits idle for most of the time. You can set up auto scaling, but auto scaling is pretty slow. So moving to serverless makes a lot of sense from a cost perspective. And so I think that's the other reason why we've seen a lot of, you know, it's not just us. It's been, I saw the banana was in the previous episode, for instance,

• 13:58Speaker 03

  like there's a couple of other vendors that also focus on this. I think cost is driving a lot of that, the demand for serverless vendors for GPU compute specifically. I also think it's something that just came up in the last few months, where like a lot of people ended up realizing like, you know, we're very good at training models, like building, you know, custom stuff. We don't want to deal with infrastructure and running this in production. And so there's been a lot of demand for vendors like Modal, where they can just take a model and publish it to Modal and run it in production and not have to think about, you know, auto scaling policies and have to think about setting up, you know, web endpoints and dealing with security groups and all that stuff. That being said, I mean, Modal, kind of going back to its roots, like we did, you know, it's not just online inference.

• 14:41Speaker 03

  Like we started out focusing a lot on, but I think it was like embarrassingly parallel problem, like this idea that like you have something you want to fan out and do a lot of stuff in parallel. So besides online inference, Modal also does a fair amount of batch inference or sort of parallelizable things. Like, like a lot of people actually use this for web scraping. Other people also use this for things like computational biotech. Large-scale chess coding, you know, it could also use this for various types of simulations or backtesting, that kind of stuff. So this is a pretty wide range of things that that model does well. But I think right now, like, the user experience of online inference is like nine out of 10, I would say at Modal. The user experience for like batch inference and large-scale like parallelism is like eight out of 10. We're working on a lot of the

• 15:25Speaker 03

  other stuff, like data pipelines, like building more complex support for scheduling and that kind of stuff. We're right now, like, it's good, but it's, you know, not quite yet where we want, you think, you know, the long-term potential is.

• 15:36Daniel Whitenack?

  So Eric, I I mentioned, and full disclosure to everyone in the world, I'm a huge fan of Modal and have been using it a lot and building things in it, including the side project I'm working on, Prediction Guard. And I think I just counted, I'm in the interface now, I have 129 Modal apps deployed right now.

• 15:59Speaker 03

  Wow.

• 15:59Daniel Whitenack?

  So I want to try to like describe from my end, like, it's hard because like this is an audio podcast and like talking about how things work without like showing something visual is a bit tough. But I want to do my best at trying to describe like how I would describe it. And then I'd love like you to fill in the gaps or correct me if I'm wrong at any point. So if you think about running something in Modal, you can write a Python script, like let's say app.py or whatever, you can have functions in that script. And then actually, one of the things I love is like dependencies is a really annoying part

• 16:38Daniel Whitenack?

  of particularly AI and ML workflows. So you can decorate certain functions in your code with like stub dot function and then define a Modal stub in your code, which is essentially like referencing a container with certain dependencies in it. And then when you execute your code, You say python app.py and when it gets to executing that function in your code, which is decorated with the stub, it actually doesn't run it locally. It spins up a container in Modal and runs that in the cloud. So you can do this either by just calling that function,

• 17:19Daniel Whitenack?

  or you can actually deploy then your app and have that function be accessible as like a serverless function or a web endpoint for your other applications or your other APIs to access. So I don't know if I did a great job at describing that, Eric. That was my initial attempt. Feel free to make that more coherent.

• 17:40Speaker 03

  No, I think that's exactly right. Like I are, I think you touched on a couple of points of Modal, you know, where we maybe think different about infrastructure and other. In particular, this guy Swix wrote a great blog post about it. It's called the self provisioning runtime. And I think that's like, to me, it's been kind of putting words to an idea that I always had around, like it's sort of similar like if you ever use like a service, like Palumi, for instance, like, you know, or like Terraform or something like that. There's sort of idea that like infrastructure is code. But like Modal has always gone further than that. It's also like infrastructure and the app code, like just put it together in the same code and have like the app itself define the infrastructure it needs to run. And so with Modal, in code, you define, you know, the containers you need,

• 18:24Speaker 03

  including Python dependencies or any other like binary dependencies you need. You can have different functions using different containers calling each other, just like Python functions, right? And it just like provisions itself, like you can say, you know, this function should run on a GPU, this function should have 16 CPUs available, this other function needs 128 gigabytes of RAM. Like in code, there's zero config in Modal. There's like not a YAML file, like there's nothing you can configure in Modal. Like everything is in code. And to me, it's all, you know, goes back to this idea of like the, how do you make developers productive in having the fast feedback loop? It's, I think traditionally we've had to give that up and basically make engineers run things locally in order to get the fast feedback loops they need. But then the problem is like, Later, they need then still need to deploy it to the cloud.

• 19:09Speaker 03

  And then you have a whole set of issues that then break. Because the cloud is running in a different environment. And so this goes back to what I said like maybe 20 minutes ago. Like, what if you can take the infrastructure and bring it into the innermost loop of how you iterate? Then you solve this problem of having different environments. Because it's always running in the cloud. And it's fast enough that like it feels, you know, some people even say Modal is faster than running things locally. Even though it's running in the cloud. You never ever have to think about these environment conflicts. Because it's always running in the exact same container at any time. And it's fast enough that you don't have to like this like frustrating thing where you have to build containers and push them around. And like, you sort of get the best of two worlds. You get the developer productivity of running things locally.

• 19:49Speaker 03

  But you have, you know, the full power of the cloud, you know, and all the power of, you know, containers and GPUs and like whatever, right? I don't know if that makes any sense.

• 19:59Daniel Whitenack?

  Yeah, it's, um, so Chris and I have talked about this at like certain points in the podcast. I have always like really had this disdain for like maintaining a whole bunch of like local environments as well. Like, I'm not a Kondi user. Like, I, I have like very minimal setup locally on my machine. And I, one of the things I think I kind of grasped onto is like, oh, well, I can develop now locally with Modal. And just like import OS and import JSON and like kind of normal-ish things and import Modal. But when I need to access transformers or PyTorch or like some random other package that like normalizes index scripts or something.

• 20:44Daniel Whitenack?

  Like, I actually have like zero concern about like setting that up locally to test. Because I can just add that as a dependency in the Modal function. And that runs in the cloud in its own container. So I actually never even have to install that locally. Now, I could do that maybe before like using like a local build of a Docker image or something. But that again, like you were talking about, Eric, has another cycle associated with it, which is also annoying. Yeah.

• 21:12Speaker 03

  Yeah, it's kind of annoying.

• 21:13Daniel Whitenack?

  So, yeah, I love that this like I can just think through like my imports are minimal. Like, I can even run like a Pytest, right? And it's just testing. As it's going to run in production, right? Because it's running in a container in the cloud already. It's running that function. Yeah, so I that's kind of like a lot of my love. I feel like that I've I've enjoyed about it. What are the surprising ways that you've seen people use? Modal that maybe like have been unlocked for users that were really either difficult for them before or like, oh, I didn't expect people to do this with modal. Have you encountered any of those things that stand out?

• 21:57Speaker 03

  I mean, model inference in itself like kind of, you know, was a little bit of a serendipitous thing for us. Like, you know, we didn't expect that people would do that in general. Like, we thought of modal primarily initially as like more of like a batch workhorse, like, you know, something that helps you scale out. But but we've seen a lot of traction on online inference and model deployments. And so for that reason, we're focusing a lot on improving startup performance right now. Because when you're doing online inference, you have to spin up containers very quickly. You also have to load models very quickly. And especially when you're dealing with GPUs. There's a lot of, you know, overhead of copying models and GPUs, etc. So getting that down has been a big focus of ours for the last few months. I guess another thing I've been like sort of surprised by is

• 22:39Speaker 03

  we enable the functionality to set up webhooks pretty easily. So in modal, you can define, oh, like, make this function exposed to the web and give it its own URL. And now you can call this a URL and it triggers something in modal. Triggers some Python code. People started leveraging that for like building like full blown web apps on modal, which I was kind of surprised by, like graphical UIs and all kinds of stuff and like hosting whole UIs. Because I never anticipated like that being like a use case. I always thought of like, oh, well, people are going to use like, you know, whatever, like Vercel or Heroku maybe for something like that. But but that's been sort of interesting to see that a lot of people are using that. And so it's pretty promising, like, maybe there's something more to be done there.

• 23:20Speaker 03

  Like, I I tend to think like our our bread and butter is like machine learning and AI and like, you know, data pipelines. So I don't want to go like all in on like sort of building more like a web hosting platform. But I think there's something interesting and sort of similar along the same lines. A lot of people have been using us more for sort of job queues type things like, you know, more like almost like I said, replacement to salary. Uh, the idea that like they can create a modal function and then they can like in queue work for it. And they never have to think about scaling or. It was sort of deployment and productionization of celery, like job queues. And that was also something we didn't really think about, but a bunch of people have been telling us to actually do. So that's kind of cool.

• 23:57Chris Benson?

  So I got a follow up question here, and you've both sort of covered it to some degree already. But as the person who has not yet had a chance to use it, I'm really curious and I'm imagining there are a few people listening as well that are wondering, could you take us, Eric, like through kind of a classic workflow with Modal? We've done that with with other technologies that you may have heard on other episodes and stuff. But I'm trying to get in my mind, Daniel is is doing it all the time, but I'm I've been left behind a little bit on this. Kind of just take us through a typical AI ML workflow on Modal, just verbally, like what the steps are. Just to kind of show us that simplicity, people probably get to be thinking about whatever they're on, you know, previously, if they're on some other platform.

• 24:39Chris Benson?

  And just as a point of comparison about how you're doing that, I'm just kind of curious if you can, any example is fine.

• 24:45Speaker 03

  Yeah, I mean, we've optimized a lot for the making it possible to deploy things and run things in the cloud in a few minutes. So it's actually pretty straightforward, like in in Modal, you basically take any Python function. So let's say you have a Python function that maybe uses hugging face, just as an example, and it, you know, it uses some off the shelf model for maybe stable diffusion. And so let's say you have a Python function, existing Python function that uses hugging face, and it takes a prompt and it returns an image. Now you can decorate that Python function in Modal with a special decorator and then annotate it and say, use this image and then define an image in code using a special modal syntax. You can also give us a Docker file, but it's actually, we support almost everyone just does it in Python internally.

• 25:29Speaker 03

  So in code, you can say, basically, you know, use Debian Slim and then install these Python packages like Transformers and Accelerate and Diffusers and a few other things. And then annotate the function to say, use that image. And then that's pretty much it. Now you can run in the command line, you can do modal deploy or modal run. And then it just takes that code, builds the container if it doesn't exist and runs it in the cloud. And that could typically take less than, if the image is already built, it typically takes about a second to take the code locally, spawn a container in the cloud, running that code. It works for any Python function.

• 26:05Chris Benson?

  I mean, that's dead simple right there.

• 26:07Speaker 03

  Yeah, and it works for any Python function and you can, you know, you can run pretty much any code you want because we support like, you know, fat containers, like meaning you can install Python packages, you can, you can install FFmpeg if you want transcodes and video, like you can install, you know, whatever thing you want. And we have a lot of functionality for manipulating images and building dependencies and doing pretty advanced stuff as part of that too. Pre-baking models into images is something people want to do sometimes to optimize cold start performance. But yeah, getting started with Modal, like we really, we really optimized for having that sort of like magic experience the first time you try Modal, like making it easy to just like, you know, install the Python package, set up a token and run code immediately in the cloud.

• 26:51Speaker 03

  We want that first experience to be magic and sort of set a tone for like, you know, what Modal is and like, you know, the fact that Modal, we think is a better way to work with the infrastructure in the cloud.

• 27:01Daniel Whitenack?

  So, one of the things I was wondering about, which I guess it was a surprise to me, like I didn't really think about it when I was first like using it. So, I don't know, everybody has their different setup, but usually I've got my like code editor over here and I've got like my terminal over here on maybe another monitor or something. So I've got both up and I was like writing my, it was a webhook in Modal and I had it like, you know, Python app, whatever. And when it's a webhook, then like the code runs and then Modal gives you this link where you can like ping like a development webhook. And of course, like I never get my code right the first time around.

• 27:42Daniel Whitenack?

  So like I bring up postman or something and I like try to hit my web like that link. And of course it like, I get whatever error and kind of without realizing, I just like went over to my code and I fixed it and I just saved the file and I saw over here in my terminal, like it just redeployed and gave me like the link again. I think that was like a really cool, like surprise for me, I guess is like, oh, I don't even have to like, I can just like keep this up over here in the terminal. How does that work exactly? And like, was that something that you stumbled upon or, because I found that a really satisfying way to develop because where it's like, oh, I just keep this up. I keep modifying the file and trying it until it works.

• 28:24Daniel Whitenack?

  And then I can just like Ctrl C and say Modal deploy and then I'm done, right?

• 28:29Speaker 03

  Yeah, for sure. I know I'm like harping on it, but like kind of thinking about like feedback loops and like, you know, the sort of iteration speed. As a CTO, I manage a lot of different teams. I manage data teams, front end teams and backend teams. And it's sort of interesting like how like different disciplines of software engineering have figured out their own iteration cycles, like the ability to get feedback loops very quickly. Back end engineers tend to build a lot, write a lot of unit tests. Like that's like their way. Like they write some code and then they run all the unit tests. Or maybe they run a specific unit test that you know is going to break. And they have that like sort of way to go to get the fast feedback loop. You go to front end engineers, they have kind of a setup like you just described. They have like one monitor with the website and then one monitor where they like write code. And when they save, it's like hot reloads the code.

• 29:11Speaker 03

  So I feel like sometimes like data and backend people don't give, you know, enough like credit to like front engineers. Like they have really figured out a lot of stuff around like software engineering for like fast feedback loops. And actually, if you look at like the modern tool chain for front engineering, I actually think it may always is like more advanced than than any other part of software engineering. And so that is the sort of feedback loop that I wanted to have with Modal. And like what I think makes engineers happy is that, you know, super snappy like feedback. You just have to save code and then it's like live in the cloud. Yeah, so, so we built that specifically for the web serving part of Modal. Uh, because that's something you, you kind of want to have is that ability to, we, we don't, you know, it's maybe less like your sort of visual feedback, but it's like the ability to like deploy something in the cloud and then you can like, you know, hit it with

• 29:55Speaker 03

  postman or curl or whatever immediately. Uh, yeah, I mean, under the hood, it's, um, it's all super complex. Actually, it refactored yesterday. It's kind of funny. We just monitored the file system and then when we see that any file was updated, we just reload the entire app, uh, in the sub-process and, uh, live patch the, the app running in the cloud. Uh, so it's pretty straightforward. We had a lot of that already built, so.

• 30:20Daniel Whitenack?

  I think the problems you've been solving for like the past two years are probably really complicated for you to like loop that into like the category of really simple problems. I think that would probably be quite complicated for many. Many people.

• 30:34Speaker 03

  Yeah, for sure. I guess like it's like simple in the sense that it's sort of, you know, we already built, you know, so much of the like underlying complexity to like make that easy relatively easy to support the hot reloading. Like the fact that we already built like so much complexity around like take code and deploy to the cloud and do that very quickly. You know, that's a very nice foundation to then like.

• 30:54Daniel Whitenack?

  Yeah, it's your sort of bread and butter.

• 30:56Speaker 03

  Yeah, yeah, like building that, you know, fast container, fast file system stuff. Uh, it's a lot of cool stuff that that unlocks.

• 31:03Chris Benson?

  So this is a particularly interesting episode, I would argue for me and probably for quite a few of our listeners that listen regularly. Because we're talking about something and Eric, we have the privilege of you as the person who's created this. But we also have Daniel, uh, whom I, you know, I've been working closely with and our listeners have been listening to. And hearing Daniel's passion and him building his own business on your platform. And we talked to lots of different companies. Um, and so it definitely has intrigued me in a way that not every different, uh, company owner, if you would, has. I'm kind of curious, I'm thinking about it from a slightly different perspective from Daniel, but you've really got me wondering like how to make this happen.

• 31:45Chris Benson?

  I work at a big company, as you know, um, we have big investments in kind of the, the big cloud providers as all large companies do. What are good strategies for companies to say, okay, we have so much in these other big names and stuff that are out there. How do we start using modal effectively? What are the kinds of things you've seen your larger customers do? In terms of migration over or things that you might recommend that enables something of a migration to be more seamless, less painful. Because normally when you think of large company migrations, they are almost always fraught with pain.

• 32:24Chris Benson?

  And misery and challenges for the IT crews. So how do people get to this thing that we're hearing about today? And mitigate all of those problems?

• 32:34Speaker 03

  Yeah, yeah. I mean, first of all, like admittedly, like we're fairly early and you know, so a lot of our customer base is early stage companies. Like starting from a clean slate who have absolutely zero infrastructure and that makes it a little bit easier.

• 32:45Chris Benson?

  It does.

• 32:46Speaker 03

  In part because there's like nothing legacy that they have to port over, in part also because they're just desperate for tools. And so the, the sales process is a little bit easier for us. I find that like the conversation when we talk to bigger customers is obviously like quite different. First of all, there's often an existing data platform that's already built in-house. There's of course also a security compliance question and that's something we're working on. I think long term, there's a lot of really cool stuff you can do around DPC peering and a lot of things to enable the, you know, big companies to have the security guarantees that they need. But I also think it's a separate conversation where like, you know, a bigger company, it's like this one person who's a decision maker, who has the credit card, there's another person who built a data platform,

• 33:29Speaker 03

  who, you know, now we're saying, oh, actually, we shouldn't use that, we should use Modal instead. And so, you know, it's a tougher conversation. And then there's maybe a data scientist and a third, you know, a third person is a data scientist who really want to deploy models. They don't really care about the infrastructure, but they heard good things about Modal. I tend to think in those conversations, it's about like finding like a niche use case. That's like, you know, low risk, that, you know, doesn't sit in sort of some sort of critical path of like, you know, the whole business relies on this. And, you know, so it could be some sort of greenfield, you know, something new, you know, deploying a model or a very simple pipeline. Something that, you know, maybe doesn't touch like super sensitive data or have like super critical like guarantees.

• 34:10Speaker 03

  So some like researchy project, like that's typically where I tend to start is, you know, and often trying to find people, you know, data scientists or machine learning engineers, who feel like the platform team doesn't really have time for them. Like, you know, they want something that lets them iterate quickly, you know, without having to bother the ops team. Those are probably the easiest conversations to have, you know, with like the bigger companies.

• 34:32Chris Benson?

  That's good guidance. I appreciate that.

• 34:34Daniel Whitenack?

  I think it's very fitting that like the platform is, at least right now, and please correct me if I'm wrong, is it is very Python centric in terms of like the development workflow and what's supported. Do you see this being sort of like, like you said, that you can support so many different types of jobs in apps in Modal? So, you know, on one side, you could say, well, this could become like very general purpose in some ways. Or it could like fill a really niche gap that obviously it is starting to fill. And just do that really well and like continue to kind of go deeper there. What do you see as kind of the path forward?

• 35:18Daniel Whitenack?

  Or maybe it's a both and with, you know, some things coming sooner than later.

• 35:22Speaker 03

  I think of Model as my 20 year project. Like, you know, like I'm finally building a tool I always wanted to have. And like, I want to spend, you know, rest of my career doing that ideally. I, I, so my sort of end goal is to build, you know, very general purpose set of tools that have helped data teams be more productive. That, that being said, kind of like what I said at the start of this show, like I, I just realized like that's a, you know, almost like make a little maniac vision. Like I, I, I think it all comes down to like in practice to finding something that resonates with customers. And, you know, drives growth and validates the net. And then sort of sequencing, kind of layering all like sort of adjacent product over time. We tend to think right now, like we have one, one sort of use case and one sort of target persona that works really well right now, which is, you know,

• 36:06Speaker 03

  deploying online machine learning inference. That, that I think, you know, is an area where we see enormous amounts of demand and traction. So kind of how that, you know, fits into sequencing. I think, you know, an obvious next step for us is to make fine tuning and training easier to do in Model. But also thinking about, you know, like pre-processing, you know, scheduling, retraining, sort of happens on a loop on a regular basis. Maybe thinking about like, you know, how do you move your data sets into Model to some extent too? Like hosting more like stateful applications. Uh, I, I think there's like a long list of sort of, you know, like layering on like step by step. More and more advanced features and gradually expand to take over.

• 36:46Speaker 03

  And because I think the demand is there, like, you know, and no one wants to have this like 35 different point solutions that they have to integrate themselves. Right. And, and a lot of the data landscape today, I think, is very fragmented. And, you know, as a result, a lot of data teams have to integrate like so many different vendors and kind of ductate them together. Like I, I think there's, you know, there's a big case to be made for either some sort of, you know, consolidation or, or some sort of like defragmentation of the space where fewer vendors do more. So long term, that's absolutely my vision is to, you know, we're starting with this like place right now. Similarly, in terms of languages, right? Like you mentioned, you know, Python versus other languages. We think Python right now is a great place to start. Uh, because that's 90, you know, percent plus of data teams use Python.

• 37:28Speaker 03

  But definitely think long term, you know, a lot of the infrastructure that we built is, you know, low level and it's written in Rust. It doesn't really care about what stuff is running. We think it could be, you know, great to add support for TypeScript or R or Go or Rust or whatever. So there is many different access to this, like in terms of like how we think about sequence and expansion.

• 37:47Chris Benson?

  I'm just saying, you saw me raise my hand. I love Rust. It's my current favorite language.

• 37:51Speaker 03

  Nice.

• 37:53Chris Benson?

  Go and Rust are on the backend and so but let me ask you a question that came to mind as you're going through that. As you're kind of exploring the world and and you have certain areas of focus, but there's also some kind of able to stretch out depending on different parts of the strategy you have. How do you see kind of to use a very generic open term, the edge out there, things that are not in the cloud? Do you see you doing anything in the future that would be kind of edge based? Or do you see yourself more as the cloud partner for things that might be out on the edge? And you have APIs and such available to those? How do you conceive either working with or including the edge in your overall strategy?

• 38:34Speaker 03

  I think edge is primarily useful for like very, very latency sensitive applications. And that's probably a segment of the market that we just feel like that's not what Model is going to be good at. Because, you know, if you do things in like Wasm or V8 isolates, like, you know, in that case, you can make it like kind of fast enough. But, you know, the way we focus on serverless right now is sort of fat, traditional like Linux, you know, distributions in containers or VMs. And that just has, you know, it's always going to have some non trivial overhead, you know, maybe a second, maybe like eventually we can get it to a few hundred milliseconds. I think to sort of edge workloads that people talk about, like, you know, that's when you really need like one millisecond, right? Like, and you're really trying to, either you're doing some like IOT type, you know,

• 39:19Speaker 03

  like controlling like devices for manufacturing or you're doing. High performance like CDN, like SEO type stuff, you know, where you want like your website to be absurdly fast. Those are the types of workloads I don't really think Modal is suited super well for, and I'm more than happy to let other vendors dominate that space. We tend to think on the time scale of like a few hundred milliseconds and up, that's where we focus right now.

• 39:43Chris Benson?

  No, that's a great answer. And definitely, I mean, trying to address every problem, you know, out there in the larger space isn't a successful approach. So hearing, when I talk to people and I hear, no, we're not going to go there, I usually take that as a very good thing in terms of focus and good strategy. So good to hear that.

• 40:02Speaker 03

  Cool. Yeah.

• 40:03Daniel Whitenack?

  Kind of as we wrap up here, I'd be curious to hear, like, obviously, you're very passionate about this project. You want to work on it for 20 years, like this is your life's work, it sounds like. What are the things that are on your mind right now in terms of the things that you're excited about seeing happen in Modal? And like over the next year, what are you like most excited about seeing come to pass as you continue working on the project?

• 40:29Speaker 03

  The thing that I personally spend the most time on is probably figuring out the like ergonomics of the SDK itself. Like in code, like how do you express programs that like execute in a distributed way in the cloud? And still making it feel like intuitive and easy to the user without having to think about the fact that, you know, this function runs in a different container than this function. We've made that work reasonably well for online inference, but I think, you know, when you go to like training and start dealing with file systems, there's certain things that are like still a little bit like gnarly, and I'm working a lot on that right now. So like making that user experience good and sort of intuitive, I think is really important. And a similar note, like Modal right now is like somewhat janky when you run it inside notebooks for some particular reasons.

• 41:13Speaker 03

  I'm not going to get into, but it's something I definitely want to make, you know, the user experience, like if you're running Modal inside a notebook, I think should obviously be, you know, we need to fix that too. It's fine, it's not like terrible, but you know, I definitely don't think it's like quite yet where it is if you run Modal in a script. There's always like backend stuff like we definitely need to like scale this up, you know, 10x or 100x the scale we are like, you know, we see a lot of demand. Modal does not have a publicly available signup right now. Like you can sign up and you go on a waitlist. And part of it is that just that like we want to have a little bit more control over the scale. Like there's a lot of work we need to do on the backend to build the foundational, you know, architecture, running all of this stuff. It's a very hard problem, like, you know, splitting, you know, essentially our own Lambda or our own Kubernetes.

• 41:57Speaker 03

  Uh, there's a lot of work we need to do on GPU support and in particular, cold start with GPU models and fast loading of GPU models. So those are some, there's a lot of cool work. We're spending a lot of time on there, especially when it comes to like, containers and in general like isolation and VMs like, you know, turns out that like supporting GPUs in a secure way i- in a multi-tenant environment is quite hard. Uh, so we're going very deep and like, you know, I'm reading about Linux device drivers and CUDA and like trying to understand all of those things. Yeah, I mean, those are all the things we're working on. I, I think in a year's time, like I think Modal, you know, we'll see a lot more traction, you know, Modal for like other things than just online inference.

• 42:42Speaker 03

  Like we're going to see a lot of people using Modal for training. We're going to see a lot of people using Modal for like parallelization. I think we're going to have, you know, much more like sort of, you know, customers on the enterprise side right now. Like we're focusing very much on the startups, but, but we're, we're doing, laying a lot of the security compliance work to be able to go up market. Uh, yeah, those are some of the things where I'm, I'm pretty excited about.

• 43:04Daniel Whitenack?

  Yeah, yeah, there's a lot to be excited about and yeah, please, uh, pass on my personal thanks again to the Modal team for making me look good, uh, today and recently. And, um, yeah, really excited about what you're doing and appreciate you taking time to chat with us.

• 43:20Speaker 03

  Yeah, of course. I'm also very excited about this, so always happy to talk about it.

• 43:33Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.