Erik Bernhardsson on Creating Tools That Make AI Feel Effortless

Today on No Priors, Elad chats with Erik Bernhardsson, founder and CEO of Modal Labs, a platform simplifying ML workflows by providing a serverless infrastructure designed to streamline deployment, scaling, and development for AI engineers. Erik talks about his early work on Spotify’s ML algorithms, what Modal offers today, and his vision for building an end-to-end solution for AI engineers. They dive into GPU trends, cloud vs on-premise setups, and when to train custom models vs use off-the-shelf solutions. Erik also shares his thoughts on the evolving role of AI in fields like coding, physics, and music. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Bernhardsson Show Notes: 0:00 Introduction 0:22 Erik's early interest in ML infra 1:22 Founding Modal Labs 4:17 State of GPU use today and what’s to come 7:14 Modal's end-to-end vision 9:00 Differentiating amongst competition 10:20 Cloud vs on-premise 12:35 Popular AI models 13:20 Gaps in AI infrastructure 14:55 Insights on vector databases 16:48 Training models vs off-the-shelf models 17:47 AI’s impact on coding and physics 22:14 AI's impact on music

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Today I'm chatting with Eric Bernhardsson, founder and CEO of Modal. Modal developed a serverless cloud platform tailored for AI, machine learning and data applications. And before that, Eric worked at Better.com and Spotify where he led Spotify's machine learning efforts and built the recommender system. Well, Eric, thanks so much for joining me today on No Priors.

• 0:21Speaker 01

  Yeah, thanks. It's great to be here.

• 0:23Speaker 00

  So if I remember correctly, you worked at Spotify and helped build out their ML team and recommender system. And then we're also at Better.com. What inspired you to start Modal and what problem were you hoping to solve?

• 0:33Speaker 01

  Yeah, I started at Spotify a long time ago, 2008, and it's been seven years there. And yeah, I built a music recommendation system. And back then, there was like nothing really in terms of data infrastructure. Hadoop was like the most modern thing. And so I spent a lot of time building a lot of infrastructure. Particularly I built a workflow scheduler called Luigi that basically no one uses today. I built a vector database that called Anoy that, you know, for a brief period, people used, but no one really uses today. So I spent a lot of time building a lot of that stuff. And then later at Better, I was the CTO and thinking a lot about like developer productivity and stuff. And then during the pandemic, I took some time off and started hacking on stuff. And I realized I always wanted to build basically a better infrastructure for these types of things like data AI, machine learning.

• 1:17Speaker 01

  So pretty quickly I realized like this is what I wanted to do. And that was sort of the genesis of Modal.

• 1:22Speaker 00

  That's cool. How did that approach evolve or what are the main areas that the company focuses on today?

• 1:28Speaker 01

  So I started looking into, first of all, just like, what are the challenges with data AI, machine learning infrastructure? And started thinking about from like a developer productivity point of what's a tool I want to have? And I realized a big, a big sort of challenge is like working with the cloud is arguably kind of annoying. Like as much as like I love the cloud for its power that it gives me, and I've used the cloud since, you know, way back 2009 or so. It's actually pretty frustrating to work with. And so in my head, I had this idea of like, what if you make cloud development feel almost like as good as local development, right? Like has this like fast feedback loops? And so I started thinking about like, how do we build that? And realized pretty quickly like, well, actually, we can't really use Docker and Kubernetes. So we're going to have to throw that out and probably going to have to build our own file system, which we did pretty early, and build our own scheduler and build our own

• 2:13Speaker 01

  container runtime. And so that was like basically the two first years of Modal, just like laying all that like foundational infrastructure layer in place.

• 2:21Speaker 00

  And then in terms of the things that you offer today for your customers, what are the main services or products?

• 2:28Speaker 01

  Yeah, so we're infrastructure as a service. So which means like on one side, we run a very big compute pool, like thousands of GPUs and CPUs. And we make it very easy to get, you know, if you need 100 GPUs, we can typically get you that within seconds. So sort of one big multi-tenant pool, which means like capacity planning is something we kind of take, it's something we solve for customers. They don't really need to think about reservations. We always provide a lot of on-demand GPUs. On the other side, there's a Python SDK that makes it very easy to build applications. So the idea is like you write code, basically like functions in Python. And then we take those functions, turn them into serverless functions in the cloud. We handle all the containerization and all the infrastructure stuff. So you don't have to think about all this sort of Kubernetes and Docker and stuff.

• 3:12Speaker 01

  And the real killer app, as it turns out, like we started this company pre-GAN AI, but as it turns out, the main thing that really started driving all the traction was when Stable Diffusion came out. And a bunch of people came to us and were like, hey, actually, this looks kind of cool, like you have GPU access. It's very easy to, you know, you don't have to think about, you know, spinning up machines and provisioning them. So that was like our first sort of killer app was like just doing GAN AI in a serverless way with the focus of diffusion models. Like now, we actually, we have a lot more of a different modalities. Like a lot of, a lot of usage is still like text to image, but we also see a lot of audio and music. So one example of a customer, I think, is super cool, building really amazing stuff is Suno, which does AI-generated music. So they run all their inference on modal, very large scale.

• 3:57Speaker 01

  There's a lot of customers like that, sort of dealing with like, you know, building cool GAN AI models. In particular, I would say in the modality of like audio, video, image and music, stuff like that.

• 4:06Speaker 00

  It's cool. And I think Suno is using all like transformer backbone now for stuff, right? Versus a diffusion model based thing, so.

• 4:11Speaker 01

  I think it's, I think it's a combination of both. I'm not sure.

• 4:16Speaker 00

  Yeah, I think they talk about it publicly. That's the only reason I mention it. You wrote in October, this post, I think it was called, "The future of AI needs more flexible GPU capacity." And in general, what I've heard in the industry is that a lot of ways that people use GPU is reasonably wasteful. And so I'm a little bit curious about your view on flexibility around GPUs. How much is actually used versus wasted? How much optimization is left? You know, even just with existing types of GPUs that people are using today.

• 4:42Speaker 01

  Yeah, I mean, GPUs are expensive, right? And I think it's sort of kind of as like a paradox. It's like means that cloud, you know, a lot of the cloud capacity is like you, they, you know, the only way to get it is to sign long-term commitments, uh, which I think for a lot of startups, it's really not the right model for how things should be. Like, I think the amazing thing about the cloud was always to me that like you have on-demand access to like whatever many CPUs you need. But for GPUs, like the main way to get access has been over the last few years due to the scarcity has been to to sign long-term contracts. And I think fundamentally, that's just not how startups should do it, right? Like, you know, and that kind of get it has been sort of supply-demand issues. But but um, just, you know, looking at the CPU market, the the fact that you have like

• 5:23Speaker 01

  instant access to thousands of CPUs if you need it, like my vision has always been there should be the same thing for GPUs. Uh, and that means, you know, especially as we shift more to inference. I think for training, it's been sort of a less of an issue because like you can sort of just like make use of the training resources you need. But for inference, especially, you don't even know how much you need, right? Like in advance, like it's very volatile. Uh, it's a big challenge that that we solve for a lot of customers is we we're fully usage-based. So when you run things on Model, we charge you only for the time the container is actually running. Uh, and uh, and that's a that's a massive hassle for for customers traditions like doing the capacity planning and thinking how many GPUs. And then and then having the issue like either you over provision and you you're paying for a lot of idle capacity or you're under provision and then you have, you know, when

• 6:08Speaker 01

  you're under the capacity shortage, it's like you have degrade degradation in service. And so whereas with Model, we can handle these like very bursty, very unpredictable workloads really well because we basically like take all these user workloads and just run a big pool of of thousands of GPUs across many different customers.

• 6:24Speaker 00

  Yeah, one of the things that always struck me about training is to your point, you kind of spin up a giant cluster. You run a huge supercomputer, right? And then you run it for months in some cases and then your output is a file. And that's literally what you've generated, you know? It's kind of insane if you think about it. And that file in some sense is a representation of the entire internet or some corpus of human knowledge or whatever. And then to your point with inference, you need a bit more flexibility in terms of spinning things up and down, or alternatively, if you're doing shorter training runs, or certain aspects of post-training, you may need more flexible capacity to deal with.

• 6:57Speaker 01

  Totally. And that's something we're really interested in right now. Traditionally, most of modal has always been inference. That's been our main use case. But we're really interested also in training. And in particular, probably focused more on these shorter, very bursty experimental training runs, not the very big training runs, because I think that's a very different market. So that's like a very interesting thing.

• 7:15Speaker 00

  How do you think about meeting people's end-to-end needs? So I know that there's a lot of other things that people do. There's, you know, a lot of people are using RAG to basically augment what they're doing, or, you know, there's a variety of different things that people are now doing at time of inference in terms of using compute to, you know, take different approaches there, you know. I'm a little bit curious how you think about the end-to-end stack of things that could be provided as infrastructure and where modal focuses or wants to focus.

• 7:44Speaker 01

  Yeah, totally. I mean, our goal has always been to build a platform and cover the end-to-end use cases. It just turned out that inference was, we were well positioned to focus on that as our first killer app. But my end goal has always been to make engineers more productive and focus on what I think was like the high-code side of ML. Like, I think we're, like, our target audience tends to be more like sort of traditional, like, ML engineers, like people building their own models. But there's many different aspects of that. There's like the data preprocessing, then there's the training, and then there's the inference, and there's actually probably like even more things, right? Like, you know, having feedback loops where you gather data and like, you know, online ranking models and all these things. And so my goal for modal has always been to cover all of that stuff. And so it's interesting, you see a lot of customers now, we don't have a training

• 8:29Speaker 01

  product, but a lot of customers use modal for batch preprocessing. So they use modal to, you know, maybe they're training a video model, so maybe they have like petabytes of video. So then they use modal actually, maybe with GPUs even, to like do feature extraction, and then they train it elsewhere. And then they come back to modal for the inference. So for us to do the training makes a lot of sense. And in general, I think it's a lot of, you know, it makes a lot of sense to sort of build a platform where you can handle the entire sort of machine learning lifecycle and many other things related to that. Also, the data pipelines and nightly batch jobs and all these things.

• 9:00Speaker 00

  Yeah, I mean, what you describe is a pretty broad platform-based approach. I think there's a handful of companies who are sort of in your general space or market. How do you feel that modal differentiates from them?

• 9:12Speaker 01

  I think, first of all, we're cloud native, like we're just like cloud maximalists. Like, we went all in and said, like, basically, we're going to build a multi-tenant platform that runs everyone's compute. And the benefits of that are, I was like, very tremendous because like we could just do capacity management much better. And that's one of the ways we can offer like instantaneous access to hundreds of GPUs if you need to. Like, you can do these like very bursty things, and we just give you lots of GPUs, right? I think the other benefit or the other sort of differentiation is to be very general purpose. We focus on sort of what I think, as I mentioned, like high code, like we run custom code in our containers, in our infrastructure, which is a harder problem. Like, containerization and running user code in a safe way is a hard problem. And then dealing with container call start. And like I mentioned, we have to build our own scheduler, we have to build our own container,

• 9:57Speaker 01

  runtime and our own file system to boot containers very quickly. And I think so, unlike many other vendors, like they're only focused on, say, inference or maybe only LMs. Our approach has always been to build a very general-purpose platform and sort of, you know, in the long run, I hope to sort of, that sort of manifestation will be more clear because I think there's many other products we can build on top of this now that we have the compute layers sort of kind of becoming more and more mature.

• 10:22Speaker 00

  When I talk to large enterprises about how they're thinking about adoption of AI, many of them already have their data on Azure or GCP or AWS, they're running their application on it. They've bought credits in the marketplace, they want to spend resident, they've already gone through security reviews, you know, they've kind of done a lot. And they worry about things like latency or pings out to other third-party services versus just running on their own existing cloud provider or their hyperscaler that they work with, or set of hyperscalers, you know, many of them actually work across multiple. How do you think about that in the context of modal in terms of your own compute versus hyperscalers versus, you know, the ability to run anywhere?

• 10:59Speaker 01

  Yeah, totally. And of course, there's also a sort of security compliance aspect of this. Like, I think, you know, it is a, it is a, you know, challenge. I look back at when the cloud came, and I remember back in like 2008, 2009, and the cloud came. And my first reaction was like, how the hell, like, why would anyone put their computer in someone else's computer? And like, run that. And I think, you know, to me, that was just like insane. Like, why would anyone do that? But over the next couple of years, I was like, actually, it kind of makes a lot of sense. And I think now even like among like enterprise companies, like this is sort of recognition that like, yeah, actually, probably our computer is more safe in the big hyperscales. And in a similar vein, I remember talking to Snowflake back in, say, 2012 or something like that. And they had a sort of similar approach where like, they basically said like, we're going to run databases in the cloud.

• 11:44Speaker 01

  And it's not going to be in your, you know, or maybe in your environment, but like, we're in infrastructure as a service. And I thought that was nuts. And then obviously, like, I think Snowflake now is a very large, you know, publicly traded company. I think they showed that like, infrastructure as a service makes a lot of sense. And so I, I think there is a little bit of resistance to adopting this like multi-tenant model. But I think, you know, when you look at like security and adoption of cloud, I think we have a lot of tailwinds blowing in our direction. I think security is moving away from sort of a network layer into a, into a application layer. I think bandwidth costs are coming down. I think there's a lot of tricks you can do to minimize bandwidth transfer costs. You can store data in like R2, for instance, which has zero EGRs fees. It's something that I think is realistically going to, you know, mean we're going to have to push a lot.

• 12:29Speaker 01

  But I think there's so many benefits of this multi-tenant model in terms of capacity management. That to me, it is very clearly like a big part of the future of AIs, like running a big pool of compute and slicing it very dynamically.

• 12:40Speaker 00

  You, you mentioned earlier that one of the things that really caused early adoption of Modal was stable diffusion and sort of these, these open source models around ImageGen. Are there any open source projects or models that you're seeing be very popular in recent days? Or in the last couple of months that have really started taking off?

• 13:01Speaker 01

  That's a good question. I think if anything, it's actually been a little bit of a shift towards more like proprietary models, but like proprietary open source models, I guess. So like Flux, I think most recently has been, you know, a model that's getting a lot of attention. I'm personally very interested in like audio. I think audio is like very underexplored. I think there's a lot of opportunity for open source models in that space, but I don't think we've seen anything really cool yet.

• 13:24Speaker 00

  What else do you think is missing in the world today in terms of AI infrastructure or infrastructure as a service?

• 13:30Speaker 01

  I'm very biased, but I think Model is missing. Like, basically a way to like for engineers to take code and run it. And look, I'm very bullish on like, you know, code and like people wanting to write code and building stuff themselves. I think outside of sort of L M space, which is like a very kind of different world, in my opinion, I think there's always going to be a lot of applications where people want to train their own models, they want to run their own models, or at least like run other models but have like very custom workflows. And I just don't think there's been a great way to do that. It's like pretty painful to do that. And so I think that's pretty exciting. I think on the storage side, there's some other really exciting stuff. Like, we haven't really touched storage at Model. Like, we focus very much on compute. So I'm personally very interested in sort of vector database. Like, how's that going to evolve? I don't think

• 14:13Speaker 01

  anyone really knows. I'm pretty interested in like, you know, more efficient storage on training data. I'm also very interested in like, I guess another thing I'm very fascinated by right now is training workloads. In order to train large models efficiently, you have to really spend a lot of money and time setting up the networking. So one of the things I'm really excited about is what if you don't, you know, what if we can make training less bandwidth hungry? Because I think that would actually change a lot of the infrastructure on training, where you can now like kind of tie together a lot of GPUs in different data centers and not have to, you know, have this like very large data centers with like, you know, infinite bandwidth stuff. So that's like another sort of infrastructure thing. I'm looking forward to see more development on.

• 14:58Speaker 00

  How important? So there's sometimes been a little bit of debate around vector DBs. And you mentioned that you actually built one when you were at Spotify. I think Spotify today hit $100 billion in market cap. I think it's one of the first European technology companies to get there, which is pretty cool. So a lot of folks I know may use one of the existing vector DBs, or in some cases are just using Postgres with PG Vector, right? How do you think about the need for vector databases, a sort of standalone? Pieces of infrastructure versus just, you know, adopting Postgres versus doing something else.

• 15:34Speaker 01

  Yeah, I feel like everyone's debating that. I don't know necessarily. Like, I think there's a lot of, there's a case to be made that, you know, you can just stick everything into relational database and you're, you're fine. To me, like the, the bigger question is like in the long run, like, you know, if you think about like, what's like an AI native data storage solution? Like, I don't even know if it's like necessarily has the same form factors, the same interface as, as a database. So that's actually a bigger question that I'm more excited about. It's like, I think people look at like vector databases and like, you know, whether it's relational or not, they sort of shoehorn it into, into this like, you know, sort of old school model of like you, you put data, you get data back. But I don't know. I, I think there's like a lot of room to sort of rethink that in the age of AI and have very different like, you know, interaction models with that data. I know that sounds a little fluffy.

• 16:18Speaker 00

  Yeah, it's super interesting. Could you say more on that?

• 16:20Speaker 01

  I mean, like, one thing I, I think a lot about is like, maybe the database itself be like the embedding engine, right? Like, instead of like, you put a vector in there, you, you know, you search by that vector. I, I think there's a lot of, you know, the more native, like AI native storage solution would be you put text in, you put, you know, video in, you put image in, and then you can search by that. Like, to me, that would be like a more sort of native, AI native sort of storage solution. So that's like one line of thought that I've had is like, maybe we're just, we're just like so early to this that like, I, I think it's gonna take five, 10 years for it to really, for it to shake out.

• 16:52Speaker 00

  Yeah, that's really cool. I guess one other thing that you mentioned was more people seem to be training their own models, at least in a lot of the areas that, that Modal works with.

• 17:02Speaker 00

  Do you think there's any heuristic that people should follow in terms of when to train their own model versus use something off the shelf?

• 17:07Speaker 01

  I think eventually, like for any company where model quality really matters, unless you kind of train your own model in the end, like, I feel like it's gonna be hard to sort of defend the fact that like, you know, you're, you have a better solution. Cause like, otherwise, like, what's your moat? Like, if you don't have your own model, like, you need to find some, a moat somewhere else in the stack. And, and that might be possible to find. Uh, it might be somewhere else for a lot of companies. But I think at least if you have your own model and that model clearly is better than anyone else, uh, then that sort of inherently is a moat in itself. I think it's more clear in the, in, outside of the LM space, when, when people are building audio, video, image models. Uh, I think if, if that is your core focus, like, it's very clear to me, like, you kind of have to train your own models in that case.

• 17:51Speaker 00

  Yeah. If I remember correctly, you were an I O I gold medalist.

• 17:56Speaker 01

  That's right.

• 17:57Speaker 00

  Obviously, you think a lot about code and coding. And how do you think that changes with A I over time? Or do you have any contrarian predictions on what happens there?

• 18:06Speaker 01

  I don't know if this is contrarian, but like, I actually think that like, you know, this is just like one out of many improvements in developer productivity. And, you know, you look back at like, you know, whatever, like compilers was originally like, you know, a tool that made developers more productive and then like higher level programming languages and databases and cloud and all these things. And so like, I actually don't know if like A I is like, you know, different than any of those changes in the hindsight. And so, and by the way, like every time that's happened, you know, it turns out like there's so much latent demand for software that actually like the number of software engineers goes up. So like, I feel like you look back at like, you know, last 40 years of software development, like every decade, engineers get like 10 times more productive due to better frameworks or better, you know, tooling or whatever.

• 18:51Speaker 01

  And it turns out, actually, that just unlocks more latent demand for software engineers. So I'm very bullish on software engineers. I think it would take a lot to sort of destroy that demand. I think people look at a lot of like A I as like it kind of fixed something, but in my opinion, it's like, no, it's a good, just got to unlock more latent demand for more things. So I'm very bullish on software engineering.

• 19:09Speaker 00

  And then I guess the other field that you touched a long time ago was, I think you won a Swedish physics competition in high school. And I'm curious if you followed any of the physics based A I models or some of the simulation really like, that's an area that strikes me as very interesting. And the way you think about the models for it are different and yeah.

• 19:29Speaker 01

  I did win the Swedish high school physics competition. I was a total mathlete nerd when I was, you know, my teenagers.

• 19:36Speaker 00

  Okay. Yeah, I think it's a really fascinating area right now. Like it's one of those areas that seems like there's some real reinvention needed and not as many people are working on it. So it's one of the areas I'm kind of excited about just in terms of there's lots and lots of different applications that you could start to come up with relative to it.

• 19:52Speaker 01

  Yeah. I think, I mean, like physics, in my opinion, it's like, you know, it looked very like the golden era of physics, like the 20s and 30s and 40s. I kind of feel like it's like, it hasn't really evolved much the field. So I don't know, maybe I would love for you to be right, that there's like a resurgence of, you know, new physics based models.

• 20:06Speaker 00

  Yeah, I don't know if it would necessarily help in the short run with basic research. I think it just helps with simulation. It kind of feels like physics as a field really kind of doubled down on sort of the Ed Witten path of physics and maybe got a little bit lost there or something. I'm not sure.

• 20:21Speaker 01

  Are you talking about like more like material, like doing more like compute based?

• 20:24Speaker 00

  It's kind of like ANSYS or other companies where, you know, you simulate an airplane wing, you simulate load bearing in a.

• 20:30Speaker 01

  I see. So like HPC, that's always existed, right? Like especially in like, you know, oil and gas and stuff like that.

• 20:38Speaker 00

  But it's a lot of kind of small bespoke, kind of fine tuned or hand tuned models for specific things versus.

• 20:44Speaker 01

  Yeah.

• 20:45Speaker 00

  You know.

• 20:45Speaker 01

  I mean, meteorology is like something I actually think like deep learning should like change, right? Like it sort of makes a lot of sense. Like, you know, deep learning should be very good at like, you know, predicting, you know, turbulence and things like that. Like, because turbulence is actually very hard to solve in traditional physics models, right? And so deep learning should in theory, I kind of feel like makes a lot of sense.

• 21:03Speaker 00

  Yeah, I think there's been a couple of papers on that out of Nvidia. And then I think Google has a team that's worked on it. And so there's a couple of different sort of weather simulation teams that have started to publish some pretty interesting stuff, it seems.

• 21:15Speaker 01

  Yeah. I mean, I would also point to like an adjacent area, like biotech, I think is like been, you know, computational methods have been enormously successful, right? Like if you look at protein folding in particular, but also other things like sequence alignment and things like that. And that's actually a field where we start to see a lot more usage of models as well. Is there's a lot of, I feel like there's like a kind of a resurgence of computational biology. It's really exciting.

• 21:38Speaker 00

  That's really neat. Yeah. Are there specific use cases you see people engage with most across your customer base relative to the sciences or?

• 21:47Speaker 01

  There's a lot. I'm not a bio person, so this is kind of superficial, is kind of looking at our customers. But like one thing I've seen a lot is actually medical imaging, because my understanding is like with modern methods, like you can do like very automated, like, you know, Get like millions of experiments and do automated electron microscope imaging of that. So we've actually seen quite a lot of customers like use Model for like then processing and doing computer vision on those images, which is kind of cool.

• 22:15Speaker 00

  It's really cool. Is there any area that you're most excited about from a human impact perspective for some of these models?

• 22:22Speaker 01

  You know, with my background at Spotify, like I think Suneo is like, to me, a very exciting thing. I think it's still like very early, sort of AI-generated music. You can still hear that it's like not right. It's sort of uncanny value a little bit. But like Suneo is like every generation of their model is like getting better and better. And first of all, like music in itself tends to be like sort of always like one of the first areas where you see real impact of new technologies, whether you know Spotify or like iTunes or piracy or like all these things or gramophones going back, right? So I always think of music as like an exciting area for that in that sense. Like it always shows like the opportunity of new technologies. And I also think like Suneo is like fundamentally something you couldn't have done before Gen AI. So that to me is like really exciting. It's like

• 23:04Speaker 01

  it's sort of really pushing the frontier, enabling a completely new product that there's like, there's no way this Suneo could have existed five years ago.

• 23:11Speaker 00

  That's cool. Well, I think we covered a lot today. Thanks so much for joining me.

• 23:15Speaker 01

  Yeah, thanks a lot. It was great.

• 23:18Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.