Multi-GPU training is hard (without PyTorch Lightning)

William Falcon wants AI practitioners to spend more time on model development, and less time on engineering. PyTorch Lightning is a lightweight PyTorch wrapper for high-performance AI research that lets you train on multiple-GPUs, TPUs, CPUs and even in 16-bit precision without changing your code! In this episode, we dig deep into Lightning, how it works, and what it is enabling. William also discusses the Grid AI platform (built on top of PyTorch Lightning). This platform lets you seamlessly train 100s of Machine Learning models on the cloud from your laptop. Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogSnowplow Analytics https://snowplowanalytics.com/ – The behavioral data management platform powering your data journey. Capture and process high-quality behavioral data from all your platforms and products and deliver that data to your cloud destination of choice. Get started and experience Snowplow data for yourself at snowplowanalytics.com https://snowplowanalytics.comChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: William Falcon – Website https://www.williamfalcon.com, GitHub https://github.com/williamFalcon, X https://x.com/_willfalconChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: PyTorch Lightning | Website https://www.pytorchlightning.aiPyTorch Lightning | Twitter https://twitter.com/PyTorchLightninPyTorch Lightning | LinkedIn https://www.linkedin.com/company/pytorch-lightningPyTorch Lightning | GitHub https://github.com/PyTorchLightning/pytorch-lightningGrid.ai | Website https://www.grid.aiGrid.ai | Twitter https://twitter.com/gridai_Grid.ai | LinkedIn https://www.linkedin.com/company/grid-aiGrid AI, From the Makers of PyTorch Lightning, Emerges From Stealth With $18.6m Series A to Close the Gap Between AI Research and Production https://www.businesswire.com/news/home/20201008005227/en/Grid-AI-From-the-Makers-of-PyTorch-Lightning-Emerges-From-Stealth-With-18.6m-Series-A-to-Close-the-Gap-Between-AI-Research-and-ProductionGrid AI raises $18.6M Series A to help AI researchers and engineers bring their models to production https://techcrunch.com/2020/10/08/grid-ai-raises-18-6m-series-a-to-help-ai-researchers-and-engineers-bring-their-models-to-production Grid.ai Launches Platform to Train Machine Learning Models in the Cloud https://www.prnewswire.com/news-releases/gridai-launches-platform-to-train-machine-learning-models-in-the-cloud-301267901.htmlUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Today's models are actually not models. Like, we need a new name because there's something that doesn't exist. Like, what do you call an encoder and a decoder working together to make an auto-encoder? Or variational encoder, right? They're not models. It's collections of models interacting together. Same for transformers, right? So that's really what the Lightning module is about. You pass in these models into it, and then how they interact together is abstracted by that, right? And I think that's a missing abstraction that was not there. So it's important to decouple that because now I have this single file that's completely self-contained, that I can now share with my team across in a different division. And their problem might be completely different with a different dataset, and they don't have to ever change the code on that model. All they have to do is change what hardware they're using and then what the dataset is. So it makes code extremely interoperable, right?

• 0:44Speaker 00

  So I think people come to Lightning because they want to train on multiple GPUs and so on, but that's only like a very small part of it. I think once you get into it, you see that the rest of it is the ability to collaborate with peers and be able to have reproducible and scalable code.

• 1:01Speaker 01

  Big thanks to our partners, Linode, Fastly, and Launch Darkly. We love Linode, they keep it fast and simple. Check them out at linode.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 1:19Speaker 01

  This episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal textbooks and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos, and all their conference talks. Plus, you can learn by doing with live, online training courses and virtual conferences, certification practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics: machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check

• 2:04Speaker 01

  out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at o'reilly.com/changelog. Again, o'reilly.com/changelog.

• 2:21Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter, we are at Practical AI FM.

• 2:47Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with S I L International and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. And this week, we have a really exciting show. I'm pumped to talk about this. We have William Falcon with us, who is creator of PyTorch Lightning and CEO of Grid AI. Welcome, William.

• 3:14Speaker 00

  Well, thank you guys for having me. Really excited to chat with you.

• 3:17Daniel Whitenack?

  Yeah, yeah, we are as well. And I think I might have even mentioned this to Chris on our Slack channel, but I saw you like on Twitter when Grid AI was launched, there was like a screencast of like, this is some things that you can do with Grid AI. And it was one of those moments, I don't know if you've ever seen like a Kelsey Hightower demo in like the Kubernetes world or something like that, but it was one of those moments where I was like, things just sort of snowballed and then all of a sudden you were running like, you were running models on all of these GPUs in the cloud with very little effort and it was pretty cool. So I'm excited to dive into that at some point.

• 3:51Speaker 00

  Yeah, I'll start to share it.

• 3:53Daniel Whitenack?

  Yeah, cool. So maybe before we get to there, let's maybe start at PyTorch Lightning. People might've heard of PyTorch, they might've heard of Lightning. I know Lightning kind of shows up in my Twitter feed quite a bit. Could you just give us a little bit of context for what PyTorch Lightning is and how people can use it, where it might fit into people's workflow?

• 4:16Speaker 00

  Yeah, so I think, you know, I'll talk a little of my experience to understand the motivation behind it, right? Because I think my sense from speaking to people in the community is that we've all had very similar problems and thought about very similar approaches, right? The difference is, you know, we open sourced this and a lot of people started contributing to it. So I started out, you know, I was a software engineer and I was working in finance. And before that, I guess, I was an undergrad and I was starting, you know, to do research. And I'd been working as a software engineer. And when I got into AI research, it was in neuroscience, right? So computational neuroscience. And we were trying to take neural activity from the brain and trying to reconstruct what generated that. Right? And that was in the context of, you know, eyesight, basically.

• 5:00Speaker 00

  And so what happened there is none of us were like really, really big engineers in deep learning. Like, we weren't experts, right? And so I started training models and, you know, back then I was using Theano, right? Which is like a very old framework. And I remember the first time we got something wrong on GPU and it was like magical. Cause suddenly my time went from months to like a few days and I was like, great. And the research continued. And what I found myself doing over and over back then was I'd have an idea about something that wasn't quite that. Right? So in neural decoding, it's basically like translating a sequence of signals into something, an image or another signal. Right? So it's a translation problem in essence.

• 5:41Speaker 00

  And so you could do things like GANs, auto encoders, you could do things like regression, so many ideas. And I would want to try a few different approaches with my teammates. And we would have to copy the code over and over again, right? You would either fork the project and then kind of like copy that code over. And then if something new came out, like multi GPU training, you have to then write it into all the code that you did. And so suddenly you're maintaining like 10 different files that are all doing the same thing. And I started abstracting that into like a kind of joint class. And I think we all honestly, I think all of us kind of do this at some point. And I think at that point I had been using SK learn for a while. So I loved like their fit and all those methods. And so I was like, okay, well, whatever, just call it fit and do that. And then I transitioned to TensorFlow because we needed to get into multiple GPUs. And it was really hard to do in Theano.

• 6:27Speaker 00

  So that took our training time dramatically down. And then, you know, continue working on it for a while. But the problem is like, it just continued. Every time I wanted to do something new, you had to copy that code over. And then new things came out all the time. Like, there was just a different way of training. And it was really hard to go back and copy and paste all that stuff. I kind of, I left that project for a bit and then kind of went into the startup world, right? And I spent a few years putting NLP models into production. And there, it was less about focus on training and more about deploying models. And so I was just like, cool, quick baseline, and then just like, put that thing in there and see what happens, right? I was less concerned about solving like a very unique problem and more about, hey, I have the data here. I just want, I don't care what the model is, I just want to see some results, right?

• 7:11Speaker 00

  So we got that working and ended up, you know, scaling that to a company that got acquired. And that was basically using NLP to help, you know, low-income, first-generation students figure out how to pay for college over text message, which was really cool. And from there, I started my PhD and kind of like, started that research flow again. And then, you know, coming from the startup world, I was like, how do I bring that speed and agility to research, right? Because we all know this, and I think our path he talks about this. I mean, we all know this firsthand, but like, the outcome of doing anything with AI nowadays is honestly a function of how fast you iterate through ideas, right? Because like, ninety percent of your ideas are gonna fail. And then one or two are gonna work, and then you're good to go.

• 7:52Speaker 00

  So literally, just how fast can you power through those ideas is probably the single biggest predictor of if that thing's going to work or not. So I knew that, and I wanted to bring that ability to my, you know, PhD research. I was like, hey, maybe I can finish this thing in like, three years, right? As opposed to six or whatever.

• 8:08Daniel Whitenack?

  Ambitious.

• 8:09Speaker 00

  Yeah. Looking back now, it's not a good idea, but yeah, that was, that was the goal, right? And so.

• 8:15Daniel Whitenack?

  I know the feeling.

• 8:15Speaker 00

  Yeah, and so I took my code from my undergrad days and, you know, kind of brushed it off. And then, at that point, I had to reswitch to PyTorch. So I was like, okay, well, let me just rewrite this thing in PyTorch and see how it goes. So I started working with, again, NLP at that point, and then we moved into like audio research, right, to do speech synthesis and so on. And all of that using the same code, right? So it was interesting because like the first code was for NLP and then I modified it to work for audio and then vision and so on. And then eventually, I don't think it was quite there at that abstraction level yet because I was still having to do a lot of bespoke code. But then, I don't know what happened, like, I guess over the winter, something clicked and then, you know, the trainer got factored out and then it just became obvious that at that point, you need to separate the model from the hardware. And so that's what Lightning became.

• 9:00Speaker 00

  Then I open sourced it and then I jumped Facebook Research that summer as an intern at Fair and, you know, continued my PhD research. And there, you have a giant cluster, right? And I was like, okay, well, you know, if I have Facebook resources, what can I do? And, and, you know, very ambitious in terms of like trying to do research ideas. So we were trying to scale up like massive data sets on the cluster as much as we could, right? So I was consistently training, you know, 500 GPU models, that kind of stuff, all the time at Fair with this framework. And then, you know, people noticed because that the cluster, there was like a handful of teams across Facebook that was using a cluster that efficiently, but the rest of the teams weren't because like it takes a lot to do, you know, training at scale. And so, you know, I started working with those people because they're experts at this, right?

• 9:45Speaker 00

  And so we embedded a lot of those practices into Lightning and then ended up with a framework now that can do really scalable training. And then at that point, you know, there was some adoption internally, then adoption externally, and then it just kind of took off after that. But, you know, I came at it from how do I move really fast through research, knowing what I know about putting models into production as well and knowing what I know about doing research as well, right? So it's just kind of like having both requirements made it really interesting. And what's really cool now is that it's evolved into, you know, my vision really was, you and I, all three of us are going to code the exact same thing in our own projects, right? We're going to code half precision, we're going to code stochastic weight averaging, we're going to code whatever new thing comes up. But why waste that effort? Like, that's not the job.

• 10:27Speaker 00

  The job is to like, you know, if you're Lockheed Martin, I don't know, predict metal, whatever, right? Like, find, you know, deficiency in materials. I don't know what you guys do there, right? But. That'll work, that'll work.

• 10:38Daniel Whitenack?

  I think that's exactly what Chris does, I assume.

• 10:41Speaker 00

  So that's the goal. The goal is not to figure out how to implement, you know, stochastic weight averaging, right? So what's cool now is that, I mean, I think we're approaching 500 contributors, but these are all like top researchers and PhDs all over the world who implement these things and put them into papers. And then, you know, within a few hours, it's ready and available for everyone. So do you have to know how half precision works with, you know, I don't know, on GPUs? You don't, right? But you just know that it's going to save you memory. And so it's been basically turned into a community project. And my vision was really like, can we build like the world's research lab, basically, right? Can we have all access to top researchers and resources? And that's what's happened so far.

• 11:21Chris Benson?

  So I noticed as you're kind of going through the story, it seems like as you progressed over those years through the different aspects of your own life, and you're kind of looking at the same problem through multiple lenses as you're going from, you know, software development, and then you're doing research, and then you're at Facebook doing research, and the scales are changing. It seems very much like, like you were scratching your own itch, but being, having the benefit of taking into account multiple perceptions of that problem, so that you ended up having a very rich understanding of what was needed and how it could satisfy multiple user groups. Do you, do you think that's a fair assessment? Or am I missing the boat? It seems like it was a really smart way of building a robust project from different

• 12:06Chris Benson?

  perspectives all rolled into one.

• 12:08Speaker 00

  Yeah, I mean, I think that's right. I think, like I said, you know, none of this was ever because I was trying to build anything for anyone else, right? I was trying to make myself move faster research.

• 12:17Chris Benson?

  Right.

• 12:18Speaker 00

  I think like, once other people started using it, they gave me that perspective there, right? And they put those constraints. And I mean, Lightning is not where it is today because of me, it's there because of the community, right? Like, there's no way I could have ever created this by myself, right? I think like, I could seed the idea and seed the templates, but a lot of my job has been to guide the community and maintain standards, maintain usability, right? So I care a lot about user experience, so, and I don't want to remember a lot of stuff. So there's just been a lot of guidance there as well. But at the end of the day, it's the community that's done a lot of this, right? But I think like, holistically...

• 12:52Speaker 00

  Having to focus on a lot of domains has made it super general, right? Because doing NLP is very different from vision and it's very different from reinforcement learning and meta learning and so on. And it's not obvious to know where they overlap. So it's been kind of a research project, really, in the long run, right? How do you factor out deep learning code and make it interruptible? Yeah, so that's been an interesting journey so far.

• 13:14Daniel Whitenack?

  You mentioned when you were introducing the motivation behind Lightning, the idea of decoupling models from hardware. And I noticed like even just on, if I look at the repository for Lightning, you talk about PyTorch, Lightning is just organized PyTorch and it's organized to sort of decouple science from engineering. And so you've got this model side and the hardware side. Could you dive into that a little bit more and talk about the specifics of what does it mean? If I'm using Lightning, what does it mean that my model is disentangled or decoupled from the hardware? Both practically in terms of how I write the code and like what happens like once I hit fit, like you're talking about?

• 13:56Speaker 00

  Yeah, so when we're working, I mean, look, I think if you're working at a company or any team, really even research, if you're working with multiple people, you need the ability to share code. And if you're at a company or even university lab, you want to share code across teams, right? And that's really hard to do without something like Lightning. Because what happens is people tend to intermingle a lot of stuff like data, model and hardware into the same files, right? Well, you know, one team may not have GPUs or may have different types of GPUs or may only be using CPUs or your production requirements mean that you can only use CPUs for inference, right? So there are a lot of constraints there. And I guess if you're not thinking about it how we are from the abstract level, you won't really realize that like a lot of the reasons

• 14:40Speaker 00

  why a lot of that code doesn't operate together is because you're mixing the hardware with the model code, right? And that's something that, you know, took us four years probably to get there, to see this, right, to have these insights. And what that means is that we can factor out deep learning code into three major areas, well, at least four, I guess, maybe. And we'll find more, right? I mean, it's ongoing research. So one is training code, right? So this is anything that has to To do with linking your model to the machine specifically. So how do you do the backward pass? You know, backward pass on the street is very different from just on CPUs, right? At least technically speaking. What happens if you have half precision there? What happens if you're stochastic good averaging? What happens if you have truncated back steps, right? So there are a lot of details that go into it. So all of that is handled by the trainer. And this is the stuff that you're gonna do over and over again, right?

• 15:25Speaker 00

  Doesn't matter if you're doing audio or speech or vision. You're always gonna have a backward pass. You're always gonna have a training loop and so on. The model is the thing that changes. The model is, it's not just, you know, I like to think about models. I guess Lightning, we have this concept of a Lightning module. And to me, a Lightning module is more of a system, right? So, you know, we can think about a model like, I don't know, like a convolutional network or a linear regression model, right? Just like a self-contained module. Today's models are actually not models. Like, we need a new name because there's something that doesn't exist. And I think the Lightning module, which is a system, right? Because models now interact with each other. Like, what do you call an encoder and a decoder working together to make an autoencoder? Or variational encoder, right? They're not models. It's collections of models interacting together. Same for transformers, right?

• 16:10Speaker 00

  So that's really what the Lightning module is about. You pass in these models into it and then how they interact together is abstracted by that, right? And I think that's a missing abstraction that was not there. And which is why people were jumping through so many hoops, right? To be like, oh, well, how do you do GANs? How do you do this other stuff? So it's important to decouple that because now I have this single file that's completely self-contained, that I can now share with my team across in a different division. And their problem might be completely different with a different data set. And they don't have to ever change the code on that model. All they have to do is change what hardware they're using and then what the data set is. As long as it conforms to the API that the model's expecting, it works. So it makes code extremely interoperable, right? So I think people come to Lightning because they wanna, you know, train on multiple GPUs and so on.

• 16:53Speaker 00

  And under the hood, we have this API called Accelerators that lets you do that. But that's only like a very small part of it. I think once you get into it, you see that the rest of it is the ability to collaborate with peers and be able to have reproducible and scalable code.

• 17:23Speaker 01

  This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights. A managed data platform that's built on leading open source tech leveraged by tens of thousands of users. Capture and process high-quality behavioral data from all your platforms and your products, and deliver that data to your cloud destination of choice. When marketing needs to make data-informed decisions, when product needs next-level understanding, and when analytics needs rich and accurate data, Snowplow is the solution for data teams who want to manage the collection, processing, and warehousing of data across all their platforms and products.

• 18:05Speaker 01

  Get started and experience Snowplow data for yourself at snowplowanalytics.com. Again, snowplowanalytics.com.

• 18:25Daniel Whitenack?

  Thank you for the great introduction to what Lightning is and how to think about some of the abstractions that you're working with. I'm wondering if you could maybe share a little bit. I've seen some different stories online, but I was wondering, from your experience with the community that's working with this, could you provide any sort of stories around how people have been able to scale things up with Lightning? Maybe in your own work, or maybe, you know, stories that you like to highlight?

• 18:55Speaker 00

  I mean, there are a lot of companies and labs using Lightning today, right? So you can get on GitHub and see that for yourself. I think, I don't know the exact numbers, but it's definitely like, you know, the thousands, like a few thousands of them, right? And they go from pharma to retail to anything you can think of, right? And I think... Today, what's interesting is that, you know, when I run into these people because they're, you know, we're coming to work with them on Grid, on some of them, it's interesting to hear the use cases, right? Like stuff that I would never imagine, right? Because I'm not at a company doing this kind of stuff. So that's why I made a joke about Lockheed Martin, but I'm sure you guys are doing much more advanced stuff. Unless I'm building, you know, planes, there's no way that I'd know to do that, right?

• 19:39Speaker 00

  So what's cool is just like, it's been super flexible. I think there are public cases that we can talk about. I mean, there are blog posts by big companies like Nvidia, Facebook and so on, about how they use Lightning, right? So you can read that. And I think something that we do specifically in the community is like, we really like to kind of protect our partners because like, this is a community and we want to keep people's work fairly private as well. So I won't get into too many details. So I'm just pointing you to open sources that you can look at and how they use it, right? But these are big projects as well. And there are probably about 3,000 projects now that use Lightning, that you can literally just go to see them. So the companies that have open sourced their work, you can see what projects are working on. So it's everything from like video prediction to segmentation to NLP, right?

• 20:23Speaker 00

  To summarization, to classification. We integrate really well with basically most frameworks out there. So if you use anything that's PyTorch based, it's very likely going to work with Lightning off the bat. Now, in terms of scaling, I mean, I personally, we've done it internally, right? But we've also heard from the corporate partners that they're training things on, yeah, I mean, I guess the number, I don't know, there's no real limits so far. I guess it's whatever PyTorch supports.

• 20:49Daniel Whitenack?

  However many GPUs you can get your hands on.

• 20:51Speaker 00

  Yeah, and like, you know, that's a big part of Grid now, right? It's like with Grid and Lightning, you can literally type in, I don't know, 1,000 GPUs. And if you have the Amazon quota, like, great, you know? And we can give you as many as we can as well, but there's no limitation, right? So you just have to run it. And like, I know it sounds crazy, but you literally just have to run it and then it'll just work, right? So it's just a function of the compute there. I mean, a few weeks ago, no, it's like a month ago at this point, we did a collaboration with Microsoft. So Microsoft has this library called DeepSpeed, which is really cool. Facebook has one also with the FairScale team. And basically it lets you scale up models dramatically by helping you use CPU memory efficiently. And, you know, the way you shard gradients and the way you shard parameters across GPUs really helps.

• 21:37Speaker 00

  So we were able to train a GPT model. I think it was like- I remember it was like twenty billion parameters or something like that. So we have a case study for that. So just for context, like the original GPT three was, I don't remember, it was like, hold on, let me see here. One hundred and sixty billion parameters or something like that. So I don't want to misquote you numbers, but basically whatever the original GPT three was, I think it was like one third of that. Well, only eight GPUs. So that's crazy. Just, I don't think anyone in industry needs that much. I haven't seen people use that much, so I'm just saying like that that's a pretty good lower bound.

• 22:16Daniel Whitenack?

  A hundred and seventy five billion, that's, or at least that's what Google is telling me on a search.

• 22:21Speaker 00

  Yeah.

• 22:22Daniel Whitenack?

  So you were very close and you said you were running that on what eight GPUs?

• 22:26Speaker 00

  Yeah, an A one hundreds, only eight of them.

• 22:27Daniel Whitenack?

  Oh, wow.

• 22:27Speaker 00

  And I mean, it's A one hundred, so they're much bigger than V one hundreds. But I mean, we'll be doing more tests. That was with these, just literally, and what's cool about it is if you're just using Lightning on your trainer, you just say, I think it's like plugin equals deep speed, like a string called deep speed. Just by doing that, you get that out of the box, right? So that's the kind of stuff that we embed into training. So, you know, do you have to know how to do that? You don't, but now you get that benefit.

• 22:52Chris Benson?

  So I wanted to real quick pop in one thing before we kind of start moving on on this. There are some people that are listening that may not, they may even be not PyTorch users. They might be TensorFlow users, but they're thinking about switching. You know, we always get into conversations. How does a workflow look like when you're integrating PyTorch Lightning into your workflow? You're using the rest of the ecosystem. Could you at a high level, just for those who haven't used it and maybe not have the something directly that they're going, oh, yeah, I've done similar to that, I can just add Lightning into that? What that looks like, what that savings, why is it called Lightning for them? They're kind of going, oh, there's this thing that may really help me. Can you kind of just top off a little bit of a workflow and how I go from the beginning to getting something productively deployed and what that looks like for somebody who hasn't seen it before?

• 23:37Speaker 00

  Yeah, absolutely. Wait, so I found the blog post, so it was actually forty five billion parameters that we scaled it up on eight A one hundreds. And you can look it up, but it's called accessible multi-billion parameter model training with PyTorch Lightning and DeepSpeed.

• 23:50Daniel Whitenack?

  We'll link it in the show notes.

• 23:51Speaker 00

  Yeah, sounds good. Okay, so basically it's how do you adopt Lightning into workflow, right? So. I mean, obviously, if you're coming from not PyTorch, then you would just, you know, start with Lightning. There's a, there's a very simple read me there. Like, I would say, you know, copy paste that read me. There's an M this example on there and you just run it. You'll notice those people will say, well, but where are the advanced examples? And my point is that that is the advanced example. Like, all you have to do is change the data and it'll still work for ImageNet, right? So That's great. That's the beauty of it. There's no different example for that. I mean, we put it if you want, but at the end of the day, just change your data and set GPUs to 64 and you're good to go. So that's the easy part, right? So if you're coming outside of PyTorch, then you can do that.

• 24:33Speaker 00

  If you're coming from within PyTorch, then what two people tend to do is when they start a new project, they'll either start it on Lightning directly or they'll convert their existing projects into Lightning. So it is really a refactor on your PyTorch project, right? So you basically take your main loop code, which usually looks something like, you know, you initialize a model, you set a bunch of flags, you set some sort of arg parse arguments, and then you download some data or link it somehow. And then, you know, it's all boilerplate. And then there's like a loop, two loops in there, which is like, you know, four epochs and epochs. And then you have in there four batch in your data loader and then you start training. So literally, everything up to that four batch in your data loader is deleted, right? So it's gone. Yeah.

• 25:18Speaker 00

  So then the only thing that you need to track is what's in there, right? Which is like the, we call that the training step, which is the meat of what you want. I mean, think about when you're doing work, like that's what you spend your time on. So that goes into this function called the training step. And then the training step goes all the way from taking your batch into generating a loss that you return with that gradient attached, right? So some graph. So, um, you know, it could be a few lines. Usually it's only a few lines because that's, that's most of what you're doing. Now, the model that you left at the top, that one, you can keep it separate and just pass it into Lightning module and just use it. You know, soft dot model equals model. Or you can define that model within the Lightning module, right? So you can literally copy paste the layers and all that into the Lightning module if you want, because the Lightning module is an nn dot module at the end of the day.

• 26:01Speaker 00

  So that gets you basically most of it. Then you need to find your optimizer and bring it into a function called configure optimizers. And then you just return it there, right? So then, you know, you're going to link up the parameters through that as well. So that's three methods, right? That's your init, that's your training step, and then that's your configure optimizer. And then the rest of that is optional after that, right? So forward, we don't actually need it. We use the forward method for inference, right? So if you train a model and you, for example, an autoencoder, right? So an autoencoder has two sides, an encoder and a decoder. The encoder maps some input into some space and embedding. And then the decoder maps that embedding back into some space. So an autoencoder can be used in two ways. You can use it as a, you know, embedder, basically.

• 26:46Speaker 00

  So you can take an image and get an embedding for it and then, you know, do similarity search and so on. So if you're building like a visual engine or something, you would do that. Or you can use a decoder for sampling. You can give it a random vector and it'll give you an image, for example, or, you know, text or whatever you want. So depending on what your use case is, that's how you're going to implement the forward. Because the forward is what's going to be called in production, right? You're going to call the model with the input to it. So we actually allow the model to be Torch scripted and put into Onyx as well, O N X, I guess, for production use cases. It's literally a function called dot to Torch script, dot to Onyx, and then you're good to go. And it does all the things for you. And then, you know, you just have to get the inputs, transform it, pass it through, and then do the return. So it's very simple.

• 27:28Speaker 00

  Now, there's other stuff left, so that's literally it. So you just have to copy that stuff. And then anything else that's left is usually around data or maybe validation or testing. The validation, we have a validation step and a test step as well, where you can just copy, paste that code in there if you want a validation loop or test loop. For the data, you can leave it as this. You can just pass in the data loaders directly to Lightning. Or you can use something called the data module, which is a completely optional abstraction. But it's basically captures your training, validation, and test data loader into one class and couples the transforms as well. Because what usually happens in big companies is that, you know, I'm working on, I don't know, let's say I'm doing, I guess, maybe selling something, right?

• 28:13Speaker 00

  And so I'm selling clothing. And so I have the dataset of our inventory with images and so on. And then when I give it to you, you're going to be like, hey, how did you transform the images? Did you crop it? Did you random flip? What did you do, right? And so unless I give you that code, then it's going to be a little bit hard and we could mess it up. So the data module embeds all of that. So I just have to say, here's a data module for the clothing dataset. And you just run it and you know it's going to be consistent across the board, no matter how you run it. So that's an optional, I mean, highly encouraged abstraction, but it's optional. Yeah, that's basically it. So if you do it, I would just recommend like, Don't delete your project, just do the refactor first, put it into Lightning, run it once, right? You can run it on CPU. When you do it with Lightning, you're gonna be able to run it on your local machine with CPUs or GPUs.

• 28:57Speaker 00

  Take a batch of data from your data set or a single example and overfit both models, like your original code and this one with the same seed and everything. And make sure you get the same results. And then, once you get that, then you're good to go. You know, you didn't mess it up. At that point, you can go ahead and say GPUs equals, you know, 128, and then off you go.

• 29:15Daniel Whitenack?

  So it sounds like that if I'm a PyTorch developer and I'm already using that API, I'm creating the layers of my model, I don't have to like throw out the way that I, the way that I created that model. In some ways, I get to sort of delete a bunch of my code, having to do with like the, you know, hardware stuff and some of the other training-related things. And I can keep my model and sort of refactor it into this PyTorch module, the Lightning module, and then call the trainer. And essentially, then I now have less code, but my code is also more, more robust in that I can run that training on a whole variety of hardware and that sort of thing.

• 29:59Daniel Whitenack?

  Am I basically summarizing that correct? Or anything you would, you would change about that?

• 30:03Speaker 00

  And it's more readable, right? You can literally give it to your colleagues and then they know to go to training step to see what's happening. Otherwise, what do you do today? You're like, hey, here's this like seven lines on GitHub.

• 30:13Daniel Whitenack?

  Yeah, it's crazy.

• 30:14Speaker 00

  You know, you can actually, they're like, wait, where, where is it what you're doing? Because most of it, it's like boilerplate training stuff, right? Now you can be like, hey, here's exactly what I'm doing. They're like, oh, you're sampling the latent space before doing this thing. Oh, interesting, right? It's not mingled with all this other stuff, so it's very easy to read as well. You know, I joke, but it is kind of like cleaning, yeah, like cleaning your house, I guess. Like, imagine, I guess roasts, right? So maybe this is a good example. So a roast, you have to cut it from a bush and trim all the stuff. And then, you know, you get this like bulb at the end, which is what you care about. It feels like that. It's like, no one's adding these other leaves because they want to, it's because they have to, right? So when you refactor your code, it's the sense of like, okay, it's a lot cleaner now. Like, I just removed a lot of unnecessary stuff and also stuff that you're likely to mess up, right?

• 30:59Speaker 00

  Like, we test very, very thoroughly and we have thousands of people testing this stuff. So did we mess up the backward pass? Definitely not, right? Did you mess it up? Hopefully not.

• 31:25Speaker 02

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes, or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 32:09Daniel Whitenack?

  Okay, I want to kind of circle all the way back to where our conversation started, because I want to get back to that cool demo that I saw on Twitter about Grid AI. So maybe you could just give us a little bit of sense of what Grid AI is, kind of how it came about, how it's maybe connected to the Lightning community, if at all, and then we can get into some of the details about what it enables.

• 32:33Speaker 00

  So, I mean, as you saw from my story, like, I care a lot about reproducibility and speed of iteration. And something that I thought a lot about as we were doing research and, you know, building Lightning was, in a corporate setting, you would want to scale this stuff up on a lot of compute, and you have cloud resources and all these different things. So the requirements for training at scale in a company are very different, right? Than just like on a Google Colab or a Kaggle. It's just like a very different world, right? So, you know, it's funny because, you know, deployment also goes into that. People are like, "Oh, here you go. Here you deploy it on this thing." And it's like, "Well, yeah, but like most APIs, most real machine learning systems are not just an API, right?" So we know that, I mean, a lot of us build these models. We've all been at companies before as well at scale.

• 33:17Speaker 00

  So we know exactly the pain points there. So the thing that kept coming up is like, cool, Lightning is letting me do all this. But how do I, you know, I'm still having to do all of this cloud stuff. Like, why, you know, if I ask for thirty two GPUs on Lightning, yeah, Lightning will do the thing. But like, you need to give me the thirty two GPUs. And giving you the thirty two GPUs, that's a lot of work to do it consistently and at scale and cheaply so that you don't have to burn resources, right? So what people end up doing generally is they build these like ad hoc internal solutions. They're like, you know, kind of put together bash scripts or things that like they string together like a semblance of a platform and they're great. And like, yeah, you will get things running, but like you won't be able to just, you know, scale them down immediately.

• 34:00Speaker 00

  You won't be able to have really fast build times because they're highly optimized. You won't have real time logs. You won't have real time metrics. You won't have time integrations, right? So all of these bells and whistles, when these things happen internally, they usually get pushed away because they're not a company priority because it shouldn't be like, you know, you're building airplanes, you're not building machine learning platforms, right? So you're normally not going to put the effort into making all the things that we care about as, you know, researchers and data scientists and machine learning engineers in there. So it's just going to make your life a lot harder. So it's about how do we bring that whole experience and encompass that model development cycle in a scalable way for the needs of like companies and even big labs, right? Because like most serious AI labs, they're training things on very, very large scales as well. Because training is a big part of the picture.

• 34:45Speaker 00

  It's not just a deployment. I think the deployment is interesting, but it's a lot easier because we've been deploying websites and things forever, right? But we haven't been training for that long. It's kind of a newer thing. So that's really the focus of great is to just completely eliminate the pain point that was left from using Lightning by not even having to deal with it. You just type in thirty two GPUs and it just happens, right?

• 35:07Daniel Whitenack?

  So I'm wondering, there's still a lot of people, I think, and maybe I have a misconception about this, that they think like maybe training models on GPUs in the cloud is always going to be more expensive than training on a sort of like, you're going to buy a on-prem server and do it in-house. Based on sort of your experience with that and like the current sort of state of cloud providers and all of that, is that perception mostly driven by the fact that, you know, and I feel very seen by the comment about like, you have all these bash scripts strung together. That's like my life, maybe. But is it because like

• 35:47Daniel Whitenack?

  that way of doing things is a bit inefficient and you waste a lot of resources and that sort of thing? Or where do you think that perception's coming from? And do you think it's accurate, I guess, is my question?

• 36:01Speaker 00

  I think, yeah, I think you hit it right on the nail. Like, if your system is inefficient, then it's more efficient to have your own machines, right? Because like, running on grid means that we install your dependencies, everything you do, link up your data in a matter of minutes, if not seconds, right? People don't generally optimize their stuff in the backend to do that. So what they end up doing is they want to run in the local machines because they don't have to install their environments, don't have to do all this stuff again, right? It's just there and it's repeatable and things start immediately, so it's a lot cheaper. I'm not going to say that running on your local stuff is not generally cheaper if you're doing things 24/7, but you're limited by bursting capabilities, right? So you're never going to have, you know, I don't know how many GPUs AWS has, but it's got to be hundreds of thousands, right? So if you have to hit a deadline or do something

• 36:45Speaker 00

  really quick and even go through ideas fast, if you're buying your own GPUs, you're going to be limited by how many you have there, right? So it's going to be more like sequential model building as opposed to asynchronous building. So with grid, you can go spin up 200 GPUs, run for five minutes and shut them down, and you just got a lot done, right? Whereas on your own machines, even if you were to do it yourself on the cloud, you would probably not even get the models running for, you know, 20 minutes and 30 while you spin up the machine, set up the, all that stuff, right? So I can take $100 on grid and get more GPU minutes out of it than you would normally with, without optimal systems, right? So it's just very, very optimized. Now, I do think that people need to know

• 37:27Speaker 00

  about things, I mean, we do a lot to lower the cost, right? And I think one of those things is spot instances, right? So spot instances are machines that

• 37:36Speaker 00

  Can be killed at any time by AWS, right? Or whatever cloud provider you're using. And then at that point, you're kind of done, right? And so, but the nice thing about SPOT is that it will be like, I don't know, fifty to eighty percent the discount, right? So if a GPU costs three dollars an hour, it could be like thirty cents an hour to maybe a dollar an hour, right? So it really depends. So I think what you're saying is true because I did the calculus myself. And in fact, I have like blog posts on how to build your own GPUs for this reason. But that was only for ten, eighty TIs, right? And it cost me maybe six grand to build that machine, which is great. Now, six thousand dollars, if I'm paying full GPU prices, I'll burn through that in like two weeks for sure, right? But if I'm paying SPOT prices, then that's good.

• 38:21Speaker 00

  That changes the game. And then, not only that, but if I'm getting more training minutes out of that, that's a lot better. And then you factor in depreciation and those other stuff, plus maintenance, then it actually becomes a little bit competitive, right?

• 38:32Chris Benson?

  It does. I'm curious, you know, and we're talking a lot about the training. Could you talk a little bit about Grid AI's deployment story and what that is? And in my mind, one of the things, like speaking for myself, I'll be training centrally in the cloud and stuff. But at the end of the day, I gotta get my model or my system of models out there into something, often some sort of edge device, not cloud-based, you know, something that's a physical thing out in the real world. Can you talk about how you work with Grid AI to affect that?

• 39:00Speaker 00

  Yeah, so today, Grid doesn't support deployments, right? So the thing that we like to focus on is making sure that we really nail certain experiences before moving on to other things, right? So we will support deployment at some point, probably very soon, right? But the thing is like, I don't think that we're fully, fully optimal on the training side yet. Like, I think we want to provide a really world-class experience there. So for our users today, like, you know, you cannot access artifacts, you can get model checkpoints and all that stuff. So the deployment, most users have a deployment system it has already, so they can just take the artifacts and do their thing, right? So we're not blocking any of that. And all of these things are like URL-based. And if it's Lightning, like, I mean, that's very easy to do. Now, we're gonna make it a lot easier for sure, like, you know, kind of the way that we do things. But today, we are laser focused on training.

• 39:45Speaker 00

  But I will say, I think like, working with Grid at this stage is great because I think companies will be able to help us influence that roadmap, right? And help us build something that they really care about as well. Because as soon as we start getting to deployment, like, we're gonna do it our way, right? And we have a very special way of doing things. So we hope that we have the feedback from the community and users to make sure that we're doing it in like a really useful way.

• 40:07Daniel Whitenack?

  And how, as a user of Grid AI, because this is really fascinating to me because I've even been struggling to get some in-house GPUs just with supply chain issues and all of those things. So running things on the cloud is something that we're actively, you know, thinking a lot about and doing it in an optimized way. Now, we kind of talked before about going, say, from PyTorch to PyTorch Lightning. Let's say I've got my Python code, I'm using Lightning, it works great, and now I want to run it with Grid AI on, you know, 100 GPUs in the cloud. What does that look like? Do I need to set up my cloud account, set up billing on that side, and then set up my Grid account? And then use a Grid tool to connect them both? How does that whole flow work from that point?

• 40:54Speaker 00

  So I think, yeah, that's a good question. Okay, so generally, I like to think about what we're trying to do, like, you know, the leap between, I don't know, like, Windows machines to Mac machines, right? Like, where things just work, right? So what is that Apple experience for machine learning, right? And I think, to answer your question, it's very, very easy. It's not as easy as I want it to be today, but it will be. So basically, there are a few ways, right? So we have three tiers of usage on Grid, right? We have the community tier, which is free. Literally, you're just paying the AWS compute, right? There's nothing in there. Like, we're just orchestrating stuff for you. But it doesn't really work for teams and like, big companies, right? Because there's a lot of stuff that needs to happen. So then we have the teams and enterprise tiers that let you do those kind of things.

• 41:37Speaker 00

  On the community tier, you literally have to do nothing. You just copy paste the link to a GitHub file. You paste it into the UI or use the CLI. And you select how many GPUs you want and you press enter and you're done, right? Like, it's that easy. The dependencies are automatically pulled for you. They're inference from the code that you have, your requirements, all that stuff. So we try to do as much as possible. Yes, there will be times when that fails and we will work with you to figure out what happened and make sure that we get it done. But dependency management is a big deal for everyone and it's a really hard problem to solve. So it's going to take us a while to fully solve that problem. But if you are at a company or at a big lab, like usually, that's, we call that community tier. That's going to work great for like side projects and public data and stuff like that.

• 42:22Speaker 00

  Kaggle, prototyping things, sure. If your data is not secret, then it's fine. So great for academics as well. But if you have corporate data, then you're going to be on the teams or enterprise tier. Right there, what you end up doing is we basically link up your cloud accounts, right? So you just set it up through Grid. You're passing credentials through there. And then those keys let us control resources on your behalf, only as much as you allow us to, right? To make sure that we orchestrate everything on your cloud. So it's kind of this hybrid on-prem versus not on-prem. We also offer on-prem if people want it. So once you do that, you basically put in your cloud credentials in there, then you're good to go. When you run stuff on Grid, instead of running on the Grid cloud, which is a community cloud, you just select your cloud, whatever you named it, and then you just run on it, right?

• 43:05Speaker 00

  And that means you can link up as many of these as you want as well.

• 43:08Chris Benson?

  As we kind of wind up here, one of the things that's really struck me through the conversation is that you are a man of substantial vision. And as we kind of wind up, I'm really curious if you would kind of look out a little bit beyond just the next product cycle and that kind of thing into where you want to go, both with Grid AI and where you see the larger industry going in general in terms of trying to make this work a little bit better for people and take the struggle out of it that you clearly have been working on for a while in various capacities. Could you tell us a little bit about what future you think we're going toward and what you would like to shape it and how you would like to shape it?

• 43:46Speaker 00

  You know, when I started in research, I was really disappointed that I had to do so much work over and over again that other people were doing and that I had to learn so much just to decode a little bit of neural activity, right? And the world that I would love to kind of help bring to the table is a world where the person, the scientist, the researcher, the machine learning engineer, the person that has the knowledge of whatever they're building, right, the doctor, the biologist, the mechanical engineer, you know, you name it, the person who really knows their domain, can basically focus on that and not like have machine learning and all of this cloud stuff just kind of fade into the background, right, and just be like Wi-Fi, just like

• 44:29Speaker 00

  your cellphone signal, like you don't think about it, right, you're just working on your problem. So how do we take that leap? And I think that's what we're trying to solve. Are we there yet? No, but we're definitely on track. And, you know, I think that working with a lot of amazing companies and getting to make sure that we support their use cases is what's going to help us get there. So the person who builds the models, who has the ideas, that doctor, they can be the ones to actually, you know, train and deploy this stuff. Because, you know, at the end of the day, I think that deployment is literally just another training cycle, except the data is live and you're not back propagating into your model, right?

• 45:02Daniel Whitenack?

  That's awesome. Thank you so much for talking to us about Grid AI and Lightning. It's been really wonderful. And like I say, we'll put show notes and everything, the relevant links that we've talked about in terms of Lightning and Grid AI, definitely check it out. And yeah, thank you so much for joining us, William. It's been a pleasure.

• 45:19Speaker 00

  Well, thank you, guys. This was really fun conversation. Thank you.

• 45:25Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it, and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whiteknack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it, and we'll talk to you again next week.