Building a deep learning workstation

What’s it like to try and build your own deep learning workstation? Is it worth it in terms of money, effort, and maintenance? Then once built, what’s the best way to utilize it? Chris and Daniel dig into questions today as they talk about Daniel’s recent workstation build. He built a workstation for his NLP and Speech work with two GPUs, and it has been serving him well (minus a few things he would change if he did it again). Sponsors: Linode https://linode.com/changelog – Get $100 in free credit to get started on Linode – our cloud of choice and the home of Changelog.com. Head to linode.com/changelog https://linode.com/changelogChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Daniel’s workstation components: CPU - AMD YD292XA8AFWOF Ryzen Threadripper 2920XCPU cooler - Noctua NH-U12S TR4-SP3, Premium-Grade CPU Cooler for AMD sTRX4/TR4/SP3Motherboard - GIGABYTE X399 AORUS PROMemory - Corsair Vengeance LPX 16GB (2x 2 packs), total 64GBStorage 1 - Samsung (MZ-V7S1T0B/AM) 970 EVO Plus SSD 1TBGPU 1 - RTX 2080 TiGPU 2 - Titan RTXCase - Lian Li PC-O11AIRPower Supply - Rosewill HerculesCase fan(s) - Coolmaster 8mmDaniel’s NUC 9 Extreme machine https://simplynuc.com/9i7QNX-full/ References: How to build the perfect Deep Learning Computer and save thousands of dollars https://medium.com/the-mission/how-to-build-the-perfect-deep-learning-computer-and-save-thousands-of-dollars-9ec3b2eb4ce2Curtis Northcut’s blog posts https://l7.curtisnorthcutt.com/tag/build-gpu-rig/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Daniel Whitenack?

  I use the air-cooled CPU cooler, not like, I didn't feel quite comfortable enough yet to run like all sorts of liquid through my machine. I figured that I, you know, knowing myself, I would break a tube and like ruin everything in my computer. And actually, I looked, there was some study I found, I actually probably couldn't find it again. And maybe it was just confirmation bias, but I found like, I think it was some YouTube video showing how the air coolers, the good air coolers, will actually cool better than a lot of the cheaper water coolers or radiators. So that gave me some confidence there.

• 0:39Speaker 01

  Bane with for change log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by Launch Darkly. Check them out at launchdarkly.com. And we're hosted on Lino cloud servers. Get a hundred dollars in hosting credit at linode.com slash change log. What up friends? You might not be aware, but we've been partnered with Lino since 2016. That's a long time ago. Way back when we first launched our open source platform that you now see at changelog.com. Lino was there to help us and we are so grateful. Fast forward several years now and Lino is still in our corner, behind the scenes, helping us to ensure we're running on the very best cloud infrastructure out there. We trust Lino. They keep it fast and they keep it simple.

• 1:24Speaker 01

  Get a hundred dollars in free credit at linode.com slash change log. Again, a hundred dollars in free credit at linode.com slash change log.

• 1:50Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at changelog.com slash community and follow us on Twitter. We're at Practical AI FM.

• 2:16Chris Benson?

  Welcome to another fully connected episode of Practical AI where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I am a principal emerging technology strategist with Lockheed Martin and with me as always is my co-host Daniel Whitenack, who is a data scientist at SIL International. How's it going today, Daniel?

• 2:42Daniel Whitenack?

  It's going great. It's a little bit interesting. You know, Chris, we haven't done one of these fully connected episodes in a while where it's just you and I. So for those listeners that have been with us for a while, you'll remember we, Chris and I used to somewhat frequently, maybe every few episodes, have an episode with just Chris and I and no guest and discuss some news things or learning resources or random topics, but we haven't really done that in a while. So it seems so nostalgic.

• 3:11Chris Benson?

  So it is, but this is a good one. This is one we have been talking about doing for quite a long time. Because you were working on on developing your own AI workstation and you built it from scratch and we heard a bit of progress across several episodes there.

• 3:27Daniel Whitenack?

  Yeah, in in the intros to a few episodes, I mentioned that I was building a an AI workstation and actually ended up, so I have a few computers sitting around me that one of those is that workstation I was mentioning. Another one is, we can talk about that one as well, actually. But yeah, it's been a bit of an interesting journey. And I think, you know, at least news wise, maybe some people are thinking about this these days, but Nvidia's latest release with their 30 series GPUs, actually even today, I was talking to someone who was trying to get their hands on one and there's like none anywhere. So this is, what is this, November 10th as

• 4:12Daniel Whitenack?

  we're recording this of 2020? And I forget when they released those. I think it was in maybe September, but they've been pretty scarce. And I think what happened is the, so there were the lower tier ones, I think primarily geared towards gamers and those basically sold out instantly.

• 4:31Daniel Whitenack?

  And so then the ones that like, I think the thirty ninety, the the higher tier ones that were geared towards like AI training and stuff. The gamers then just started going to those and basically everything sold out for anyone. And this morning, I was on a call with a guy and he was like, we were taking the call. While he had Best Buy pulled up and like he was getting notifications when they would. Like release some on Best Buy, but he never got one. So like he, you know, went into the thing and. Uh, push the add to cart and then they make you wait. And then by the time you're like, they're waiting in the queue is done, then. You know, your cart opens and it's sold out and no longer available.

• 5:14Chris Benson?

  Yeah.

• 5:14Daniel Whitenack?

  It's kind of a vicious cycle like that.

• 5:16Chris Benson?

  So you went virtually shopping with him on his.

• 5:19Daniel Whitenack?

  Yeah, yeah. He screen shared and everything. So we got the experience together. Um, and that was.

• 5:25Chris Benson?

  You don't just have your own frustrations. You get to share in his frustrations too.

• 5:29Daniel Whitenack?

  Exactly, exactly.

• 5:30Chris Benson?

  Very good.

• 5:30Daniel Whitenack?

  So yeah, for um, ma- maybe listeners that are newer to to AI, which we'll, we'll go into this a bit with the workstations, but um. A lot of people use um, GPUs and. Uh, specifically Nvidia GPUs to accelerate their workloads, in particular, training is a. Big thing to accelerate inference as well, but um. Training mainly. So when people are building their computers for uh, training, deep learning models or AI models, a lot of times they're in the market for a GPU and. Such are not available at this time as we've been discussing. So that makes things a challenge. Actually, uh. I mentioned I have the one workstation.

• 6:13Daniel Whitenack?

  Um, so I guess I can kind of talk now about the the things that got set up.

• 6:17Chris Benson?

  Go for it. Yeah.

• 6:18Daniel Whitenack?

  So I have the one workstation that I built that has two GPUs in it. And I've been using that for training models. Um, and then I've got a separate one, which is. It doesn't have a GPU in it, but I've been using it. We've been training models for Intel-based edge devices. And so I've got this Intel machine, which actually is fairly beefy, but to test those models on an Intel-based architecture without a GPU for inference, but also it's got a bunch of RAM for like pre-processing and things like that. And I can kind of talk about how I've experienced both

• 6:59Daniel Whitenack?

  of those. But those are the two things that I've been using recently. And so I really haven't done much other than connect from my laptop to those machines while I've been working recently. And that's some interesting characteristics, I guess is the right word. I don't know if I would call them features, but interesting workflow implications as well.

• 7:22Chris Benson?

  Well, you're diving into that now. So you gotta, you gotta tell us a little bit about what.

• 7:25Daniel Whitenack?

  Yeah, I mean, this is not really AI or computer. Well, I guess it's, I guess it's computer related, but so I SSH into either one of these machines or actually on the workstation. Sometimes I SSH in and just run jobs and sometimes I'll connect to a Jupiter server. The other machine, I can SSH in or actually even just connect a monitor and work on it directly. Yeah, I, we have Comcast Xfinity Internet here. And, you know, just like the home internet network, I have the Xfinity router. So just the one that comes from the ISP.

• 8:05Daniel Whitenack?

  And I guess they assume that most people aren't going to be wanting to like connect in that way.

• 8:12Chris Benson?

  Yeah.

• 8:13Daniel Whitenack?

  Or, or like when I am at my wife's business, she owns a manufacturing business and when I'm there and I want to connect back home to my AI machine, then I have to open up a port in our home network, right? And then do this port forwarding thing. And all of that's kind of. I would say difficult in the Xfinity firmware that's on that router. So our listeners may also remember that I installed a new network at my wife's business, which has some really great network appliances and ubiquity gear. And so I'm thinking actually of taking both of these machines and maybe just like putting them in the IT closet at her work.

• 8:56Chris Benson?

  Yeah.

• 8:56Daniel Whitenack?

  And then whenever I need to connect just SSH into that network and find the computers. I think that's actually the better solution. That's like the workflow of things that I mentioned in terms of connecting to them.

• 9:08Chris Benson?

  Gotcha. And for what it's worth, for those out there who may not be intimately familiar with that, the two second explanation on SSH is a secure connection to a remote computer. And you talked about going through your router and having to open ports. And those are like doorways you might think of on your router where you have to, you have to open them, you have to have access to that and they have to be able to connect to each other. So there's a lot of plumbing. It's, it's internet connection plumbing to make work.

• 9:36Daniel Whitenack?

  Yeah. And when you spin up a Jupiter server, let's say locally, or you type like Jupiter space notebook on your, on your terminal, and that spins up a local server. And you might notice if, if a tab pulls up in your browser normally it'll say like local host 8888.

• 9:52Chris Benson?

  Yep.

• 9:53Daniel Whitenack?

  That's the host and the port. So I'm just connecting to that host running the Jupiter server, but it's a remote computer, not the local host on the same machine.

• 10:03Chris Benson?

  Gotcha. And I have a question before we move off of this. I know this isn't the primary topic, but do you ever have a reason to get on and do the actual work on the workstation itself locally? Or do you always come in from your laptop through a port and use the workstation as a remote server in that way?

• 10:19Daniel Whitenack?

  Yeah, I haven't really done that much. I don't really have much installed on the workstation other than Ubuntu is the OS, Linux OS. And then I have Docker installed and then the like CUDA libraries and such for the GPU.

• 10:40Daniel Whitenack?

  And so there's really not much, it's not meant to be a, you know,

• 10:44Chris Benson?

  A primary work.

• 10:45Daniel Whitenack?

  Primary workstation with all sorts of things installed on it necessarily. So yeah, I haven't really connected to it in that way. Outside of when I very first set it up and booted it up before all of that, I, I connected up a monitor and keyboard and mouse and that sort of thing.

• 11:03Chris Benson?

  Gotcha. So you're kind of using it as a remote SSH server to do the work and you're still on your laptop.

• 11:08Daniel Whitenack?

  Yep.

• 11:09Chris Benson?

  Yeah. When you're doing that, but it's providing those resources.

• 11:12Daniel Whitenack?

  Yep, exactly. And I mean, there's no reason why you couldn't use it in that way. But also if you're doing like, if you're using this workstation primarily for those types of workloads, like training workloads and that sort of thing. If you're pulling up a monitor and you have like 17 Chrome tabs up and like it's eating away at the memory and then like you're also have, you know, the GPUs then potentially supporting various graphics operations, not that it's consuming a lot in terms of the same amount that training consumes, but it's just not optimized. So yeah, there's really not anything running on there other than, than those few things that I mentioned.

• 11:54Chris Benson?

  Gotcha. So I guess what was your motivation for doing it this way? Cause we have talked and anyone who's listened to a number of our episodes, prior to you doing this, we kind of for a while, we're talking about, you know, really focusing on you. We have all three major cloud providers out there with all these services. I remember us having conversations going, ah, why do it elsewhere and stuff? Clearly you had a change of heart at somewhere along the way. So what was it that motivated you to do this?

• 12:23Daniel Whitenack?

  That's a great question. I think there were a few things. It wasn't any one particular thing. The first, which is not really practical in any way is just because my brother-in-law was building a new gaming computer. And I was kind of watching him do that and we were talking about it and I kind of realized that. You know, back in college, I guess it would have been, would have been the last time I built a personal computer of, of any kind.

• 12:52Chris Benson?

  Right.

• 12:52Daniel Whitenack?

  And I used to do that, you know, fairly frequently, especially since I was in, um, I worked as a IT person at my college. And so I, I would do that fairly frequently or dig into the, you know, guts of computers fairly frequently and switch out parts and whatever.

• 13:10Chris Benson?

  I did too in college, but that was a different time.

• 13:12Daniel Whitenack?

  Yeah.

• 13:15Daniel Whitenack?

  Yep. I won't ask about that.

• 13:17Chris Benson?

  Yeah. You don't want to, you don't want to go back to the age of dinosaurs. You know, no, no, definitely not.

• 13:21Daniel Whitenack?

  I really enjoyed doing that back then and I realized I had, I just hadn't done it forever. And so part of it was like all of these new parts and how things have updated. Everything's different now. Um, there's a different spin on all sorts of things, you know, hard drives are different. They don't have like these spinning bits in them, which they always did.

• 13:41Chris Benson?

  Yeah. Platters.

• 13:42Daniel Whitenack?

  Yeah. I remember, uh, in high school, college, we would take old hard drives and take the tops off. And then you had like action figures and we duct taped the action figures to the drives. And then like put little, uh, swords or something in their hands. And then, you know, spin them both up at the same time and the last one standing would, uh, you know, win the prize or whatever.

• 14:06Chris Benson?

  So kind of a centrifugal force contest, right?

• 14:08Daniel Whitenack?

  Yeah, exactly.

• 14:09Chris Benson?

  Who survived the longest.

• 14:10Daniel Whitenack?

  Who can like, yeah, yeah.

• 14:12Chris Benson?

  Got it. Very good. At least you found a use. You are the only person on the planet who has found a use for these old hard drives rather than just checking them.

• 14:20Daniel Whitenack?

  I will say that the network video recorders and a lot of file servers now, I think, still use those types of drives. They'll have

• 14:29Chris Benson?

  Do they?

• 14:29Daniel Whitenack?

  12 slots for, um, regular, you know, spinning hard drives, you know, when you have like that much data. It, it doesn't need to be accessed quickly. It's just cheap storage.

• 14:40Chris Benson?

  Gotcha.

• 14:41Daniel Whitenack?

  But, um, I totally forget where, where we, oh, uh, the motivation for. Got into action figures on hard drives.

• 14:48Chris Benson?

  It's fine, you know.

• 14:49Daniel Whitenack?

  Yeah.

• 14:50Chris Benson?

  Come back from the action figures. Why build?

• 14:53Daniel Whitenack?

  Yeah, so the second reason was a lot of the stuff I'd been doing up until the summer was natural language processing text based, though. So text, corpora, or data sets are generally fairly small in terms of size.

• 15:11Chris Benson?

  Compared to audio or, or video for other non-NLP.

• 15:13Daniel Whitenack?

  Yeah, compared to audio or video.

• 15:15Chris Benson?

  Yeah.

• 15:16Daniel Whitenack?

  But I've been doing a lot of speech projects this year. And so also having a system where I could run something, you know, for four or five days and do that over and over and over and over without, you know, incurring the cloud costs.

• 15:31Chris Benson?

  Yeah, so there's a crossover.

• 15:33Daniel Whitenack?

  There's a break even point. Yeah, there's a break even point. So even, I think, when I did the research, there are some really great affordable, I mean, depending on your definition of affordable, but there are some cheaper manageable solutions for using GPUs in the cloud. One of those was Paperspace, which I was using before for longer training runs. That's a really great system that integrates with a lot of things now. I think even like hugging face transformers, I saw an article about an integration there. And that's easy to use. I like the design pattern there.

• 16:05Chris Benson?

  Yeah.

• 16:06Daniel Whitenack?

  And they have integrated notebooks and all of that. There's also Google Colab and things like that where you can even get free GPU usage.

• 16:14Chris Benson?

  Yeah.

• 16:14Daniel Whitenack?

  Our organization uses G Suite. And so we have access to team drives and all of that. So I would occasionally save, you know, Colab notebooks and other things and work on them with people in that setting. Yeah, so there is a break even point. And I think I kind of calculated out a bit of what we were going to need to run. And we were sort of right at that break even point. And I knew, you know, we were kind of pushing our limits in terms of the, the costs, so.

• 16:43Chris Benson?

  So you actually did an analysis, though, at least a back of the napkin kind.

• 16:48Daniel Whitenack?

  Right.

• 16:48Chris Benson?

  To figure out what your break even would be.

• 16:50Daniel Whitenack?

  So like, it was something like, you know, over a summer, you know, we were going to spend at least. You know, eight to $12,000 on GPUs in the cloud for training. That would be kind of the lower range. And, you know, when I started looking at blog posts about now, some deep learning workstations that you'll get from that are like pre-built, we can talk about that too, are not going to be a cost savings with respect to that. But you can also, if you sort of roll your own and build your own, you can get into a lower cost zone with that. And that's one of the things that we can talk about moving forward.

• 17:31Chris Benson?

  Gotcha. So in the end, it was just a simple, but was that the whole enchilada, so to speak? Was that was basically you figured out that your crossover, your break even point made it make sense after that? And then as soon as you got to that point, was it anything else that contributed?

• 17:46Daniel Whitenack?

  I think those were the two main things. I also knew that there were a couple other people coming on to our team that would need this sort of resource. So, you know, when I'm, you know, not running things, it could also benefit others on the team that needed GPU resources and, you know, we're either going to pay for those or get their own machine.

• 18:11Speaker 03

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes or point your favorite web browser to changelog.com/plus. Once again, that's changelog.com/plus.

• 19:11Chris Benson?

  Okay, so now that we've kind of talked about why you chose to do it, let's actually get into the machine itself a bit.

• 19:17Daniel Whitenack?

  Sure.

• 19:17Chris Benson?

  And let's talk about what you built. So, you know, in any way you want, just take us through a rundown of what it is that you've done and maybe some of the choices along the way.

• 19:27Daniel Whitenack?

  Yeah, sure. So I'm actually going to, while we're talking, I'm going to pull up a couple of links where I started my research. So originally, when I was looking at this, I was like, you know, do I really want to sort of roll my own workstation and get all the parts and do all that? Or just get a really cool workstation that is still going to be really cool and sit by my desk, but you know, isn't going to require sort of all of that work. So I started looking at places like Lambda Labs or System 76. They have these sort of pre-built deep learning workstations with, you know, four, eight GPUs, different configurations.

• 20:09Daniel Whitenack?

  And I really liked the look of those and what you could do with them. But when I started calculating out the cost, it was going to be, you know, pushing $10,000 to $12,000, which rightfully so, they need to make, you know, profit on those. And there is a significant amount of work in terms of, you know, fabrication of what they're putting together and like the design in that and making sure it's maintainable and all that. But I was looking at definitely a lower price tag than that in terms of what I could accommodate. So I found a couple of blog posts that appealed to me right out of the bat.

• 20:50Daniel Whitenack?

  The first one of these was from a guy named Jeff Chen, who is out in California. I believe at the time he wrote the blog post, he was at university during research. This is from 2019 and the title is "How to Build the Perfect Deep Learning Computer and Save Thousands of Dollars," which is obviously very appealing to me. The next set of blog posts was from a guy named Curtis Northcutt. Actually, he has a few different builds, but... One of them, he talks about the best four GPU deep learning rig only costs $7,000, not $11,000. So if you remember, I was kind of looking at those pre-built ones around $11,000.

• 21:33Daniel Whitenack?

  I didn't think I needed four GPUs in terms of the initial workloads I was going to run. So I thought maybe I could also bring it a little bit under that, so under that price point. So my thought was I could follow something like these blog posts talked about and get a workstation up and running for much less than what we would be spending over the summer in cloud costs if we were to run our workloads there. But also I wanted to get it expandable for the future, I guess. So start out with maybe two GPUs, but also have it expandable to maybe four GPUs. Because I knew, you know, our workloads might expand or more team members might come on the team eventually

• 22:18Daniel Whitenack?

  and that sort of thing as well.

• 22:19Chris Benson?

  So a quick question there. You, you mentioned, you know, kind of that you knew that you really only needed two right now. You wouldn't need the four. As an aside real quick, how did you make that evaluation?

• 22:29Daniel Whitenack?

  Yeah, not in a high-tech way.

• 22:31Chris Benson?

  Okay.

• 22:31Daniel Whitenack?

  And so my logic on that front was as follows, which is maybe not the best logic to follow, but.

• 22:41Chris Benson?

  Oh, no, I can hear him caveating all the way through this. You just finger in the air kind of.

• 22:45Daniel Whitenack?

  Yeah, yeah, and I should, uh, maybe I'll just make a general disclaimer that I'm not a hardware person. So a lot of the choices I made, I'm sure I'll have people come into our Slack channel and be like, why did you do that? Why did, I'm sure, and I already know a couple of things that I would change if I could, which we can go into those later.

• 23:03Chris Benson?

  Yeah, we'll get into that in a bit.

• 23:04Daniel Whitenack?

  So this was a learning process for me. I'll just disclaimer that. But my thought process in terms of the two GPUs was that I didn't yet want to worry about multi GPU trainings consistently. I've found in the past that. Generally, the time I sink into getting multi GPU accelerated training working, I mean, that's it's more complicated and the time I put into that can be saved by just sort of loading that workload into a single higher tier GPU. So a lot of times I find that it's most easy to just do single GPU training, but on a higher tier GPU than do like multi GPU trainings all with

• 23:49Daniel Whitenack?

  lower tier GPUs, if that makes sense.

• 23:52Chris Benson?

  Got it. So, yeah, so let me ask this. Do you so? Is it fair then to say that your training runs are, since you have two GPUs, you're doing two concurrent training runs versus doing a single one where you're using both of the GPUs there?

• 24:06Daniel Whitenack?

  So this is the second logic with this is that I knew that primarily on these speech projects, I was working with a single other main developer on this work. And so my thought process with that was while I'm running something on GPU one, he can be running something on GPU two.

• 24:25Chris Benson?

  Got it.

• 24:26Daniel Whitenack?

  So that's where the two came from was the fact that I assumed we would be doing mostly single GPU runs and that there were two of us.

• 24:37Chris Benson?

  Okay. No, fair enough.

• 24:39Daniel Whitenack?

  Yeah. And, you know, at some point, it would be interesting for me. I would like to try some multi GPU stuff. When I looked into this, there's a whole like rabbit hole you can go down on this front because like certain cards. So I actually have two different GPU cards in the machine. That one is different from the other one. There's a 2080 Ti and a Titan RTX. And those you actually, so there's a special link between GPU cards called an NVLink, which is, I think, in my understanding, the best way to communicate data between the two cards.

• 25:18Chris Benson?

  Yeah.

• 25:18Daniel Whitenack?

  The fastest way. Yeah.

• 25:21Chris Benson?

  And that's an Nvidia technology between the two cards for networking.

• 25:25Daniel Whitenack?

  That's my understanding.

• 25:25Chris Benson?

  Just for those who aren't familiar with it, that it's proprietary technology from Nvidia that gets you that great throughput for training purposes.

• 25:33Daniel Whitenack?

  So eventually I'd like to try something with that. I think my understanding is that if you don't use that link, the sort of standard link through the motherboard, you don't always get the sort of boost that you might expect because of the communication bottlenecks between the cards.

• 25:49Chris Benson?

  Yeah, and that's my understanding as well.

• 25:52Daniel Whitenack?

  Yep. So eventually I might like to try that, but it may depend also on future workloads that I'm running and what's needed there. So.

• 26:01Chris Benson?

  Gotcha.

• 26:01Daniel Whitenack?

  Yep. So that was the logic behind that, which meant that I needed a motherboard in the workstation that could support two GPUs, but I also wanted it to be expandable.

• 26:15Chris Benson?

  Expandable. I remember you mentioned that.

• 26:16Daniel Whitenack?

  Yeah. So the blog post by Jeff Chen, who Jeff, if you're listening to this, thank you for your blog post. Reach out if you are listening. We'll grab you a shirt. But it talks about all sorts of things and probably some of it is, so I noticed some of the things, like some of the parts had even been updated since he wrote the post, but most of it was still relevant when I went through. And he also talks about trade offs and he goes down and he talks about expandable. So making sure that you build things expandable and that sort of thing. So that's where I kind of got that idea. What I ended up doing was

• 26:57Daniel Whitenack?

  getting a motherboard, which actually, this reminds me, so in our show notes, I think one of our listeners asked for this at some point in our Slack channel too. I'll post the part list that I used in our show notes if anyone's curious about the parts that I use. So don't feel like you have to frantically write down part numbers or something and I would probably mispronounce the names anyway, but I'll put the parts list in our show notes. But it was from Gigabyte motherboard.

• 27:27Daniel Whitenack?

  And I don't know how to say this. This is gonna be where my ignorance comes in. Aorus? A- Aorus? A O R U S?

• 27:36Chris Benson?

  Good enough. I have no idea.

• 27:39Daniel Whitenack?

  And so that that supported actually four. So I could I could run four, although it seems like a tight fit. I don't know if it's just like me and my inexperience and not knowing that like things are that tightly fitting. I could definitely fit four, I think. But right now I have two spaced out. So there's a slot in between the two that I have in there. There's actually a slot on the bottom and then I have one in the next slot up. And then there's this empty slot and then the final one. So they're actually spaced out a bit. If I were to put all four in there now seeing that they would be pretty tightly packed. So depending on ventilation stuff, that may be an issue.

• 28:21Chris Benson?

  That's what I was gonna ask, you know, coming at it from someone who also does not do hardware and has not built a system in quite a while. But, you know, having previously put Nvidia cards in that had big fans and things like that. Is that still, when you talk about slotting it in, is that including those ventilation capabilities? Or is that without that?

• 28:42Daniel Whitenack?

  Yeah, so as I learned, there's a bunch of options here as well. And there's two main types of cards. There's actually a blower style card, which pulls in air from one side and actually shoots it through the card and out the back of the computer. And then there's another style, which people might see in pictures where there's like two or more fans on the bottom and it just sort of blows down. Like it blows from top to down and draws air away from the card. So this is actually really interesting. On Curtis Northcutt's blog, he talks about double deep learning speed by changing the position of your GPUs.

• 29:22Daniel Whitenack?

  And when he's talking about position, he's actually talking, he has a mix of blower style and non blower style GPUs. And so what happens is when you put one of those non blower style GPUs on top of another one, it actually just blows the heat into the other one and heats it up and it degradates performance. So the positioning is actually really important here in terms of managing the the heat on those. So you kind of wanna shoot for those blower style, at least depending on.

• 29:53Daniel Whitenack?

  How many you're gonna have and how you position them? You're gonna wanna worry about that positioning and the flow of air, obviously. Uh, actually, you know, this is one of my new moves as well, but I ordered case fans and I thought I was just getting like a case fan. But the, the order that I did on Amazon, it was like actually a pack of four. So I, I ordered what I thought was two case fans and I ended up with like eight.

• 30:21Chris Benson?

  Okay.

• 30:22Daniel Whitenack?

  So I just plugged in as many fans as I could into the motherboard. And yeah, I, I forget how many I ended up with, but there's, there's a lot. So that's it. Yeah. The, the ventilation and all of that is definitely a big deal with the GPUs.

• 30:38Chris Benson?

  So are there variances though in, in, in what you saw? And maybe you didn't have a big enough sample, but in motherboard design, I mean, it's clearly that can have a, a fairly profound uh, impact on what your GPU can do or GPUs can do.

• 30:51Daniel Whitenack?

  Yep. That's a great point. So I'm speaking a little bit outside of my domain of expertise here, but um, this is what I learned off of sort of blog posts and that sort of thing. One of the things you wanna look for with your motherboard is what it talks about with PCIe lanes.

• 31:08Chris Benson?

  Yep.

• 31:08Daniel Whitenack?

  So this is in my understanding, the sort of lanes of communication between your CPU and various other things that are connected to your motherboard, like your hard drives or, or other things. So

• 31:22Chris Benson?

  Right.

• 31:22Daniel Whitenack?

  What's gonna happen is when you start adding GPUs, you're gonna start filling up those lanes. So according to, you know, the Jeff Chen blog post, as a general rule, this is probably not true, maybe across the board, but as a general rule, you might think that each GPU requires at least eight of those PCIe lanes. And so what you have to do is you have to find a combination of CPU cause that determines some compatibility with the motherboard. So combination of CPU, then a combination of motherboard.

• 31:56Daniel Whitenack?

  That gives you enough PCIe lanes to support the number of GPUs that you need to support. So there's a little bit of arithmetic there, nothing too complicated, but you have to be sure because some CPUs and motherboard combinations, the pairing, will not support the needed number of PCIe lanes that you need to run those. And then also run, so I added a solid state drive for my hard drive and I've actually on my desk, I have a second one because I already filled it up. That's one item on my didn't do it right list.

• 32:32Chris Benson?

  We'll get to that in a few minutes.

• 32:33Daniel Whitenack?

  So I have a solid state drive and that, that also, you know, takes up some of those. I have an M.2 solid state drive, which is just a little bitty, you know, gumstick sized hard drive. And that requires a certain number of lanes. You also want to think about, you know, your storage and that sort of thing and how much RAM you need to support in the machine as well. All of that stuff went into the motherboard and I just kind of took some recommendations from various blog posts on that. Again, I'm not, I don't follow this stuff religiously, so I don't, I don't know all of what's out there, but I was able to find a combination that worked.

• 33:19Chris Benson?

  Do you have a current list? I know you're going to put it onto the show notes, but I'm just curious if you can highlight each of the, you know, what the RAM was, you know, at least size wise, that kind of stuff.

• 33:29Daniel Whitenack?

  Yep.

• 33:29Chris Benson?

  And just give us the highlights without going too deep.

• 33:32Daniel Whitenack?

  Yeah, here's the highlights. The motherboard I mentioned, the CPU I got was actually, so this is one thing that probably looking back I would change. For some reason at the time I got a Ryzen Threadripper CPU and not that it's bad, it actually works quite nicely. So it's an AMD chip. I think, looking back, especially with my experience with the other Intel machine that I have, I would go back and change that to an Intel CPU, and we can talk a little bit more about that here in a second. But that was the CPU I used. Then I ended up using a total of four sticks of RAM for a total of 64 gigabytes of RAM.

• 34:15Daniel Whitenack?

  And then I have one of the M.2 solid state drives, which is a terabyte Samsung drive. One of the other things you have to be careful about with this is the power supply. You need enough power to power the GPUs and all of that as well. So that was actually one of those things that turned out to be more expensive than I thought in terms of looking around and finding an appropriate. I ended up choosing one that, actually, someone already gave me some feedback, I think, in our Slack channel. Maybe it's not the best power supply, but it does the job. A Rosewill Hercules and then some Case fans.

• 34:52Daniel Whitenack?

  I used the air-cooled CPU cooler, not like, I didn't feel quite comfortable enough yet to run like all sorts of liquid through my machine. I figured that I, you know, knowing myself, I would break a tube and like ruin everything in my, everything in my computer. So, and actually I looked, it's, there was some study I found, I actually probably couldn't find it again. And maybe it was just confirmation bias, but I found like, I think it was some YouTube video showing how the CPU or the air coolers, the good air coolers will actually cool better than a lot of the cheaper water coolers or radiators. So that gave me some confidence there.

• 35:38Chris Benson?

  So now that you've kind of talked about these components and you started to tell us a little bit about some of the things that you've learned and might change as you've gotten some education. Kind of go back through and what would you do different? You know, you've already started on that line, but I'm kind of curious. There's a big learning process that you had to go through this. And neither one of us are hardware people. So what kind of, what do you know now that you didn't know before?

• 36:00Daniel Whitenack?

  Yeah, so there's a couple of things that probably the funniest thing is my case doesn't close. So.

• 36:06Chris Benson?

  Does that affect the airflow then?

• 36:08Daniel Whitenack?

  Well, I think that, I mean, if anything, it improves the airflow, or at least that's what I tell myself. So I got a case, um, is actually, I think it's the same case that from that blog post that I keep mentioning. And I was like, well, I don't want to worry about like finding a different one. I'll just use the same case. But what I didn't realize is, um, so that each of the GPUs, at least the certain GPUs require power directly to the side of the GPU. And the power cables I got with my power supply are fairly rigid.

• 36:43Chris Benson?

  Yeah.

• 36:43Daniel Whitenack?

  They're not like super bendy.

• 36:45Chris Benson?

  Yeah.

• 36:46Daniel Whitenack?

  So when I bring them in the side, they kind of stick out the side and I can't close my case.

• 36:52Chris Benson?

  They're that rigid?

• 36:53Daniel Whitenack?

  Yeah, yeah.

• 36:54Chris Benson?

  So I mean, they're, yeah.

• 36:55Daniel Whitenack?

  They do make these little like 90 degree connector deals. You can buy, basically I just, I got it up and running and set the side of the cases, you know, to one side and I haven't really looked back. Probably at some point, maybe I would look into those, but, or, or like bendy cables or just a larger case. It's a pretty big case, but it's that point like between the motherboard and the GPUs and the, and the case that width. It could actually expand out in the case the other way. So that's one thing, there's that spacing. But I mentioned this other machine that I've been running, which is, I actually really enjoy doing stuff on this machine.

• 37:36Daniel Whitenack?

  It's a Intel based machine that we're using to, as a target system, you know, to test out certain of our pre-trained models for inference and test out our applications on there. And, um, we have some speech applications we're testing and that sort of thing. It's one of these, um, Intel based, uh, NUC computers. They're like little smaller computers. And they have, I think the one I have has I7, Intel I7 processor in it. No GPU in it, um, but it has a lot of RAM, so 64 gigabytes of RAM. And I found a lot of the sort of speech processing and preprocessing stuff on there is

• 38:18Daniel Whitenack?

  super fast. I really like the performance on there. Not that it's amazingly slow on my workstation. That's sort of our target system for our applications is an Intel-based system. And so at the time when I was building the workstation, looking at what was available online for building these, you know, I could get an Intel-based motherboard and processor, but it was a price hike.

• 38:42Chris Benson?

  Right.

• 38:43Daniel Whitenack?

  And looking back, I would tell myself to just go ahead and take that price hike. Because if that was the case, I wouldn't constantly be moving models around between these two systems to do my testing. And, you know, the Intel, the way I understand it, like the Threadripper has, the AMD chip has more cores. And so if you're doing a lot of multi-threaded stuff, it's really good. But the single core speed on the Intel processors is higher. And so depending on what your workload is, then you could actually get a performance boost, even though you have fewer cores, but with a higher core speed with the Intel. So all of that stuff, it's, you know, it seems to work a little bit better

• 39:27Daniel Whitenack?

  for me. And this is just my own personal experience.

• 39:30Chris Benson?

  This is all inference you're talking about here, right?

• 39:32Daniel Whitenack?

  So inference, but also preprocessing. So CPU, I'm doing all my preprocessing on this machine too. And preprocessing is also, you know, part of that CPU workload you have to consider. And I use Intel's Open Vino toolkit to optimize the models for running on Intel CPUs. So if all of that was in a single workstation, like I had the Intel CPU, the motherboard, and I could do the preprocessing, I could do the model optimization, all of that together, rather than sort of swapping back and forth, or doing that in separate places. I think just the saving in terms of my workflow would be useful.

• 40:14Daniel Whitenack?

  So that may be one thing to keep in mind is like, those other pieces of your workflow, like Optimizing your model for certain hardware or pre-processing as well, those are things that you need to think about. You know, those are equally valid things to think about when you're designing a system.

• 40:32Chris Benson?

  So before, just as an aside, since you're kind of touching on it for a second. If you look at what your existing workflow is from beginning to end and the choices that you've made. What are each of the big pieces? Like what frameworks are you using and all that? Just to throw it out there so that people can get a context for it.

• 40:50Daniel Whitenack?

  Yeah, so we're using both TensorFlow and PyTorch in our work. It depends on the certain application, but we have a couple of different products that we're developing that include multiple models in each of these products. Some of them are TensorFlow models, some of them are PyTorch models. And we have both text and speech preprocessing elements that are in there as well. We generally save most of our training data in Amazon S3 or Digital Ocean Spaces, some object store. And then we'll pull that down to the machine and pre-process it. And then save that pre-processed data for training and testing and evaluation.

• 41:34Daniel Whitenack?

  We'll save that actually on the workstation. That way, we don't have to constantly be pulling from some remote source for the data. And then, once we have that locally, then we can iterate on our model training, whether that be with TensorFlow or PyTorch. That's all done within Docker. Like I mentioned, we don't actually have really much at all installed on the machine itself.

• 41:55Chris Benson?

  Right.

• 41:56Daniel Whitenack?

  So we pull down a TensorFlow or PyTorch or our own custom Docker image. Run that. We haven't had any issues in terms of connecting to the GPUs with Docker. So that's worked out great.

• 42:10Chris Benson?

  Are you basing it on Nvidia Docker containers?

• 42:13Daniel Whitenack?

  Sometimes, yeah. So sometimes we have our own custom containers we're using as well. Or, like I mentioned, sometimes we also have containers for certain optimization frameworks that we're using, like Intel's OpenVINO or others that optimize our models for certain targets. Those are sort of a mix. And then after that, in the products we're building, we have a custom way of bundling our models together in what we call a model bundle for a specific deployment. And so that model bundle needs to be built and encrypted. And then that's actually pushed back out to our object store in S3 or digital ocean spaces, depending on the application.

• 42:58Daniel Whitenack?

  And that model bundle is saved there. Then for deployments of that system, that would then pull down that model bundle from the object store and run with that.

• 43:09Chris Benson?

  So earlier when we were talking about motivations and you talked about the break even there, you know, you kind of figured out what made sense. But obviously, uh, when you made the choice and you're not going with cloud because the break even in that case, what are you thinking in terms of your upgrade? You know, because we talked about, you know, being able to fill in with more GPUs.

• 43:28Daniel Whitenack?

  Yeah.

• 43:28Chris Benson?

  Uh, and enhance the capability of the box. But what about over time as, as GPUs start to age? What's your thinking there? And, you know, that makes that a, a good choice from your standpoint?

• 43:38Daniel Whitenack?

  Yeah, I mean, right now, um, I have no choice there because I can't get my hands on any more GPUs because they're all.

• 43:45Chris Benson?

  It's a good answer.

• 43:45Daniel Whitenack?

  They've all ceased to exist on the face of any website on the internet. Um, so I, I can't get my hands on any, but, um, I think right now we're in a pretty comfortable spot in terms of the workloads that we're running. However, we're ramping up some of the work on these prototypes that we built. And I'm guessing if those kind of catch on and are productized a little bit more, then we'll have a need for an interesting new workflow, which is a bit more automated. Um, so right now, most of what we've been doing on this machine at least is prototyping

• 44:24Daniel Whitenack?

  work and preparing very custom deployments, um, for very specific applications. In the end, we're going to build a bit more of an automated workflow where, for example, one of the things we're doing is spoken language identification. So audio comes in and then we detect what language is being spoken in that audio clip. And so if, for example, in a certain application, we want these four languages instead of the six languages that we originally had in our model, we want to be able in a very quick way to spin up a pipeline that would redo the training with sort of different combinations of data. And so that's something that I think would be interesting.

• 45:08Daniel Whitenack?

  I know we've had a couple of conversations recently. I, there's a lot of ways to do that. Um. One of those, we, we've talked to the, the Pachyderm crew before on the podcast. They have a great pipelining, uh, system.

• 45:21Chris Benson?

  Definitely.

• 45:22Daniel Whitenack?

  We've also talked to, I remember Allegro AI has a sort of, uh, training and automation system that you can run on an on-prem instance. Um, and so that may be another option that I can install that Allegro system on the workstation and run it that way. Um, there's a lot of different options there that we could possibly use. Um, and also upgrading the sort of, um, monitoring there. Right now, um, that process is fairly manual as well. So maybe utilizing either, like I mentioned, um, maybe something more than tensor board. So we just had a conversation with Weights and Biases CEO.

• 46:00Chris Benson?

  Sure did.

• 46:00Daniel Whitenack?

  Lucas Bewald. And he talked about the great stuff that they're doing. And, you know, all of that's, I think, not too hard to implement in that workflow. So those are some of the things on my mind.

• 46:11Chris Benson?

  So, you know, as we're closing out here, you know, at this point, would you do the whole thing over again? Because it, it sounds like you, you've learned a lot. There's definitely some cost advantages to what you've done. But I also know that as we recorded various episodes over the weeks that you were engaged in this, that, you know, there was a lot of work involved.

• 46:31Daniel Whitenack?

  There was.

• 46:31Chris Benson?

  As you were kind of statusing along the way, was it worth it?

• 46:34Daniel Whitenack?

  I think it depends on, uh, I mean, of course your need, but also your personality. I really enjoyed this. It was something that I found a lot of enjoyment in, and I think that was worth it in and of itself. But if I was to look back, I would definitely do it again. And I think, you know, the things like I mentioned that I would kind of tell myself in retrospect is, you know, don't skimp on the CPU or the RAM and over project your data storage. So go ahead and get yourself whatever it is, four terabytes or, you know, whatever it is, something that you're not going to fill up quickly as soon as you start working with speed or video data or something like that, one terabyte. It seems like a lot, but it's not. It turns out, you know.

• 47:20Daniel Whitenack?

  It's easier than you think to fill up that space. So get yourself, even if it's some cheaper hard drives that can be just sort of cold storage, do that. But yeah, it was a fun experience and I would definitely recommend doing it. You know, it's, it's fun to have that sitting, you know, by me in my, now my home office. And, you know, now we don't have to heat our house quite as much. So it's just winter. So it's good timing.

• 47:47Chris Benson?

  Side benefits you get there. Okay. Well, that was pretty cool. It's a long overdue, this conversation. I'm looking forward to seeing what listeners inform us because you know, both of us are, this is not our specialty.

• 48:00Daniel Whitenack?

  Be gentle.

• 48:01Chris Benson?

  Be gentle on Daniel. It'll be interesting though, to see what we get in the Slack channel and other social media, the LinkedIn group and stuff, and see if anybody can school us a little bit. Maybe this is one of their, their things and they have that knowledge. So looking forward to, to hearing people share that. Well, Daniel, thank you very much for taking us through this. I'm glad that you did. And thanks for teaching.

• 48:23Daniel Whitenack?

  Yep. Talk to you later.

• 48:24Chris Benson?

  Talk to you next week. Yeah.

• 48:29Speaker 03

  Come hang out with Daniel, Chris, and hundreds of other AI practitioners in our community Slack. It's a cool place to be. Not a lot of noise, some great signal, and best of all, it's totally free. Check it out at changelog.com/community. And don't forget to follow the show on Twitter for AI news and links, highlights from past episodes, and more. We are at Practical AI FM. We love to have you following along. Thanks to Daniel and Chris for hosting Practical AI, week in and week out. To the mysterious Brake Master Cylinder for the excellent beats you hear on all Changelog podcasts. To our sponsors who have our back, Fastly, Linode, and Launch Darkly. And to you for listening. We appreciate your time and attention. That's all for now. We'll talk to you again next week.