Worlds are colliding - AI and HPC

In this very special fully-connected episode of Practical AI, Daniel interviews Chris. They discuss High Performance Computing (HPC) and how it is colliding with the world of AI. Chris explains how HPC differs from cloud/on-prem infrastructure, and he highlights some of the challenges of an HPC-based AI strategy. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean now offers three managed databases https://www.digitalocean.com/products/managed-databases/ — PostgreSQL, MySQL, and Redis. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: HPC course on Udacity https://www.udacity.com/course/high-performance-computing--ud281Google Colab https://colab.research.google.comSIParCS internship at NCAR https://www2.cisl.ucar.edu/siparcsUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:46Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and dig into learning resources to help you level up your machine learning game. I'm Daniel White neck. I'm a data scientist working with. With SIL and today's a special episode, a fully connected episode. It's always Chris Benson, my, my co-host and I that do these episodes together. We kind of chat back and forth, but we're talking about kind of what's going on in the AI news and some of the things we're seeing.

• 2:26Daniel Whitenack?

  And one of the things that, that I was kind of seeing was some mentioning, more mentioning of HPC or high-performance computing clusters in the AI context. And it turns out that Chris Benson, my co-host is somewhat of an expert in this area and working very closely in this area with Lockheed Martin. So I thought today we could just take some time and I could interview Chris a little bit about HPC clusters. And we could discuss, you know, what they are, how they're being used, what the future is and all of that. So you ready for that, Chris?

• 2:59Chris Benson?

  I sure am. It's, uh, I'm looking forward to this episode. It'll be, it's turning things on, on its head just a little bit here.

• 3:05Daniel Whitenack?

  Yeah, you get to, get to have some empathy for our, for our guests and, uh, figure out what that's like. You can let me, give me some pointers about my, my interview skills after, afterwards.

• 3:17Chris Benson?

  Okay. I'm, I'm nervous for the first time in forever, Daniel.

• 3:21Daniel Whitenack?

  All right. We'll see how it goes. See if, if either one of us crashes and burns, uh, in, as we're doing something slightly different. But I think we'll be all right. Maybe just to start things out, could you kind of remind maybe new listeners or those that haven't kind of listened to the intro episodes? Remind us what you're doing, where you're working and kind of how you ended up crossing ways with each, the HPC world.

• 3:46Chris Benson?

  Okay. So I'm a principal artificial intelligence strategist at Lockheed Martin. And I work directly for the chief data and analytics officer, um, whose name is Matt Terrassio. And we do a lot of things for the company at a corporate level, supporting the four big business units. And they each are entities unto themselves. You can go out on Wikipedia and, and check Lockheed Martin. We have missiles and fire control. We have aeronautics. We have space. And we have Rotary and Mission Systems, and those are the four business units. And they're all doing incredibly cool things. And I'm not trying to sell them. It's just it's a really interesting environment in which to work. And so we're, our team tries to support that.

• 4:28Chris Benson?

  And one of the things of many that we are doing on our team is supporting the high-performance computing efforts along with other teams. It's not just us doing that, but we are very involved in high-performance computing strategy on how to support all the different people. And Lockheed has something on the order of 110,000 people, give or take. And so it's a large company and a diverse set of things that different teams do throughout the company. And so what we're trying to do is we have lots of HPC capability already within the company, but we are reassessing. And so in terms of how we're doing it and providing support. And so what I can talk to is certainly about kind of all the types of decisions that have to be made.

• 5:13Chris Benson?

  I obviously won't be talking about Lockheed's specific decisions and how we're implementing because it's proprietary knowledge, but we are kind of neck deep in all the different decisions and how do you do this in 2019? And this is a changing space. So I'll turn it back over to you and then we can dive forward.

• 5:31Daniel Whitenack?

  Yeah, sounds good. I'm not sure if you knew this, but I had a very brief interaction with the HPC community back after, well, I mean, I started out in academia. So in academia, if you're in any sort of computing research field, oftentimes you'll interact with HPC clusters. But then after my undergrad, I did an internship with the National Center for Atmospheric Research in Boulder, Colorado. They operate several, I'm not sure actually at the moment what they operate, but at the time they operated several big supercomputers. And, you know, I was doing some like benchmarking of vector computing on a new IBM Power Six computer. So it's been a while for me.

• 6:14Daniel Whitenack?

  So I'm interested to kind of like, I have some concept back in my head of like my interactions with the cluster at that time and what people were doing, but I'm sure it's just like vastly different. Now, so I'm I'm really interested to hear how things have progressed and how that intersects with AI. Maybe to begin with on that subject, could we just kind of define what is high performance computing or HPC? Like, how is it different than, let's say, some things that might also be in people's mind, like cloud computing or something like that?

• 6:46Chris Benson?

  Yeah, so just like defining AI can be harder than one might expect because of all the diversity of opinions on what it is. High performance computing is also undergoing quite a transformation at this point in time. And so I would suggest that I will offer my take on what it is and that there will certainly be people out there in the audience that will disagree with me on this. But just as if we were defining AI, so really there are applications in the world that need a tremendous amount of computing resource. More so than you're typically finding from either on prem or traditional kind of cloud resources. And you need to be able to scale up with a processing capability that is often done massively in parallel

• 7:29Chris Benson?

  to be able to tackle a computationally intense problem. And so when I talk to people that have been in this space for a long time, and I've only been in it for several years now, not only Lockheed, but at previous employers. But the the nature of the field has changed a lot. And so like if I talk to people that have been doing things like simulation, you know, obviously, you know, Lockheed builds platforms for our customers that, you know, in various environments, space and underwater and whatever. And so that requires a lot of simulation. And so if you talk to to people that have been running high performance computing clusters for a period of time,

• 8:07Chris Benson?

  what they're trying to do is say, I need to take maybe a new vehicle or something and and try it out in a simulated environment to solve problems and figure out, you know, while it's still in this state, what it needs to be able to do and what the problems are and stuff like that. Before you get in the real world with a real device and have it have it not working the way you're expecting. And so the traditional way has been having these massive clusters of CPUs. And it's been incredibly expensive. To do that historically and you often saw that in, you know, government sponsored laboratories that, you know, were associated with government programs. I mean, like off the top of my head, things like the Livermore Computing Center, the Lawrence Livermore National Laboratory.

• 8:48Chris Benson?

  They maintain and there are others out there, but that's kind of what I think people traditionally think of is being able to say, I need to apply, you know, 10,000 or 20,000 CPU cores to a problem in massive parallel to work through it. And so that's how I see it historically, but it's not really how I engage it personally.

• 9:09Daniel Whitenack?

  So maybe we can just kind of dig into a couple of these jargon things that you mentioned. So when you're talking about a cluster, I mean, we're talking about a certain number of compute instances, whether those be virtual machines or physical nodes that are working in concert to do something. Now, I'm sure people are also maybe thinking like, oh, well, in the cloud, I can have like a Kubernetes cluster. I can have a bunch of instances, right? Or on prem, I can buy a bunch of servers and hook them together. But am I right in like, one of the elements of a HPC cluster is really that the nodes are tied together in a specific way,

• 9:49Daniel Whitenack?

  like even hardware wise, that makes them, for example, communicate very differently, maybe than a standard kind of on prem infrastructure for running web servers or something like that. Like they can communicate in a sort of very efficient way and also handle kind of very large amounts of data. Is that like one of the differentiators between like, let's say, a bunch of on prem servers running websites and a high performance computing cluster?

• 10:16Chris Benson?

  Yeah, I think so. And as I try to answer this, I want to acknowledge that the CPU cluster side and the software stacks that go into that is not my area of expertise. And so I kind of want to go into that. I'm bringing kind of the AI ML perspective where I'm much stronger talking about the things that you and I often do Kubernetes and stuff. But part of it I've discovered is, is really cultural. There are software stacks that are applied to tie these clusters together. They tend to be close. Or to the hardware in a lot of cases that I've seen kind of generalizing some of the use cases I've seen where people have put together different architectures.

• 10:52Chris Benson?

  You'll sit down at a terminal and do like a virtual desktop, but the virtualization across clusters is very, very close to the hardware to pull that together. And so the traditional view of that is very different from how we look at it in the AI world these days. And so it brings some challenges into the case that as A I M L is becoming part of this, and we'll talk about that obviously in a few minutes. But as it's becoming part of that, you have very different paradigms on how these clusters are constructed and how you interface with them. And so there's really, I've learned the hard way, there's really not a great one size fits all across all the use cases.

• 11:32Chris Benson?

  And so if you have all those use cases, the maybe lucky people don't, maybe they're they're fortunate that they have a particular specialty they're addressing, which reduces the total scope of what they have to do. But if you're addressing many, many different types of use cases, then it's it can be a struggle to do that, to be able to do that. But like, you know, we were talking about Kubernetes and like, if you look at more of the traditional CPU side and and kind of what I just described about being able to get close to the hardware. There is a technology called Singularity, which is kind of Kubernetes like, but it's kind of, you know, and I'm going to give a completely non-technical squishy definition. It's containerization, but it's not in this quite to the extent you think of like when you think of Docker and Kubernetes.

• 12:18Chris Benson?

  It would be, it's an open source project that is a lot more like it's containerization, but it's a lot closer to what we traditionally think of as V M's. And and that's one of the popular technologies I've seen in this space that people are looking at. And it's not the only one, but it's being open source. It's a good one to talk about since we tend to advocate for for open source solutions on the show here. And so that's one where that community is culturally trying to take advantage of containerization, but probably not the way some of us who have come from a traditional Docker Kubernetes over in recent years would think of. It's not quite to that point.

• 12:53Daniel Whitenack?

  So kind of what I'm hearing a little bit is that, and I'm just kind of trying to break it down for my simple mind, I guess, like if I think about, let's say I have instances in the cloud, right? There would be ways for me to spin up a huge number of instances, run some like Python thing on all of them that communicates all between all of the nodes and all of that. But really what I'm doing is I'm spinning up these sorts of generic environments that are really geared towards a wide set of applications from web servers to data processing, to databases, to whatever it is I can run on those instances. They're meant to be generic, right? Whereas kind of what I'm hearing is that an HPC cluster,

• 13:36Daniel Whitenack?

  like from the start, you say, well, this cluster, I'm going to build this so that it can run massively parallel data intensive applications at scale. What do I need to put into this cluster to make that happen? And I guess that could include things like specialized connections between the nodes. It could include specialized hardware. It could include specialized software setups, specialized like queuing systems and job scheduling, specialized ways of dealing with containers and virtualization. So it's really kind of the amalgamation of all of those things together that are really geared towards the specific use of the cluster, I guess. Would that be accurate?

• 14:17Chris Benson?

  It would be, and most of that is going to be outside my area of expertise because we have other amazing people on the team that know that stuff inside out. And so I am learning that. And when I say that, I'm talking about the kind of the CPU side of the equation. And so there are schedulers, like I mentioned Singularity, and it does scheduling, and it does containerization, and it's designed to take advantage of all those processors across the cluster. And so in that way, it's similar to the world that you and I, you know, more often operate within. But it's not exactly the same. And if you currently, if you try to apply more of a traditional CPU-based simulation paradigm, it doesn't work well in a Kubernetes cluster because that was one of the first things I

• 15:00Chris Benson?

  learned as I explored this thing is why aren't we using that? Or why wouldn't somebody use that? And so there are reasons that are largely beyond my expertise that I was given. And I trust those experts at that. I have learned that they really know their stuff. And so we have that represented on our team and I kind of leave that alone. And so one of the things that I have really been focusing on myself is more on the AI ML side, which looks a lot more like the environment that we're used to in terms of largely, you know, not even specific to our company, but in general, there is an expectation in high performance computing now that AI ML use cases are now requiring that level of computation.

• 15:40Chris Benson?

  And so we're seeing this kind of rapid race up the curve where, you know, originally people would say, I have a GPU and, you know, to run things on. And then they said, I have a small cluster that's either on prem or, you know, now cloud providers that are providing that. And as our industry on the AI side and machine learning side becomes more sophisticated and our models are becoming more complex and stuff, the need to drive that computation for highly complex use cases is really shooting up. And also it's interesting that a lot of the traditional simulation side that would traditionally have been done on a CPU based cluster, you're seeing some of that come over into a GPU world at this point.

• 16:21Chris Benson?

  So the AI ML workload perspective was really not part of high performance computing until fairly recently. And now you're seeing those two worlds merge right now. So it's a very fast moving field.

• 16:47Speaker 03

  This episode is brought to you by Brave. The Brave team is on a mission to fix the web by building an open source, privacy focused and performance oriented browser, browse the web up to eight times faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built in basic attention token. Yes, you heard that right, a real world use case for blockchain. Download Brave for free using the link in the show notes and give tipping a try on changelog.com.

• 17:27Daniel Whitenack?

  Okay, so when we say something's going to run on an HPC cluster and it's massively parallel and processing massive amounts of data, could you give us kind of a perspective on like what approximately the scales we're talking about are? I know you mentioned a certain number of CPUs. Could you kind of just give us a perspective on how big are these clusters that we're talking about?

• 17:53Chris Benson?

  Sure. So on the CPU side, a large use case can consume tens of thousands of cores to run simulations, you know, in tremendous detail and be able to do all the, all the parallel computation that's required of that. That is not just, people from our side with our bias tend to think, oh, well, that's going to be eclipsed and the world goes GPU. But there are many use cases that are not necessarily specifically optimized for GPU. You're seeing some crossover there and there are companies out there, you know, that are in the GPU space, Nvidia being one of them, that are basically trying to pull traditional CPU-based use cases over into the GPU world. And you have to kind of do that assessment of what that means to your organization and

• 18:38Chris Benson?

  the projects that you're involved in. But, you know, it's kind of funny on, so you can get to that level on the CPU side, but on the GPU side, it's interesting that as HPC is really addressing the artificial intelligence machine learning space at this point, then you get into a situation where you can almost consume, for really sophisticated training techniques, you can consume a tremendous amount of computation. So it's really not always about just, I have X number of GPUs. Okay, that's my requirement. Going forward, we have, you know, the concepts, you know, in training of like, you know, mass hyperparameter exploration, where you're trying to find optimal sets of hyperparameters for your AI model and, and you're training them in parallel varying hyperparameters so

• 19:23Chris Benson?

  that you can find the, the various, you know, performance gains and optimizations to do that. And that's one way where you essentially can kind of absorb all the compute that's currently available. To you. And then there are other things like deep reinforcement learning. We'll get into things like large scale self play where you are. You're allowing the agents to run and going through that training cycle of deep reinforcement learning. Also in parallel to speed up and to also find different avenues through that. And then at the end of the day, those are kind of served by auto scaling anyway. So it's less of, well, I have X number of GPUs and I'm going to run with that over a given period of time. And that meets my requirement more like if we're going to do something like this, how much capacity do I have right now?

• 20:06Chris Benson?

  It may be that in my prior effort with a slightly different approach, I only needed a certain number of GPUs. But if I'm, for instance, going to jump into doing this mass scale hyper parameter exploration. I might try to suck in every GPU I can to get through that so that I can get through it in minutes or hours instead of days or weeks or months. And so there's this, I guess, the elasticity necessary in your high performance computing cluster becomes very important. And so you have to have strategies that can accommodate those types of use cases.

• 20:40Daniel Whitenack?

  I'll definitely say that it's impressive. Like you said, the amount of compute that's needed even to train like a single model in certain cases. And it certainly explore hyper parameter spaces. So for those that might not kind of understand this whole idea of hyper parameter optimization. Like if I'm going to train my neural network, I have to make decisions and put in user defined parameters, right? These, these parameters that are not set through the training process that are things like, you know, the number of nodes in this layer or my learning rate or like a dropout or something like that. So, so there's all these parameters and one way of figuring out how to best set those parameters to get the best model is to just try a whole bunch of them, right?

• 21:22Daniel Whitenack?

  Which obviously takes a lot of computational power, but you are kind of exploring that whole space. I just read some people probably say you might've seen this too, Chris. There was an article recently that kind of showed how some of the large scale language models that are being trained now, like training one model took about or contributed as much. Carbon input to the atmosphere as running like five cars for their entire lifetime of use, which is just like, I don't know, like putting it in terms of something that like hits home real, real world, like that you interact with daily rather than like petaflops or something like that. It just really hits you that this is significant in very technically interesting ways,

• 22:04Daniel Whitenack?

  very impactful ways and in a positive sense, but also potentially there's there's side effects there as well.

• 22:12Chris Benson?

  There sure are. I mean, anyone who's been with his, listening to the show for long knows that you and I are both incredibly social conscious people in terms of, you know, how we perceive the world and the kinds of choices that we make. And so this is definitely a weak spot in in dealing with providing massive amounts of computation in within a reasonable time period that needs to be addressed. And so, yeah, I I remember when that came out about the, you know, running five cars or whatever it was for a for a year and I was a little bit stunned. And so it's one of those things that we need to figure out.

• 22:44Daniel Whitenack?

  Yeah, definitely. So let's kind of jump back to and maybe turn a little bit towards the HPC for AI and how these worlds are are colliding because I remember, for example, when I did that internship that I mentioned, the primary applications that I was working with were climate modeling applications. So I know people have kind of used these sorts of clusters for quite a while for these sorts of climate models. I know also in grad school, when I was doing like computational chemistry calculations where you're basically trying to calculate properties of materials based on what you know about how the physics work for for atoms and molecules.

• 23:25Daniel Whitenack?

  So we were kind of submitting jobs to HPC clusters at that time in a couple of different places around around the country. And so I know that there's been kind of this history of HPC clusters being used for these large scale, like you said, simulations and scientific computations and and those sorts of jobs. But I also know, you know, recently I was talking to, so I live in the same town where Purdue University is and talking to one of the data Scientist that works for Purdue University and he was saying that now they have a sets of nodes and they're continually buying more that are specifically geared towards AI applications. So I know that that this is happening and obviously you're working in this area. So could

• 24:09Daniel Whitenack?

  you kind of describe maybe the why HPC is relevant for AI and maybe when would I want an HPC cluster for doing AI versus maybe just spinning up some stuff in the cloud and vice versa?

• 24:24Chris Benson?

  So yeah, it's a it's a level of really where a use case requires the horizontal scale of what a cluster provides because you're still using the same GPUs, you know, for that. But the question is, you know, the cluster gives you the advantage of saying like, I can go get the latest Nvidia GPU or any of their competitors and be able to say, okay, I'm going to go do this for my project. You can be a student and go do that and run it on one or go to these cloud services where you say, okay, I'm gonna I'm gonna lock into a, you know, a really good GPU there and use it. In industry, though, there are use cases and I haven't only run into this at Lockheed, I've run into it previously as well. You'll be dealing with a solution that may not only have challenging

• 25:09Chris Benson?

  models to train, but but in many cases, you have many, many models that are working together, that are collaborating, where each model is narrowly performing a particular task, you know, in terms of its inference, that it's performing it with great accuracy, but because of the scope of what you're tackling in the problem set, there may be many of those type tasks and you have different models that are applied to each one, but they have dependencies across them. You know, they're not standalone in the sense of each model may be only attending to its own input and inference and output, but some of those inputs may be from other model outputs and stuff. And so you may have to manage quite a few models that are interrelated and those

• 25:54Chris Benson?

  relationships matter as much as just the construction and training of the model itself. And so one of the advantages about having a cluster is if If you're dealing with a complex use case like that, where you have these kind of tight dependencies between different models, then it may not be just retraining one. It may be that one model and how it's performing and what it's doing and the choices you make there affect other models that are highly dependent on it. And you may change what kind of parameters you use, your hyperparameters you're using and stuff like that. They can alter the inference itself, but you're having to look at it from a system perspective instead of just a model perspective. And so clusters can be really effective when you're iterating on those types of things.

• 26:38Chris Benson?

  And you're having to get a whole lot of training done for every iteration and then be able to go back. And you don't want to wait weeks or months because it's not realistic in the real world. Without the cluster, the problem that you're trying to solve would not be practically doable in the real world.

• 26:55Daniel Whitenack?

  It's super interesting and I definitely see the advantage there. But maybe it's like my cheapness or the fact that I work for a nonprofit or something. But I'm thinking like, oh, like it seems like there's so much risk. Like you're saying, no matter what comes out from Nvidia, like the latest GPUs, the latest accelerators, types of software that you can run on certain architectures. All of that's pretty much available very quickly in the cloud. And so you can have access to that sort of thing very quickly in a flexible way. And it just like, it kind of brings about a little bit of fear in me. If I think about, oh, like we're going to choose to invest in a specific like architecture and build out this huge cluster, which I'm guessing like takes a ton of time.

• 27:39Daniel Whitenack?

  It obviously takes a ton of money. And then like technology is progressing so quickly that like, how are you not scared that like you build this thing? And then like it's obsolete in a year. How does that, like, how does that sort of work in a company or in your strategy?

• 27:57Chris Benson?

  No, that's a great question. And the, your HPC strategy has to accommodate that natural refresh, that natural progress, because you don't want to buy into a technology and expect to just to leave it there. So it's not some Something that you just go do and walk away from you. You're going to do it in many phases that accommodate, you know, changes in what's available to you. And you try to, you try to look ahead and structure that automatically. And then you try to take advantage of what you're trying to accomplish. For instance, some of the kind of more typical principles that you're going to find in HBC strategy that you're trying to accommodate is you're trying to remove barriers to innovation.

• 28:38Chris Benson?

  And you're trying to allow with one of these clusters, the ability to kind of develop anywhere with a consistent user experience and deploy wherever you need, based on your different use cases. You need a solution stack that is, that is kind of consistent with what people would expect to find, whether it's inside your own organization or external to that as you bring in new talent and be able to allow that stack to, to evolve over time and, and refresh. And it needs to support things like agile development and iterations and such as that. And then kind of what I, what I just alluded to is that user experience is really, really crucial because you can put a huge, amazing cluster that you're investing many, many millions of dollars into.

• 29:22Chris Benson?

  But if your user experience is a very bad one, either you drive users away and they seek out other alternatives, or you reduce their productivity and you reduce their ability to rapidly work on the problem set that they're trying to. And all those things hurt your organization. And so with all of those into account, taking those into account, you have to think about, you know, some of the, the more obvious things are like on-prem versus cloud in terms of how you're structuring and hybrid is another popular thing that people are talking about. And so how does your population of, of data scientists, how do they use it? Are they, are they tending to do it just here in certain hours? Or maybe will they have a week of intense usage and then they're not doing much model training in the weeks to follow?

• 30:08Chris Benson?

  Or do you have a consistent level of training requirement available kind of round the clock, seven days a week that you're doing? How do those moments spike over time?

• 30:18Daniel Whitenack?

  Yeah, so I'd love to dig into that user experience. So I should clarify that the main reference I have for this is like back when I was doing this stuff in my internship and other things. And just for reference, like the user experience with that was like. Okay, here's what I'm gonna do. I'm going to log in to my home space in the cluster. The first thing I've got to do is install, or actually compile this Fortran code against whatever sorts of things are on the cluster, which I'm not totally sure what's there. And like, basically, it would take me days to get things like configured correctly. And then I'd have to submit a job to a job queue, and wait for that job to get queued on the cluster.

• 31:01Daniel Whitenack?

  And then, like, it would get queued, whatever, however many hours later, I would get a notification that I had oversubscribed memory and like, my job crashed or something. And then I'd like, kind of, queue it again and go again. If I compare that to sort of my workflow now, where I'm thinking about, "Oh, well, I'm gonna write some Python, I'm gonna push it up to a node in the cloud, and I'm just gonna use like a S3 client to pull down some data from an object store." And then I run my job, there's a GPU on there. Like, that workflow is just so vastly different from what I remember from the HPC world. Like, even the data side of things in the HPC world,

• 31:43Daniel Whitenack?

  I remember like, there was literally a tape silo. For those that don't know what a tape silo is, it's like, a data store where things are written to these physical tapes. And there's a little robot arm in there that like, when you submit a job and you say, "I want to attach this storage," the robot arm goes over and grabs the tape and like, puts it in to something. I don't know exactly, I still don't know exactly how it works. And then connects that to your node so you can have access to it. So it's just like, these two worlds, there's like, the workflow side of things, user experience side of things, at least in the back of my mind and how I think about it, seems so vastly different. Are there ways to bridge that gap now, to where like, "Oh, I can spin up a Jupiter notebook

• 32:26Daniel Whitenack?

  and run a job on a cluster." I don't actually know what's possible.

• 32:29Chris Benson?

  Yeah, so I mean, probably a good place to start is for that answer is looking at the cloud providers that we already are familiar with and creating that user experience for the best possible workflow for a typical user is something they are spending an enormous amount of time on. And so, you know, all of them, you know, Microsoft and Amazon and Google and Nvidia with their cloud and stuff, they all have approaches. Speaking only for myself, I think my favorite of all those, having used multiple of them, is probably Google Collaboratory. And so, where it gives you kind of a free Jupiter Notebook environment that you can use within costs, like if you're doing something as a project on your own

• 33:11Chris Benson?

  time, and not necessarily working with clusters and stuff. But the ability to just get into a notebook and do that with just individual GPUs, they've really, I think, done a good job of making that seamless. And I think so one of the challenges right now that there are, you know, companies that are doing, like one of them is a proprietary solution by Domino Data Lab. I know they're out there and there's others as well, that are trying to take that kind of simplicity in your workflow and apply it to large-scale clustered environments. In my view, I don't, I haven't seen anyone that I thought was perfectly there yet. There's nobody that I'm seeing do that so far, that I think is doing it with the simplicity that

• 33:54Chris Benson?

  I love, like I said, in Google Collaboratory. And so, I'm hoping to see that. Particularly, I would love to see open source solutions that will allow us to get there. But I really think if you don't support your users well in that way, you're just going to reduce your productivity and increase your cost.

• 34:09Daniel Whitenack?

  Okay, so we've talked a lot about like the, what an HPC cluster is, how the experience kind of differs, what the scale of these things are. Could you describe some of the AI use cases? So you mentioned like reinforcement, learning, self-play, these other things. I'm curious if there's like particular, oh, I guess you mentioned hyperparameter tuning as well. If there's particular types of AI problems or parallelism that fits really well in an HPC setting? And maybe also if there's types of AI workflows that would not fit well in a HPC setting?

• 34:47Chris Benson?

  Sure. I think it's less about the specific application and it's more about how you're combining different models across. So you may have a problem where you're building a model and it can be any of the things. It can be a CNN convolutional neural network, or it could be a generative adversarial network, or whatever, and you can do those without clusters. I think the place where the cluster becomes very advantageous is when you are combining a bunch of those together. You could almost think about it as like using Legos and you have different Lego parts that are each representing a different building block and you put those together to build your little Lego house, or whatever

• 35:30Chris Benson?

  you care about. And when you're trying to iterate on those issues where you're combining a bunch of different models and there's a lot of dependencies between those models, I guess in that way, as I'm hearing myself talk, it's not so different from enterprise scale software development in general. It's not even specific to AI. You develop a model architecture to solve the problems and you, and so it's not one at a time. It's that massive horizontal parallelism that you need to iterate. And when you find a use case that needs that massive horizontal scale to iterate effectively and in a timely manner, that's where the cluster really helps. The other place would be the fact

• 36:11Chris Benson?

  that if you're a large organization or a cloud provider that is serving many, many different teams of people working on problems and you have, they all are using a certain amount of your capacity and you're trying to accommodate many different use cases with many different characteristics and how they're using the resource, then that's where a cluster and being able to provide it in a form of a cloud, I don't necessarily mean a cloud provider like Amazon, Google, Microsoft, it can be an internal cloud, but you need to be able to handle those and make sure that all the people, all the teams in your organization are able to be productive when they need to be productive without you being the single point of constraint on them. And so that certainly has a lot to do with

• 36:56Chris Benson?

  why any large organization is going to invest in these, is being able to ensure that all of their business units are never constrained by compute resources, that they may be constrained by their problem set or whatever, but that the compute is not the issue.

• 37:11Daniel Whitenack?

  So I guess, I mean, there are cases where companies will have a sort of on-prem infrastructure that's more generic, like kind of a cloud-like environment, their own infrastructure where they're trying to enable generic workloads. But it sounds like a lot of the things that you're talking about, or at least there are a good number of organizations like Lockheed or others that are specifically building clusters that are geared specifically towards AI, right? I'm guessing that like, you're not going to build out a whole, like a thousand GPUs or whatever. I don't know what the scale of the GPUs are in these sorts of clusters, but you're not going to build out that sort of thing for just generic workloads.

• 37:56Daniel Whitenack?

  Like, you're making a commitment for the long term to really invest in AI applications with this cluster, right? What sort of pressure does that create in terms of, I'm assuming once you have that cluster in place, and you know, I don't know what the scale of the investment is, but I'm sure it's amazing. What sort of pressure does that create to really be squeezing everything you can out of that cluster? How do you kind of make sure that your AI is so dynamic and it's changing so quickly, right? How do you kind of guess, you know, oh, I'm going to need this scale of cluster for all of these AI sorts of problems that we're solving when AI itself is changing so rapidly and the types of models are changing so rapidly and all of that?

• 38:41Chris Benson?

  Well, I think, first of all, we have a lot of great strategic partnerships out there with other organizations that have similar interests and in some cases, similar scale in terms of what they're trying to address. So it's not only building out the infrastructure, you have to buy the hardware and you have to kind of make an estimation on what you think your GPU utilization might be. If you don't have a lot of history in that, that can be a real challenge. And I think every organization I've ever been a part of over the last few years or talked with has had to tackle that. And I don't think, I don't know that there is a great way of doing it. But I think part of it, part of the challenge in Answering that question is that to some degree, if you do a good job of it, then if you build it, they will come.

• 39:26Chris Benson?

  You know, to use the Field of Dreams quote there. If you have a great infrastructure that suddenly increases people's productivity, then whatever your historical thing has been in terms of utilization and uptake on your systems, you're very much likely to to have an uptick on that when you provide a great way of engaging on that. So you kind of have to accommodate that your success factor of I'm of wow, I'm meeting everyone's expectations. And now it's almost getting the better of me if I'm not careful. And then you have to make sure that you're that not only is the hardware refreshable, but that the software is extendable too. If you think how fast any one of the things that go into this, you know, if we talk about a Docker Kubernetes stack and you think about all the advances in Docker and Kubernetes and that they are constantly evolving

• 40:10Chris Benson?

  because they're in such widespread use. And then those are not specific to an AI workload. So I know that like because they have public stuff out there, you know, Nvidia has a production grade AI platform that they use internally on their own massive, massive stack that they have for their self-driving car stuff, which is called Maglev. And you can Google Maglev and there's some information out there on it. Not a whole lot because it's not a product that they sell. It's an internal thing, but they, I know that's how they approach within the context of being inside a Kubernetes cluster. It's how they approach all the AI specific workflow tasking that has to happen to make something work well. And Google has their approach and Amazon and Microsoft, they all have their approach.

• 40:54Chris Benson?

  And we may eventually see some good open source solutions to be able to do that. But it's not just a hardware thing. We tend to get locked up into when you think of HPC as just the hardware, but you have to have a stack which is enabling all the things you have to do in your workflow to get it done. So it's quite a lot to think about and especially allow to be growing at varying paces throughout the stack.

• 41:17Daniel Whitenack?

  Yeah, that's a great explanation. I guess maybe to turn to a more forward-looking question, what excites you about the future of AI on HPC? Or like, what trends are you seeing in terms of HPC usage for AI applications? And what of those are kind of the things that you're most excited to follow and be part of?

• 41:42Chris Benson?

  I think it's really tied into how I see AI itself going forward, because the AI side of HPC, you know, the HPC is the enabler for the, for what you're trying to do with, with AI models. And so what we're seeing an explosion of over the last few years is, just in the time we've been doing this podcast, you know, people, it used to be people talked about building a great model to solve their problems. And now you're seeing production cases where you might have many models working in a solution, and you're seeing the commoditization and democratization due to these oftentimes open source software driving it, that is really driving the cost down. So just as people's desire to solve more and more complex problems with a variety

• 42:27Chris Benson?

  of neural network solutions working collaboratively together is increasing very rapidly, I think you're going to see that the providers are, they are doing that. You're going to see all the major cloud providers certainly, moving from, let me grab a GPU, to clustering as a service and being kind of the way you think of it, to where you may have service agreements that accommodate a baseline of a number of different GPUs with different types of elasticity models in there. If I go and look right now, I'm not up to date on the latest offerings in all cases, and I need to see, some of them may already be starting to do that at this point. But that's going to become really common, I think. Because certainly, all the large organizations like the one I belong to are, this is becoming a standard part of many things.

• 43:10Chris Benson?

  You know, a year or two ago, doing neural network development was still a bit of a specialization. And I think what we're seeing is that you're, is it is now becoming part of system and software development is a, in general, and becoming a very standard skill that's expected to be part of any solution going forward. And so I just think the HPC space is rapidly expanding and advancing to be able to accommodate this exponential AI growth that we're all experiencing.

• 43:39Daniel Whitenack?

  Given that, I know there's probably people out there thinking maybe that are excited by this and they would eventually love to be working with Lockheed or other larger organizations, maybe even government organizations that actually have the ability to build out these large-scale clusters, right? Or maybe there's people in, working in academic research, a lot of times academics use HPC clusters that they get grants for and that sort of thing. Assuming people don't have the funds to set up their own HPC cluster in their basement or something, which I'm guessing there's very few of those people out there, are there any ways to kind of learn at least a little bit about the, some of the ideas around HPC

• 44:24Daniel Whitenack?

  that people could dig into in terms of a learning resource?

• 44:28Chris Benson?

  Yeah, I think we're just getting to where this kind of updated version of HPC is starting to develop now that AI is now part of that story. And I know that I ran across a Udacity course that is being done, it's High Performance Computing by Georgia Tech, and I have not taken the course myself, but it's one of their nano-degree programs, and I think that would probably be a good place to start. And not everything, just to point out, I mean, you can have a smaller cluster that you can build and, you know, you can get several HPC servers or something, or put that, you know, your own solution and network them and try out with some of the software that's becoming available. And then, like I said, especially to a handle, that makes more sense if you're going to be running

• 45:11Chris Benson?

  models kind of around the clock. If you're very occasional, like I need a lot once in a while, then cloud providers are probably a more economical way to go, or some hybrid of in-between, of about a baseline of training that you're looking at, and you can set up a much smaller version in your organization than what a Fortune 100 company might be doing. And so, it doesn't all have to be at massive, tens of millions of dollars or more scale to be able to do that. As the cost of hardware has driven down, try something small, mess around with it, take these courses, there's new information coming online all the time if you're Googling, and see what the various vendors are offering, because a lot of great information comes by looking through the various vendor sites.

• 45:51Daniel Whitenack?

  Yeah, I'd also like to advertise a little bit. If there's anyone listening to the podcast who is in college or studying computer science, studying sciences or AI in college, and you're looking for an internship after college and you're kind of interested in this high-performance computing space. I mentioned that I did that internship at NCAR and that was just like, yeah, it was such an amazing time for me to be able to be hands-on with these just amazing supercomputers and getting to like really be hands-on with some of the latest technology on that front, even getting to participate in a high-performance computing

• 46:32Daniel Whitenack?

  conference and other things. So that internship program, it's called the SIParCS internship, and they have it every summer. It's still going on. So if any of you university students out there are looking for something to do, I'd highly recommend that, and I'll link that in the show notes as well. But it was really great to have this conversation, Chris. Things are changing so much and it's really nice to kind of orient myself with a little bit of this technology and excited to see where it goes and how you're involved with it over time. So thanks for indulging me and letting me interview you a bit today.

• 47:10Chris Benson?

  I appreciate it. I'm going to take this awkward guest hat off and put my co-host hat back on, which is a much more comfortable hat for me. So thanks for doing this, Dan.

• 47:19Daniel Whitenack?

  Yeah, definitely. We'll talk to you soon.

• 47:22Chris Benson?

  Thanks a lot. Bye.

• 47:26Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address. Get our weekly email, keeping you up to date with the news

• 48:11Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.