Exploring NVIDIA's Ampere & the A100 GPU

On the heels of NVIDIA’s latest announcements, Daniel and Chris explore how the new NVIDIA Ampere architecture evolves the high-performance computing (HPC) landscape for artificial intelligence. After investigating the new specifications of the NVIDIA A100 Tensor Core GPU, Chris and Daniel turn their attention to the data center with the NVIDIA DGX A100, and then finish their journey at “the edge” with the NVIDIA EGX A100 and the NVIDIA Jetson Xavier NX. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: NVIDIA Ampere Architecture In-Depth https://devblogs.nvidia.com/nvidia-ampere-architecture-in-depthNVIDIA DGX A100 https://www.nvidia.com/en-us/data-center/dgx-a100NVIDIA EGX A100 https://www.nvidia.com/en-us/data-center/products/egx-a100NVIDIA Jetson Xavier NX https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nxPractical AI – Episode #56 – Worlds are colliding - AI and HPC https://changelog.com/practicalai/56Practical AI – Episode #15 – Artificial intelligence at NVIDIA https://changelog.com/practicalai/15Learning Resources NVIDIA Deep Learning Institute https://www.nvidia.com/en-us/deep-learning-ai/educationDocker and Kubernetes: The Complete Guide https://www.udemy.com/course/docker-and-kubernetes-the-complete-guideUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  So now we have this new Nvidia DGX A100.

• 0:04Daniel Whitenack?

  Maybe you'll get one. I don't know if I'm gonna get one.

• 0:08Chris Benson?

  But we'll see, we'll see.

• 0:09Daniel Whitenack?

  Yeah, yeah.

• 0:10Chris Benson?

  Sadly, I'm not in charge of procurement and I am certainly not in charge of procuring one for my own personal use. We'll see.

• 0:17Daniel Whitenack?

  I'm guessing that my nonprofit's not gonna get one, but if you happen to be getting your DGX A100 and you'd like me to run my training on it, I would be more than happy to do some benchmarking for you.

• 0:31Speaker 02

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:49Speaker 02

  Linode makes cloud computing simple, affordable, and accessible. Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and skill you need to take your ideas to the next level. We trust Linode because they keep it fast and they keep it simple. Check them out at linode.com/changelog.

• 1:14Speaker 03

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM. Okay, take it away, guys.

• 1:40Chris Benson?

  Welcome to another Fully Connected episode where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up your machine learning game. So, welcome to the Practical AI Podcast. My name is Chris Benson. I'm Principal AI Strategist at Lockheed Martin. And with me as always is Daniel Whitenack, Data Scientist at S I L International. How's it going, Daniel?

• 2:06Daniel Whitenack?

  It's going great. It's a beautiful day outside. Hopefully I can take a walk after this. And I've been staring at my screen most of the day since like 7:00 a.m. So I'm ready for a walk around the block or something.

• 2:19Chris Benson?

  That sounds like a good idea. I am bleary eyed from screen time as well. So get outside and enjoy it, especially now that at least where I'm at in Atlanta, the worst of the pollen seems to be passed. So that's good. No more yellow cars from pine pollen everywhere.

• 2:36Daniel Whitenack?

  Yeah, that's rough. We need like an AI model that like takes in pictures of cars and people's driveways and like tells you whether it's safe to go outside yet because of the pollen levels or something.

• 2:48Chris Benson?

  There you go.

• 2:49Daniel Whitenack?

  I'm sure there's easier ways to do that, but um.

• 2:51Chris Benson?

  That's right. So we'll have to prep people next year. So we'll just ask everybody in the audience next year to to send us your images of your cars covered in pine pollen with a date.

• 3:01Daniel Whitenack?

  Yeah.

• 3:01Chris Benson?

  A date time attached to it and we will have a great project to go on this.

• 3:05Daniel Whitenack?

  Yeah, exactly.

• 3:06Chris Benson?

  So and before we get going, I know that you've been doing some training classes. How have those been going?

• 3:11Daniel Whitenack?

  They went, they went really good. It was interesting because normally.

• 3:15Chris Benson?

  Teaching by the way, not not taking.

• 3:17Daniel Whitenack?

  Yeah. So normally when I do like AI trainings in industry or like at conferences or something, obviously there's um, you know, normally like a whiteboard there. There's a lot of kind of uh, changes as the class goes on because there's a lot of easy kind of back and forth. So it was interesting to figure out the virtual dynamic with everybody being at home. I think actually it ended up having some benefits because it sort of forced me, like normally I kind of write things at the whiteboard. I'm able to like make changes as I go. But to be able to, or to have to sit down and work things out in a strict set of slides

• 3:58Daniel Whitenack?

  that I'm showing made me really think about like, what is the proper flow to explain this idea? And like, show certain things. So in that respect, actually, it was, it was a learning experience for me.

• 4:11Chris Benson?

  I bet.

• 4:11Daniel Whitenack?

  Because it helped clarify some of that logic and even in my own mind. So, but it went really well and the students. I had some great questions and went, you know, all the way from what is AI to convolutional layers and recurrent layers and training cool things. And so, yeah, it was a good time.

• 4:31Chris Benson?

  You know, I've always found that no matter what the topic you teach, it forces you to assess everything that you think about because you got to explain it to other people and answer all that. And you all, every time I've done that, I've learned so much more about whatever it was I was going to teach. So.

• 4:45Daniel Whitenack?

  For sure. Yeah. And people ask questions that just never have come into your mind, not because they're bad questions. It's just a different point of view. So it forces you to like backtrack yourself and kind of look at things from different directions. Yeah.

• 4:59Chris Benson?

  Well, that sounds interesting. I guess you probably looked out. You got through your training classes right before Nvidia this year at their GPU Technology Conference, GTC, made all their new hardware announcements and the things that go with that. So.

• 5:16Daniel Whitenack?

  And you know, it's interesting. I actually was spending the evening yesterday with my brother in laws who are living with us. So we aren't like social gathering yet, but my brother in laws are living with us right now while they're back from college. And so they don't work, they're not like in computer science or anything like that, but they are pretty heavy gamers. And they were both like, we started talking about some of this stuff and they had even seen the keynote from the Nvidia conference even before I started talking about, even though it was mostly AI related, it was like a, you know, already a sort of general meme that like the Nvidia CEO was like presenting all of this cool GPU stuff from his kitchen, which was like,

• 6:01Daniel Whitenack?

  you know, you could see the spatulas in the background and some very interesting like fresco above his oven and such. So it was really interesting in that sense that there was like,

• 6:14Daniel Whitenack?

  like people, even people that aren't in this space, like it made some impact on their life, which was really interesting.

• 6:22Chris Benson?

  You know, that's a great point. It's worth talking about. I mean, you know, we talk about Nvidia and Google and other major players in this space often because you can't really talk about AI in a lot of cases without talking about the biggest influencers. And in Nvidia's case, they were this gaming company and GPUs were originally to promote graphics and, you know, computer gamers, you know, were constantly using it. And now we're doing this in the AI space, you know. So any thoughts on how or insight into how that evolution came about and why we're using GPUs for all this stuff?

• 6:58Daniel Whitenack?

  Yeah, and actually also demand from the Bitcoin mining.

• 7:02Chris Benson?

  That's true.

• 7:03Daniel Whitenack?

  Space as well. So it's very interesting that like if you look at Nvidia's rise over time, of course, they existed for quite some time, but it's almost like, so they existed for a reason and they were really good at this reason. And then all of a sudden, like the things that they were good at became like the most important things in the world. That's how kind of how it seems. And then they just like, they were already there. So it's like they just exploded. So yeah, you're right. So if you think about like video gaming and that sort of thing, or like things you would want to do in video processing or graphics, for example, like you might want to apply a filter to

• 7:42Daniel Whitenack?

  some image or frame of a video, right? Like to darken it or to apply a gradient of color or, you know, something like this. And so you're essentially applying some operation to the pixels of an image, which are set up in a matrix and have like even some depth because there's a color dimension, right? So you have like this matrix of numbers and then you apply some operation on the elements of this matrix or really this volume, this input volume. In AI, of course, with convolutional layers, like you are doing almost that exact same thing because you're applying like a series of weights and bias and, you

• 8:26Daniel Whitenack?

  know, functions like activation functions to individual elements of a matrix or an input volume. But even in recurrent layers or like fully connected neural networks and that sort of thing, the types of networks that might be relevant to other things like text or just like general classification problem, even those take some input vector or matrix and just apply a series of weights to those, apply functions like activation functions like tangent and sigmoid and all of these in an element wise way. And so you're really doing the sort of matrix operations that graphics cards were always good at. And so

• 9:08Daniel Whitenack?

  it turns out that it's really good to use those sorts of graphics cards for those sorts of operations which are done in specifically in AI training. Of course, we're going to talk maybe about inference today too, but I think it came about because these are the sorts of things that happen iteratively thousands and thousands and millions of times when you do training for an AI model.

• 9:31Chris Benson?

  You know, that has to be the most accessible explanation to that evolution that I think I've heard anyone say. I think, I think you did it better than Nvidia actually says it. So that was well done.

• 9:43Daniel Whitenack?

  Well, they can pay me if they like. I mean, or they could send me a graphics card. That would probably actually be better. Nvidia, if you're listening, hint, hint, you know, Titan RTX. I won't even take the newest one. Don't even give me like an A100.

• 10:00Chris Benson?

  Give them the newest one.

• 10:03Daniel Whitenack?

  I don't, just small potatoes. Yeah, the, that $5,000 or what, I don't even know how much it is, Titan RTX.

• 10:11Chris Benson?

  Well, now that we have been pleading for free stuff, let's move on to some of the things that they announced, which many organizations around the world are going to be trying to evaluate and figure out how they're going to incorporate, buy into, and basically utilize this new hardware and the supporting software capabilities that go with it.

• 10:32Daniel Whitenack?

  Yeah, definitely.

• 10:34Chris Benson?

  So, you know, I guess one of the things to, to talk about here is even before we get to announcements are the types of GPUs that are currently available and what forums, you know, what kind of off brand GPUs are out there because Nvidia isn't the only player in the space. Any insight into that?

• 10:52Daniel Whitenack?

  Yeah, I mean, it's probably worth. Distinguishing a few things here, I guess first is like accelerators that are out there and types of GPUs that are out there. And also like access patterns to those, whether that be like locally or in the cloud or whatever. At least from my perspective, and I'm by no means an expert on this. On the like graphics card front, actually probably my, my brother in laws could do a better job, but. There has been a progression and most of the time you'll see like graphics cards referred to by some series of numbers and acronyms. So like recent ones have been like something like 1080 RTX or Titan RTX or something.

• 11:34Daniel Whitenack?

  So those are the graphics processing unit that you would like buy. And then you have to plug it into like some computer, right? So some people say like, okay, I'm gonna, I'm gonna develop AI models. And so I'm gonna buy a, like a computer, like a tower or desktop computer. And then I'm gonna buy a graphics GPU, like a, one of these RTX GPUs or something. And I'm gonna put it in like my PCI slot in my motherboard. And then when I do AI training, then I'm gonna like offload the training, some of those training operations to the. Graphics card or GPU that's input to my computer.

• 12:17Daniel Whitenack?

  So that's sort of a, one of the first ways you might think about doing this is like, I'm gonna do AI development. So I can buy a computer and then I'm gonna buy a graphics card and just put it in there. And a lot of those, of course, come from Nvidia. They make a lot of those cards, but there's off brand ones that kind of are similar to the models that Nvidia has. And then there's also like other brands that have their own style of graphics card and that sort of thing. Have you ever built or thought about building this sort of like workstation for your home to like sit by your desk or something?

• 12:57Chris Benson?

  I think I'm way too lazy to do that. At this point, I'd much rather go to a cloud provider if I'm at home and use what they've built. I've noticed that most of the people that had workstations specifically for their AI workflows seem to have moved off those in recent years, either in the cloud, or if they're big enough into more of a data center, or at least a, you know, like a workstation level, you know, where they're buying a workstation versus buying individual GPUs. It was one of the things I was thinking as you were just talking about that was we had one of our early episodes, which was episode 15, called Artificial Intelligence at Nvidia. We had Nvidia's chief scientist, Bill Dallie, on the show, and he absolutely schooled us.

• 13:39Daniel Whitenack?

  Yeah, definitely.

• 13:41Chris Benson?

  In the hardware, do you remember that? He really schooled us in the hardware.

• 13:44Daniel Whitenack?

  Much deeper than we'll go on this episode. So we definitely take a look at that. Yeah.

• 13:49Chris Benson?

  Yes. If you're wondering, and we asked him against other architectures, you know, and he went there and described it. So if that aspect of it, not just the Nvidia architectures, but how they compare to other things, I would encourage listeners to listen to that episode, and he will absolutely school you in the fundamentals there.

• 14:07Daniel Whitenack?

  Yeah, for sure. Cause there's not only, so like I described like the graphics card or GPU, which is what a lot of times what people think of when they think of GPU or accelerator in the AI world. They think of one of these GPUs or a series of them. But there's other options too. So there's like the TPU or tensor processing unit from, is it tensor processing unit or tensor flow processing? I don't know if they put the brand in there. I think it's tensor processing unit.

• 14:35Chris Benson?

  I believe it is.

• 14:36Daniel Whitenack?

  The TPU from the Google developed, which is another type of accelerator. But there's even other architectures out there other than CPU, GPU, TPU, there's FPGA and other things. And yeah, there's a lot of options out there. And like you said, there's options also. So I kind of described like, okay, if you're developing AI, you could just like create, just buy one of these computers to have at your desk. But there's also just like how other forms of compute have been commoditized via the cloud. There's easy access to. Cloud resources for GPUs too, and in all the clouds and in even special-built GPU,

• 15:19Daniel Whitenack?

  like cloud services like Paperspace and others. I know when I was looking around a while back for a project, I don't know if it's still the case, but I was trying to find like, what is the cheapest way to use a GPU in the cloud? And I ended up going with Paperspace. I don't know if it's the cheapest anymore. I do use like Google Colab, as I've mentioned a lot of times on the podcast. And of course, you can have access to a free GPU there. There's trade-offs because it's in a notebook and that sort of thing. But anyway, there's a lot of ways to access them which aren't buying a computer and setting it on your desk.

• 15:54Chris Benson?

  That's true.

• 15:55Daniel Whitenack?

  That's definitely nice.

• 15:57Chris Benson?

  So why don't we dive into some of the announcements that Nvidia made at GTC? As we're recording this, I think it was about roughly a week ago that they made the announcement, and it'll be another week as it rolls out. But I'll start us off with, they started in, I'm going to probably butcher the pronunciation, the Nvidia AMPIR architecture. Did they, did I get that right? I've read it, but I haven't watched the video to see.

• 16:22Daniel Whitenack?

  AMPIR?

• 16:23Chris Benson?

  There you go. How he's pronouncing it.

• 16:25Daniel Whitenack?

  I think in reference to like AMP in electronics, I think.

• 16:29Chris Benson?

  There you go. I didn't make the connection there. Okay.

• 16:31Daniel Whitenack?

  So I don't know. At least that's how I was saying it. So.

• 16:35Chris Benson?

  Yeah, I know that essentially this is what they've used to replace kind of the existing architecture and expand it. They're really focusing on, I think, a more realistic in the sense of kind of, I say cloud, but when I say cloud, I don't necessarily strictly mean cloud providers. I mean, if you were putting together a data center with a whole bunch of GPUs or GPU servers in it, you know, they're really focusing on not only the performance sides, but the usability, as I was reading through it.

• 17:04Daniel Whitenack?

  Yeah. And I think that what I was gathering also in talking with some other people about this is, so the generation before this latest one was focused more on the ray tracing elements, which is the RTX in a lot of these cards, which, to be honest, I'm not a big expert on ray tracing.

• 17:23Chris Benson?

  Nor am I.

• 17:24Daniel Whitenack?

  That has implications, of course, in graphics and that sort of thing. But it wasn't like a huge advance in terms of like the size and capabilities of the graphics processing unit itself. It was more of this kind of generation of additional ray tracing capabilities. Whereas this next architecture which they're releasing, which they're calling the A100 or the Ampere architecture, which includes this A100 card or GPU, that this is a fairly significant jump in the like size and capabilities of the graphics processing unit itself. I think part of that has to do with, I guess, the way that they've laid out the transistors

• 18:06Daniel Whitenack?

  and all of that on the substrate, that it's much more dense in my understanding. Yeah.

• 18:12Chris Benson?

  Am I recalling it was something like 20x performance improvement over the V100?

• 18:17Daniel Whitenack?

  Well, yeah, so it's 20x greater flops, which is like a measure of, actually, you probably are better versed in the acronyms, but this is like a common way to like measure the performance of like computers, like supercomputers and that sort of thing. So 20x greater flops for AI. Although they do give some benchmarks, which is pretty nice just for reference. And what I was looking at, they give some benchmarks for training BERT, large-scale language models, which we have an episode on BERT as well, if you'd like to learn more about that.

• 18:56Chris Benson?

  Yes, we do. We've mentioned it in several, actually.

• 18:58Daniel Whitenack?

  Yeah, it's good that we ended up having that conversation. But the BERT models are these very large language-related models, NLP models, that have just tons of parameters. And actually, these large language models have even billions of parameters now. I forget how many BERT has. But they give some benchmarks, both for the training and inference on speedups on training BERT. So on BERT itself, they're saying that above the V100, so the V100, if you go to like Google Cloud, or if you go to Paperspace or one of these platforms, at least right now, I think the the best GPU

• 19:41Daniel Whitenack?

  that you can get access to is called a V100, which is a previous generation. And it's it's pretty wicked fast. I mean, I've I've used this in a in a couple of projects and it's quite astoundingly faster than the sort of entry level GPU.

• 19:58Chris Benson?

  Yes. And it's the basis for the DGX line of servers as well.

• 20:03Daniel Whitenack?

  I think so.

• 20:03Chris Benson?

  Or was, prior to this release.

• 20:05Daniel Whitenack?

  Prior to this, yeah. And they're saying that there's a speedup between 3 and 6x in the training for the BERT large-scale training. And the difference between the 3 to 6x has to do with the precision of the floating point numbers that you're using in the model. So I'm stepping way beyond my bounds into like, computer science land, where I don't deserve to step. But but in the in the models, obviously, you have all these weights and parameters and the matrices that you're transforming in these models. And computers work with numbers and those numbers have to be represented, right, in some form, in some precision. You can't,

• 20:49Daniel Whitenack?

  like, if you have, if you're representing pi, you're not going to represent, like, all digits of pi. In pi, you're going to have to cut it off somewhere, right?

• 20:59Chris Benson?

  Yeah.

• 21:00Daniel Whitenack?

  This is having to do with that precision of the numbers. And you can actually, if you reduce the precision of how you represent numbers, you can sometimes speed up your your performance. And so that's what they're talking about there with that difference to 3x to 6x.

• 21:15Chris Benson?

  3 to 6x, yeah. And I think I'm looking at their inference, and I know they're they're saying it's a 7x speedup on inference. So it's substantial in that case.

• 21:25Daniel Whitenack?

  So they say, like, this card, this A100 accelerator.

• 21:30Daniel Whitenack?

  They bring up this idea of, what do they call it? The, I don't know if they're, if you say it meg, I'm thinking of like the fighter jet, but the multi-instance GPU, which is really intriguing idea.

• 21:42Chris Benson?

  Do you work for us now in that way?

• 21:44Daniel Whitenack?

  Yeah. They're saying it's multi-instance GPU, which in my understanding, are they saying like, you can basically treat the GPU as seven GPUs? Is that what they're saying?

• 21:54Chris Benson?

  So I was wondering that myself. And so a big topic that I spend my time at work is, is around multi-tenancy in your workflows and, you know, the accessibility of compute in those. And I was taking it in that way, but I'm not sure because they're a little bit ambiguous in the, in the way they use some of the terms. Um, and another one that I noticed is they, they talked about the, the need for no code changes. I'm, and I'm assuming that's coup de code changes in this case, but they weren't always as specific as, as they might've been in terms of their explanations here.

• 22:29Daniel Whitenack?

  I was wondering about that as well. Of course, there's certainly are ways to make changes like this transparent, but there's a change somewhere, right? It's just maybe at the abstraction level you're working with in TensorFlow or something. You don't have to make a change in TensorFlow, but in the underlying libraries, somewhere it seems like there, there's some type of change.

• 22:50Chris Benson?

  Yeah. It talks about on the multi-instance GPU, as I'm looking through that, it's talking about, uh, seven different isolated GPU instances, running different applications simultaneously.

• 22:59Daniel Whitenack?

  Yeah. So it seems like when they say seven X speed up for BERT large inference, and they have under their, in parentheses, seven meg. Yeah. Seven multi-instance GPU.

• 23:13Chris Benson?

  They're using them all?

• 23:14Daniel Whitenack?

  What I'm assuming that is meaning is they basically are running seven inferences in parallel on the seven GPU, which seems to be the same performance that they're indicating for as the V100. So for inference wise, it seems like, you know, the

• 23:36Daniel Whitenack?

  Change is that you're able to run things in this parallel way. Whereas like on a V100 or something, maybe you couldn't do that. And so there wasn't that that speed up. I'm making some assumptions here.

• 23:50Chris Benson?

  That's true. I know for a fact that there are folks at Nvidia that listen to the podcast. So hopefully if we are getting this wrong, they can.

• 23:56Daniel Whitenack?

  Yeah, clear us up.

• 23:57Chris Benson?

  Yeah. They can clarify for us and we'll come back at a later time on a later episode and say, we were wrong. We're happy to do that. So we're making the best of it.

• 24:06Daniel Whitenack?

  We were wrong. And if you send us a GPU, then we'll prove that we were wrong on our own local system.

• 24:13Chris Benson?

  You're back to begging. Oh, oh.

• 24:16Daniel Whitenack?

  But it seems pretty cool. I mean, I like the idea that if you've gone from a stage of training to inference, basically, whereas before, maybe you had this full, like powerful GPU that you were basically running inference on, but not like soaking up all of the goodness of the GPU and the compute. Here they're basically saying, okay, well, you can sort of parallelize the inference over that and still utilize this whole compute capability. But now you're just have this ability to split it up in nice ways. So I definitely think that's pretty cool. And, you know.

• 24:57Chris Benson?

  It's interesting with the parallelization of this. There was an image that I saw Nvidia had put out where they were kind of comparing the old architecture with the new A100 architecture. And they basically had, you know, one little server for the new that was the equivalent. They were showing, you know, rows of racks of servers in terms of its productivity. But it was definitely an impact. It was something that me and some folks I work with were passing around. And so, yeah, gotta, gotta keep up with times, I guess, if you're going to keep, keep driving forward on compute. Anything else on the architecture at large before we talk about DGXs or dive into the processors themselves?

• 25:35Daniel Whitenack?

  I think the, the one thing that you mentioned about like the speed up without code change, I think. They do introduce this new idea where, as people before had talked about floating point 16 and 32 numbers.

• 25:48Chris Benson?

  Yeah.

• 25:48Daniel Whitenack?

  Where again, these are having to do with the sort of precision with which you're representing numbers. They introduced this new idea of like tensor float 32.

• 25:59Chris Benson?

  I saw that.

• 25:59Daniel Whitenack?

  Which apparently with float 32, obviously if you have more digits, you can represent more numbers, right? There's like this kind of range, but you know, it's not as fast as using floating point 16 in some cases. So what they're saying is they're trying to balance the two, I think, in that they have a wider range of numbers they can represent in this representation, but with lower precision such that they can, you know, speed up training. So, you know, again, hopefully I've represented that well in terms of how they're thinking about it. There's an image of this on a blog post that um, we'll link in our show notes

• 26:38Daniel Whitenack?

  if you want to kind of understand how the floating point 1632 and tensor float 32 compare. But this is definitely a new representation on this chip that I don't think has happened on any other, other architecture yet. So that might be worth pointing out.

• 26:58Chris Benson?

  Yeah, totally. Another thing that we probably should mention from the architecture is that they've, they've gone to the new third generation for NVLink and NVSwitch.

• 27:08Daniel Whitenack?

  Oh, yeah.

• 27:09Chris Benson?

  And that manages the network scaling of how you're moving, you know, your data around through the chips and stuff. And I think that it's something like a 10 times bandwidth, if I recall, in terms of what it can do compared to, or it may have been 10 times more than PCIe generation four. If I, I think that was what it was that I was recalling reading.

• 27:29Daniel Whitenack?

  I'm going to get the number wrong, but they said there was like so many terabytes per some insanely small time. So it was like a bunch of data you could transfer back and forth very, very quickly via these links.

• 27:43Chris Benson?

  Absolutely.

• 27:44Daniel Whitenack?

  So the NVLink is, that has to do with communication of data between GPUs? Is that the idea? That's what I've always assumed. I don't have the opportunity too often to run on like, run my training on like 32 GPUs. So this is where I'm kind of getting to the edge of my understanding. But I did watch a YouTube video. And I think that's what they implied.

• 28:10Chris Benson?

  Is that like staying at a Holiday Express?

• 28:12Daniel Whitenack?

  That, yeah, I've stayed at a Holiday and Express.

• 28:14Chris Benson?

  There you go.

• 28:15Daniel Whitenack?

  Yeah, so I've watched the YouTube video. My understanding was like, because people also build these Bitcoin mining rigs, right? And they have all these GPUs on top and they're running all the time. And the way they do that is they basically connect a bunch of them to PCI slots on a motherboard. And to do that, they have these little adapters called risers that like come out of the motherboard. But apparently those are very slow in terms of communication between the GPUs.

• 28:44Chris Benson?

  Yeah.

• 28:44Daniel Whitenack?

  And PCI is is slow in that way. And so I, at least that's what they're implying that like in Vlink and some of these other things from Nvidia help facilitate that communication of data. And like you're saying, it helps scale out to like, now if you have 32 GPUs in your data center, you know, and you're trying to run some computation across them, you're going to need to have very quick communication.

• 29:11Chris Benson?

  Yeah.

• 29:12Daniel Whitenack?

  For like scientific applications or AI applications that are not just Bitcoin mining, which is just running operations, there's actually communication that's needed.

• 29:21Chris Benson?

  Yeah, if I recall correctly, and it's been a while since I've I've delved into this, back when they originally released the architecture, I believe that NVLink connects GPU to GPU. It gives you that interconnect between the two. And then that, you know, essentially that that mesh is something that NVSwitch thing connects at a higher level, combining the different NVLinks too.

• 29:43Daniel Whitenack?

  I see. So NVLinks GPU to GPU and NVSwitch. We'll call that now, but if listeners, if you know we're wrong, let us know and and we'll put a note in the show notes or something. That's good to make that connection.

• 29:56Chris Benson?

  Okay.

• 29:57Daniel Whitenack?

  Yeah, and of course, these, because they're connectable and scalable in this way, it seems like this is their new way of replacing what they did have in the DGXs, which the DGXs were the sort of boxes that they put in data centers, GPU data centers to like, scale up like a AI supercomputer of some type.

• 30:22Chris Benson?

  Correct, or a cluster of them.

• 30:24Daniel Whitenack?

  Okay.

• 30:24Chris Benson?

  Which is becoming more and more common. In the earlier days, you know, people would get like when the original DGX1 came out and it had eight GPUs in it and people would get that and that, that in itself, people were calling a supercomputer. And, you know, we talk like that's such a long time ago. It's only been a couple of years. But then they moved to DGX2 and then that was 16 and then they've actually scaled back. And in just a moment, let's talk about that.

• 30:58Speaker 03

  The change log is deep discussions in and around the world of software and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

• 31:08Speaker 04

  At the time drones were like predators and global hawks and military industrial. They were classified and super, you know, $10 billion things. And we had just built a drone with Lego pieces around the dining room table programmed by a nine year old. And it's like, okay, that should not be possible. You know, it's when, when a nine year old can do something that is classified, that literally export controlled as ammunition with Lego, with toy pieces, it was something important in this world has changed.

• 31:37Speaker 03

  Leaders like Devin Zougal from GitHub.

• 31:40Speaker 05

  In the like 10 to 15 year range or 20 year range, what I would really like is for, if you have like three 12 year olds hanging out and one of them's like, I want to be a firefighter. Another one's like, I want to be a lawyer. I want one of them to say, I want to be an open source developer.

• 31:54Speaker 03

  And innovators like Amel Hussein.

• 31:57Speaker 06

  I've yet to kind of see applications at scale that don't use multiple languages, that don't have just arcane stories behind why this weirdo thing exists, you know, like, all right, when you open this file, you're going to have to turn around three times and tap your nose once.

• 32:14Speaker 06

  Like it's just the most hilarious stories, you know, but applications are living, breathing, they have craft, that's normal. So I want to normalize weirdness because that's just how applications evolve over time.

• 32:29Speaker 03

  Welcome to the change log. Please listen to an episode from our catalog that interests you and subscribe today. We'd love to have you with us.

• 32:49Chris Benson?

  So now we have this new Nvidia DGX A100, which they've kind of broken the paradigm of their labeling. So they went from DGX one originally to DGX two, and now they've gone to DGX A100.

• 33:02Daniel Whitenack?

  Maybe you'll get one. I don't know if I'm gonna get one.

• 33:06Chris Benson?

  But we'll see, we'll see.

• 33:07Daniel Whitenack?

  Yeah, yeah.

• 33:08Chris Benson?

  Sadly, I'm not in charge of procurement and I am certainly not in charge of procuring one for my own personal use. We'll see.

• 33:15Daniel Whitenack?

  I'm guessing that my nonprofit's not gonna get one, but if you happen to be getting your DGX A100 and you'd like me to run my training on it, I would be more than happy to do some benchmarking for you.

• 33:28Chris Benson?

  Gotcha. I'll, I'll talk to my boss's boss's boss's boss and see, see what's possible.

• 33:33Daniel Whitenack?

  Yeah.

• 33:34Chris Benson?

  I'm there for you, my friend.

• 33:35Daniel Whitenack?

  I'm really into the mooching today. Yeah.

• 33:38Chris Benson?

  Totally. I got it. I'm good. We're there to support you. I'm there for you. But yeah, I mean, with this new architecture, it's much more performant, but they've actually cut the number of GPUs in the server back down to eight from 16. But it has the enhancements that we just talked about that are at the processor level architecturally.

• 33:54Daniel Whitenack?

  So it's interesting that they, they kind of cut that down, but they have this multi-instance GPU capability. So actually, they say you can run 56 applications.

• 34:06Chris Benson?

  GPU applications.

• 34:08Daniel Whitenack?

  Seven per GPU on the DGX. Yeah. And like you were saying, even though there's, there's fewer here, because the size increase of A100, they kind of showed this picture in the, in the keynote, which people can watch, but supposedly you can kind of reduce the size and footprint of your data center because you're doing more computation. Per box, per DGX than you were before.

• 34:36Chris Benson?

  Correct.

• 34:37Daniel Whitenack?

  And this is interesting. They were saying like, you know, each box, let's say, I think the price they said was like a million dollars. Right? So this is not what I'm going to be putting on my desk, but certainly within the range of like compute budgets for some companies. So like each one was that expensive, but you could do the same that you could if you spent previously like 11 million dollars on your data center. So like scaling wise, you can do more with less, I think is the, the idea.

• 35:09Chris Benson?

  Yeah. When I was originally looking at these announcements as they came out, I think one of the callouts here, and this architecture does start to address that. But I think people in organizations that can afford to get DGX systems and they do choose to invest in those. They underestimate what it takes to get productive with them. And so they kind of just think, ah, I can go buy a DGX and just everything's going to work out after that.

• 35:36Daniel Whitenack?

  And then all my training will complete in three days and I'm done.

• 35:39Chris Benson?

  Exactly. Nothing to do with it. But I think the challenge is when you're scaling up to one or more DGX systems. Then you are talking about an overall, I'm not just talking about a DGX architecture. You're talking about an overall systems and software architecture in your organization and specifically data architecture that can support. Moving a lot of data around through training in an organized way that flows in with your business processes. And that is a big challenge. And I think, and being able to make all that work in your own organization. Is where a lot of organizations are struggling. And I know Nvidia works hard to throw them a bone. They work hard. They recognize that. And there are a lot of tools that they put out there to try to help you through that process.

• 36:24Chris Benson?

  But I think this architecture has kind of accounted for some of those pain points of the past. And they're trying to make it easier to utilize, you know, in number of GPUs across multiple DGXs. Which is good because there are cases, there are very highly scaled cases where you might be doing. A lot of experimentation with like hyper parameter optimization. And you want to try just an insane number of different possibilities.

• 36:52Daniel Whitenack?

  Yeah.

• 36:52Chris Benson?

  When you're doing your training and have the ability, not just to train one time, but to train many, many, many times. And, you know, thousands or millions even.

• 37:00Daniel Whitenack?

  Yeah.

• 37:00Chris Benson?

  And I think they've understood that and that this architecture is starting to address that highly skilled use case.

• 37:07Daniel Whitenack?

  Yeah, I think that gets to the point of sort of, you know, maybe something that is on people's mind as they listen to this is like, why not just the cloud and like use GPUs in the cloud, which you can certainly do. So like you could, you know, if you wanted to run a thousand experiments to test all your hyper parameters, you could spin up a thousand GPU nodes in Google Cloud or Amazon or wherever. But if you're doing that at any sort of frequency or length, that's gonna, the bill's gonna add up pretty crazy fast on that. So if this is something that like a company actually wants to do and, you know, AI is central to their strategy,

• 37:51Daniel Whitenack?

  to their products, and they want to get that very best model, and they want to do that experimentation over and over and over again. If this sort of DGX system is capable of supporting, you know, the usability side of things like you're talking about, then they could run those over and over again as much as they are able to usability wise. And so I think that that that kind of gets to the point. For some people, like, you know, I keep joking that I'd love to have access to this, but I probably wouldn't, you know, just me, myself, since I'm the AI person doing a lot of the AI things on my team. And I don't have a team of 40 different people trying to run things all the time, then, you know.

• 38:36Daniel Whitenack?

  I'm pretty okay with using like a GPU instance in the cloud when I need it because I might run a training for 48 hours or even four days or something, but I do that not very often. And it's just me. But if you've got like a team of 40 people or you've got multiple teams throughout your organization and they all need to run that stuff, that adds up really, really quick.

• 38:58Chris Benson?

  It does. I have been pleased just in general, when you combine the advances in NVLink and Switch, when you combine that with the multi-instance GPUs that these A100s are at this point, the scalability technology, which without diving into it is called Melinox Connect X-6, if I'm saying that right, it's a nice blend of architectural considerations to get you there. And you know, we haven't even talked yet about advancements on the edge.

• 39:26Daniel Whitenack?

  Yeah.

• 39:26Chris Benson?

  And that is a huge, huge area at this point.

• 39:30Daniel Whitenack?

  I'm glad you bring that up because it's probably, even though you know, I may not get access to the sort of DGX system, I am thinking about various applications at the edge. And in fact, I had a conversation earlier today with another guy who's working on totally different stuff in manufacturing, but they're not a large company, but they do stuff at the edge, in the manufacturing setting with low-power devices already, like think like a Raspberry Pi and that sort of thing. But if you could bring the power of like this sort of GPU to like right to the edge to a machine where you're doing computer vision to detect like anomalies in your manufacturing

• 40:10Daniel Whitenack?

  process or something like that, that's a pretty major advantage. And that brings that sort of capability to those sorts of people that are working on smaller teams and have that specific use case for running AI at the edge. They have the Nvidia, so along with the A100, they have the EGX A100, which they're releasing, which seems to benefit from some of these things that we talked about with the A100, but they also talk a lot about security, security and an end-to-end, an encryption of AI models, encrypt all the things. And I have some ideas about why, you know, that may be important at the edge. But you have any thoughts on that?

• 40:50Chris Benson?

  Well, we live in a time where, you know, we've had so many episodes where we talk about malicious actors and they could be anywhere from, you know, state level all the way down to teenagers that are savvy and having some fun. And we're in a world now where you just can't really assume that you can put anything that's not secure out of the edge. And that doesn't have to be in the defense world. You know, where I live, that can be really anywhere, any industry at this point. So they have had, obviously, their previous kind of edge oriented offerings. And we like to, you know, they, you know, there's, there's the smaller scaled stuff that we like to play with. You know, they have the NX out now. They've had Nanos out the last couple of years and so like that. But as industry really gets serious about pushing inference

• 41:33Chris Benson?

  out to the edge and, and having it both widespread and pervasive, having kind of a comprehensive and sophisticated security model that they can deploy onto these platforms is, is pretty key. And, and I think that's really, at this point, it's no longer a specialty thing. It's now something we're all having to, to acknowledge.

• 41:52Daniel Whitenack?

  Yeah. Because like, if you think about products, some products that have come out over the past years, like if you think about a drone that's kind of come out, I think there's multiple drones now that have come out that have some sort of AI model running on them that, you know, does something like it follows you around or like whatever the thing is, does some, some operation, object detection or something. If you're thinking about releasing a product that has like this sort of edge GPU running inside of it, whether that be in a manufacturing sense or like, like the drone or, or robot sense or something, really the AI model that you're releasing with that is part of your IP, right?

• 42:36Daniel Whitenack?

  And you've spent hundreds of thousands of dollars into it. So you've got like the malicious actor side of thing, but you've also got the fact that like, oh, if I buy a cool thing to strap on my manufacturing machine that has one of these GPUs in it, and it's like doing something sophisticated, well, if they're giving me the model in this product that I'm buying, why don't I just like unscrew the hatch and like plug, plug my computer in and just take the model off of it. And now I don't have to pay them for that product anymore. Right? So we've gotten to a point where the actual AI model is, is a piece of IP and is, and is extremely valuable. So you wouldn't want

• 43:19Daniel Whitenack?

  Like your client or your, you know, competitor especially to just be able to buy one of your products, unscrew the thing and like, you know, copy, you know, CP my model. P B from, from the machine over to their machine and then they've rid of themselves of their need for buying your product. Right? So, um, yeah.

• 43:43Chris Benson?

  I was just gonna say, it's funny. I've noticed this a lot lately and that when we talk about the fact that you're now seeing models being deployed to the edge, you know, just in massively parallel, deeply pervasive in whatever your business is, you know, as you know, I have a daughter who's young. We just went through a birthday and the toys that you can buy these days are now incorporating this stuff in it is you can actually buy toys that have convolutional neural networks in them. You can buy them that have NLP capability. And I think that's the moment where I find myself surprised because we're so used to talking about it in these kind of business oriented contexts.

• 44:20Daniel Whitenack?

  Yeah.

• 44:20Chris Benson?

  But then, you know, that's also someone else's business is to make these toys and yeah. And I, you know, I keep being surprised at these toys that she unwraps and they have these capabilities. Of all people, I should not be surprised, I suppose, but I am just to see it in that context.

• 44:35Daniel Whitenack?

  Yeah. Well, and especially in that case, like at least depending on the age of the child, you know, it would be important for that AI model to run offline on the device. And like, let's just keep that thing offline and it's good if it acts as a toy, but let's not connect it to the wild west of the internet just yet. So yeah, I definitely see, you know, you'd want to run that sort of model at the edge itself and upload it to the device. I think the other thing I wanted to mention is, um, so I was going, you mentioned the nano. So if people are thinking about like Raspberry Pi and this gets down maybe where it does bring some accessibility to a lot of people. Um, so there's like Raspberry Pi devices, which are like single board computer devices, which have been of course wildly

• 45:20Daniel Whitenack?

  popular, but Nvidia released a.

• 45:25Daniel Whitenack?

  Jets in Nano, which is like a single board computer with a little GPU on it. And I was actually thinking about getting one of those, but I don't know if it was in this series of releases or just very recently, they released this Xavier NX, which is like a next greater version of this. They actually call it like a little AI supercomputer and it's, it is a single board computer. It's got like something like a 10x computer or something like that of the Nano. And so when I was going to get that one, I just ended up getting the other because it seemed pretty awesome. And I think that a couple of things struck me about this. One is I'm always trying to think of, you know, like, for example,

• 46:09Daniel Whitenack?

  the, the cases that we work with and the people that work in our organization around the world, of course, work in a disconnected setting a lot of times because they're out in the field. And of course, a lot of people around the world don't have internet. But then also, you know, we're not flowing with money. So like, what is a way to like get things running at the edge in a reasonable way, in a disconnected offline way, but also at a cost effective way. And I find it really interesting that some of these things are coming out that have sort of a GPU capability and the Xavier NX, it's interesting that it's, it's got the GPU and you can run inference on it, but you can actually update your models

• 46:54Daniel Whitenack?

  as well. So they talk about doing transfer learning, which is like an update of a model. So you're redoing some of the training, maybe you're training some of the layers or you're training additional layers that you add on to your model. So I'm really curious when this comes in. I actually, it should be coming in today. So I'm kind of watching out my window right now.

• 47:13Chris Benson?

  Yeah, I'm going to reveal before we started the episode, when, when Daniel and I were talking, he's waiting, he's watching out the window for UPS to show up.

• 47:22Daniel Whitenack?

  I'm stationed right by my front window.

• 47:23Chris Benson?

  We might possibly get a package opening here on air as you know, with that new NX.

• 47:28Daniel Whitenack?

  Fingers crossed. Yeah, um, so I'm curious to kind of try, I, what I want to try actually is just to like, start with a small model and see like how the training compares to like in some like with a better GPU and paper space or something. And then like try it all the way up to like, how far can I push the training on the, on the NX? Can I actually train like, how big of a model can I train from scratch on it? And then like, how big of a model can I, do you transfer learning on? Cause yeah, I, I find that incredibly interesting. The other thing that they talk about with the NX is cloud native things at the edge. And I know both you

• 48:10Daniel Whitenack?

  and I are really big fans of Docker. So I, I find it interesting where as before, I didn't see them emphasize a lot of things about using Docker at the edge to run like AI related workflows. And now they're saying, well, this is how you should do it in this device. I find that really interesting.

• 48:27Chris Benson?

  Yeah. And not only Docker, but Kubernetes as well. It's, you know, I mean, uh, and we've talked, I know we've talked about this on other episodes when we were hitting slightly different topics, but we really, this whole kind of AI revolution that's happened over the, especially if you're looking at the last three to five years, we really, really benefited from what had the revolution that had just swept through the software development world, um, and software systems deployment world out there. And that Docker and Kubernetes became the systems to build on. And we landed on top of that and just took that over. So it's really good to see all of the hardware, whether you're talking about, you know, the lower end GPUs such as, you know,

• 49:11Chris Benson?

  the, the Nano that you talked about, uh, all the way up to, you know, the latest here, this, you know, DGXA 100, all using that same architecture. And so if you learn at one place, you can use it from the most scaled down to the most scaled up version. And you can use it in the data center and you can use it in the edge. And that is a wonderful, wonderful thing that we've inherited.

• 49:31Daniel Whitenack?

  Yeah, I totally agree. I, I've really enjoyed talking about all of these things. I've got a lot to learn on all of these fronts. And I, if you're thinking like, oh, all of this CPU stuff and like accelerated AI is very new. Don't be afraid. I there's still like I I didn't come from a sort of computer science background, but there are there is tooling that's accessible for you to, you know, get into some of these topics. And one learning resource, a lot of times in these fully connected episodes, we like to mention learning resources. So actually Nvidia themselves have what they call the, I think it's the Deep

• 50:10Daniel Whitenack?

  Learning Institute, Nvidia Deep Learning Institute. And they have a series of of courses that talk about everything from like getting started with AI on the Jets and Nano. That's the little single board guy that we were talking about all the way to, you know, more advanced topics with high performance computing, high performance computing with containers. They talk about various GPU accelerated frameworks like Rapids and AI in the data center and all sorts of topics. So if you're interested in this sort of accelerated AI topic, you know, you might check that out. We'll definitely link it in the in the show notes as well. I know I have a lot to learn there myself.

• 50:50Chris Benson?

  So I'm going to go slightly off topic, but I and I it just occurred to me as we were talking about this. For the learning resource I'm going to throw out there, it's going to be one that a friend of mine mentioned just earlier today that he has utilized. For those of you who may be familiar with the learning site Udemy, U D E M Y dot com, there is a course on there called Docker and Kubernetes, the Complete Guide. It's not expensive, especially there's a lot of coupons. You can get it at a very low price, like $10, $12, $13. And so this person had gone through that course and was like halfway, two thirds of the way through and just thought it was fantastic to

• 51:31Chris Benson?

  ramp up on it. So given that recommendation, I'm going to recommend that to everybody and we will put a link in the show notes because if you're going to be in the AI world, it really pays to understand Docker and Kubernetes well.

• 51:43Daniel Whitenack?

  Awesome. Yeah, well, check those things out. Reach out to us on our Slack channel or on LinkedIn or Twitter with any questions or thoughts that you have. And I hope that this has been a fun episode. It has for me.

• 51:55Chris Benson?

  It has been.

• 51:56Daniel Whitenack?

  We will see you next week. See you, Chris.

• 51:59Chris Benson?

  See you later. And I apologize to the Nvidia people who are going, oh, my gosh, those guys, they need to know more about it.

• 52:05Daniel Whitenack?

  We accept feedback and in the show notes and everything is on GitHub. So you can submit a PR. So yeah, okay, feedback. Welcome.

• 52:15Chris Benson?

  See you next time, Daniel.

• 52:16Daniel Whitenack?

  Bye.

• 52:21Speaker 03

  Thank you for listening to Practical AI. We appreciate your time and your attention. Word of mouth is the number one way people find new podcasts. If Practical AI has helped you on your AI journey, please do tell a friend. Hey, they'll thank you later. Special thanks to Brake Master Cylinder for the beats and to our awesome partners for their support. Shout out to Fastly, Linode, and Rollbar. If you and your organization would benefit by speaking directly to the AI community, you should sponsor Practical AI. Podcast advertising is highly effective and we would love to work with you. Head to changelog.com/sponsor to learn more. That's all for now. We'll talk to you again next week.