Artificial intelligence at NVIDIA

NVIDIA Chief Scientist Bill Dally joins Daniel Whitenack and Chris Benson for an in-depth conversation about ‘everything AI’ at NVIDIA. As the leader of NVIDIA Research, Bill schools us on GPUs, and then goes on to address everything from AI-enabled robots and self-driving vehicles, to new AI research innovations in algorithm development and model architectures. This episode is so packed with information, you may want to listen to it multiple times. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean is simplicity at scale. Whether your business is running one virtual machine or ten thousand, DigitalOcean gets out of your way so your team can build, deploy, and scale faster and more efficiently. New accounts get $100 in credit to use in your first 60 days. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogFeaturing: Bill Dally – Website https://www.linkedin.com/in/billdally/Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: NVIDIA Research https://www.nvidia.com/en-us/researchNVIDIA Deep Learning Accelerator (NVDLA) http://nvdla.orgCUDA https://developer.nvidia.com/cuda-zoneAI-Powered Self-Driving Cars https://www.nvidia.com/en-us/self-driving-carsProgressive GANs https://research.nvidia.com/publication/2017-10_Progressive-Growing-ofOptical Flow and Deep Learning https://research.nvidia.com/publication/2018-02_PWC-Net%3A-CNNs-forTensorRT https://developer.nvidia.com/tensorrtDeep Learning Super Sampling https://developer.nvidia.com/rtx/ngxNVIDIA DGX2 https://www.nvidia.com/en-us/data-center/dgx-2NVSwitch and NVLink https://www.nvidia.com/en-us/data-center/nvlinkNVIDIA Deep Learning Institute https://www.nvidia.com/en-us/data-center/nvlinkUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Beta for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log. This episode is brought to you by Digital Ocean. They now have CPU optimized droplets with dedicated hyper threads from best in class. Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application image. This gives you immediate access to Python three, R, Jupiter notebook, TensorFlow, side kit and PyTorch. Use our special link to get a hundred dollar credit for Digital Ocean and try it today for free.

• 0:45Speaker 00

  At the do.co slash change log. Once again, do.co slash change log.

• 1:03Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community, follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:29Chris Benson?

  Hey there, welcome to another episode of the Practical AI Podcast. This is Chris Benson. I'm an AI strategist and my co-host is Daniel Whitenack, a data scientist. We have a real treat in store for you today. We have a special guest that we have have looked forward to having on the show for a long time now, and I am super excited about this episode. So that guest is Bill Dally, who is the chief scientist and senior vice president of research for Nvidia. He is also a professor at Stanford University. Welcome very much, Bill. It's great to be here. And how's it going today, Daniel?

• 2:07Daniel Whitenack?

  It's going, it's going great. I'm excited to talk to Bill. I'm of course, a huge fan, as everyone is, of everything Nvidia is doing in this space. So I'm excited to hear more.

• 2:18Chris Benson?

  Yep. So the genesis for this episode came back earlier this year in March. I was at the Nvidia GTC conference in Silicon Valley, and I got to attend a small group session called AI for Business CXO Summit. And in that, the Nvidia CEO, Jensen Huang, was kind of in a small group environment. And it was just an amazing amount of wisdom that I got. And I was thinking, and as I sat there, that was very, very business oriented in a lot of ways. But I kept thinking, if we had Nvidia's chief scientist come on board to talk us through kind of what Nvidia does, but but give it to us as practitioners of neural

• 2:59Chris Benson?

  network technology and other AI technology, that would just be amazing. So Bill, thank you so much for coming on board. I really appreciate it.

• 3:07Speaker 03

  Hey, very welcome.

• 3:08Chris Benson?

  So I wanted to real quick ask if you could just give us a little bit of background. I mentioned that you were the chief scientist at Nvidia and a professor at Stanford. Could you tell us just a little bit about yourself before we launch into questions?

• 3:18Speaker 03

  Sure. So I'm, you know, sort of a hardware engineer who's been working on both hardware and software for AI in recent years. My first experience with neural networks was in the 1980s, when I took a course from John Hopfield at Caltech and was building Hopfield nets and things like that. I was on the faculty at MIT for 11 years, where I built a research group that built a number of pioneering supercomputers, collaborated with Cray on the design of their Cray T3D and T3E, and then moved to Stanford in 1997, where I continued to lead research on high-performance computing and special-purpose processors for numerous tasks, including graphics. I first got involved with Nvidia in 2003, when I

• 4:02Speaker 03

  was hired as a consultant to help with the, what was called internally, the NV50, became the G80 when it was announced, and in particular, to help on the extensions to the G80 that enabled CUDA, the ability to run general-purpose computing programs on GPUs. And I really got to like the folks at Nvidia, and particularly Jensen. And he convinced me to join full time in 2009. So since 2009, I've been building Nvidia Research, the research organization at Nvidia and myself, doing research on numerous topics. Most recently, on some of the path planning algorithms for self-driving cars and on very efficient AI inference.

• 4:38Daniel Whitenack?

  That's awesome. Yeah, that's an amazing background and I'm sure, yeah, I mean, it sounds like you joined Nvidia at a really exciting time. Of course, things have really kind of exploded in a good way for them. And I'm sure it's a lot of excitement and thrills being at the center of that.

• 4:56Speaker 03

  Yeah, it's a really fun place to be.

• 4:58Daniel Whitenack?

  Awesome. Yeah. So I was wondering, from my perspective, kind of growing up, the context in which I heard about Nvidia was kind of Nvidia processing and gaming, which kind of led to the rise of the GPU. I was wondering if you could speak a little bit to how and why that transition into this very AI-oriented approach that Nvidia is taking now and kind of comment on how that evolution occurred and how you see it from your perspective?

• 5:28Speaker 03

  Sure. So Nvidia's roots are really in graphics. Gaming is one aspect to that, but we've also always done professional graphics. And if you think about what the graphics problem is, it's basically simulating, you know, how light bounces off of a scene and appears at your eye or at a camera. And doing that simulation, basically rendering the scene, shading each pixel, is a very computationally intensive task, and it's a very parallel task. So GPUs evolved to be very efficient, parallel computers with very high computational intensity. And it turns out, a lot of other problems have this nature of having a lot of computational intensity and being very parallel. And so, early on, probably in the

• 6:09Speaker 03

  early 2000s, people started trying to use GPUs for tasks other than graphics. It's sort of a movement called GPGPU, general purpose GPUs. And around the same time, I was leading a project at Stanford, what we called Stream Processors, which actually wound up developing the right set of programming tools to program GPGPUs. We were developing a language called Brook. The lead student on that project, Ian Buck, graduated, got his PhD, came to Nvidia and evolved, along with the people in Nvidia, including John Nichols, who was heading the computer architecture group at the time, evolved Brook into CUDA. And that basically made it very easy for people to take. The huge number of arithmetic units who are in GPUs and their ability to

• 6:50Speaker 03

  to execute parallel programs very efficiently and apply them to other problems. And so at first, they were applied to high-performance computing problems. And GPUs have continued to be very good at that. We currently provide the arithmetic for the number one supercomputer in the world, Summit at Oak Ridge National Laboratories. And they've been applied to things from oil and gas reservoir modeling to simulating more efficient combustion engines, to simulating how galaxies collide. All sorts of high-performance computing problems, predicting weather, climate change, stuff like that, are now done on GPUs. So it was very natural, since we basically now had the platforms. This is, we announced CUDA in 2006.

• 7:34Speaker 03

  A few years later, a substantial fraction of all the large supercomputers being built were based on GPUs. It was very natural that when other very demanding problems came along, that people would apply GPUs to them. And so if you look at deep learning, and particularly the training for deep learning, it's a very computationally intensive problem. When this was first started to be done, it was taking weeks on the fastest GPUs we had. And it's very parallel. So it was a perfect match for GPUs. And so early on, we saw this match and applied GPUs to that. For me and for Nvidia, the start really came when I had a breakfast with my Stanford colleague, Andrew Ng, and I think it was probably in 2010 or early 2011.

• 8:19Speaker 03

  And at the time, he was at Google Brain and was finding cats on the internet by building very large neural networks running on 16,000 CPUs. And when he described what he was doing to me, I said, you ought to be doing that on GPUs. And so I found somebody in Nvidia Research, a guy named Brian Catanzaro, who now runs our Applied Deep Learning Research Group. At the time, he was actually a programming language researcher, but he was interested in deep learning and had the right background knowledge. And his assignment was to work with Andrew and move Andrew's neural network for finding cats to run on GPUs. We were able to take what took 16,000 1000 CPUs and run it on, I think it was 48 GPUs in, I think even higher performance than he was getting.

• 9:03Speaker 03

  And the software that came out of that turned into CUDA DNN, on top of which we basically ported just about every framework there is. Now, the other thing that happened is the first, the GPUs we had at that time, which were, you know, our, it was right around our Fermi to Kepler transition, weren't originally designed to do deep learning. They were designed to do graphics and high-performance computing. And so they had, you know, good 32-bit floating-point performance, good 64-bit floating-point performance. But it turns out what you want for deep learning training is FP16. And what you want for deep learning inference is int8. And they weren't actually particularly good at either of those. So as we learn more about what deep learning needed, our subsequent generations of GPUs have been specialized for deep learning. We've added support for FP16 for training.

• 9:46Speaker 03

  We've added support for int8 and int4 and int1 for inference. And we built tensor cores, which are special-purpose units that basically give us the efficiency of hardwired deep learning processors like the Google TPU, but without giving up the programmability of a GPU. So while the original GPUs were good at deep learning, now that we've gotten more experience with deep learning, learned what it really needs and have specialized and optimized GPUs for that, the GPUs today, especially Volta and Turing, are really great at deep learning.

• 10:16Daniel Whitenack?

  Yeah, that's awesome. I know I was just kind of trying to so-called that up. There's so much context and great information that I know I didn't, I wasn't aware of before. For example, you know, the evolution of CUDA and how it came from this Brook language that you mentioned and, you know, how the classifying of cats fit in and all of that. I don't know, were you aware of a lot of that, Chris? A lot of that's great new context that I wasn't aware of.

• 10:43Chris Benson?

  Yeah, I mean, he took topics that I now would consider a shallow understanding of up until at this point and went deep, which is fantastic. So be careful, Bill, because we have a whole bunch more questions for you. We're going to dive deep into some of these things you're telling us about.

• 10:58Speaker 03

  Okay.

• 10:58Daniel Whitenack?

  Yeah, so in particular, I know you mentioned a lot of things that I would love just a little bit of clarification on for those in our. Audience that maybe maybe new to them. So you mentioned kind of the evolution of of CUDA. You also mentioned, you know, how GPUs were were kind of integral to this the scaling of the of the deep learning training and all of that. I was wondering if we could just kind of take a step back and from your perspective, kind of get get your explanation of, you know, what a what a GPU is generally, why why it's useful for for deep learning in particular, and how CUDA fits fits into that, what it is like, what what that interface looks like today?

• 11:43Speaker 03

  Yeah, so a GPU generally is just a very efficient parallel computer. You know, Volta has 5120, what we call CUDA cores, which really means 5120 separate arithmetic units could be operating in in parallel. And coupled to that is a very efficient system for supplying data to those units and and accessing memory. And so, you know, for any problem that's very parallel, they are orders of magnitude more efficient than CPUs. CPUs, in contrast, are optimized for single-thread performance and for very low latency. But to do that, they wind up spending enormous amounts of energy, reorganizing your program on the fly to schedule instructions around long latency cache misses, right?

• 12:25Speaker 03

  So if you try to access memory and you're lucky, you get a number in three clock cycles. If you're not so lucky, it might be 200 clock cycles. And so they've got to, you know, do a lot of bookkeeping to to work around that uncertainty. The result of that is a huge amount of energy that's spent and therefore performance and energy efficiency that's orders of magnitude less than a than a GPU. A GPU takes advantage of the fact that if you have a very parallel program, you can hide any memory latency with more parallelism. You work on something else while you wait for the data to come back. So they wind up being extremely efficient platforms for tasks like deep learning, where you have many parallel operations that can be done simultaneously before you get the results of of one of them back.

• 13:05Daniel Whitenack?

  And that's like for the the the matrix type operations that you're talking about and and also the kind of iterative training processes, is that is that right?

• 13:15Speaker 03

  Right, so you know, at the core of deep learning are convolutions and and matrix multiplies. And in fact, you can turn the convolutions into matrix multiplies through a process called lowering. So fundamentally, if you can do a very efficient matrix multiply, you can do really well at at deep learning. And GPUs are very good at doing those matrix multiplies, both because they have an enormous number of arithmetic units, because they have a very highly optimized memory and on-chip communication system for keeping those arithmetic units busy and occupied.

• 13:47Chris Benson?

  So that that is really a great explanation, and that's helping me a lot. I would like to understand, beyond just Nvidia's GPUs, those of us, you know, that are out here kind of consuming information in the space, are always hearing tons of other acronyms, and if you and, you know, CPUs, TPUs, ASICs. If you could explain to us a little bit what's different about a GPU from those other architectures that are out there, and what are some of the advantages and disadvantages? You know, why is it that Nvidia is able to lead the way with this GPU technology that you've been bringing us for the last few years?

• 14:21Speaker 03

  Sure. So I already mentioned some of that by comparing CPUs and GPUs. A CPU, a central processing unit, like, you know, an Intel, you know, Xeon, or, you know, AMD's latest parts, is optimized for very fast execution of a single computational thread. And as a result of that, it spends an enormous amount of energy rescheduling instructions around cache misses. And as a result, winds up burning something on the order of, you know, a, you know, a nanojoule per instruction, where the actual work of that instruction maybe only takes 1% of that energy. So that you can think of them as being 1% efficient. GPUs actually spend more than half of their energy doing the payload arithmetic on computational

• 15:04Speaker 03

  intensive problems. So they are many times more efficient than CPUs at that. Now, CPUs have vector extensions that try to get some of the efficiency of GPUs. But if you look at the core CPU, they're extremely inefficient, but very good at, you know, doing a single thread. If you don't have any parallelism, you need the answer quickly, a CPU is what you want. If you've got plenty of parallelism and you can hide your memory latency by working on something else while you're waiting for that result to come back from memory, then a GPU is what you want. Now, you mentioned also TPUs and ASICs. Well, the TPU is a type of ASIC, right? It's an application-specific integrated circuit. In this case, it's the application it's specific for is doing matrix multiplies.

• 15:46Speaker 03

  The Google TPU, especially the TPU1, which they just had an article in CACM about, is a big unit that basically has a systolic array to multiply two matrices together. And it's extremely efficient at that. And so all you need to do is multiply matrices, it's very hard to beat a TPU. Now, the approach we've taken with our latest GPUs is to put tensor cores in them. Now, what tensor cores are, are little matrix multiply units. They're very specialized to multiply matrices together. The difference is by specializing by adding a unit to a general-purpose processor, we get the efficiency of that specialization without giving up the programmability of the GPU. So if you need to write a custom

• 16:28Speaker 03

  layer to do, you know, a mask because you're doing, you know, a pruning and have a sparse set of weights, or if you need a custom layer to do a new type of nonlinear function that you're experimenting with, or you want to do some type of concatenation between layers that is a little bit different, it's really easy to write that in CUDA, program it on the GPU, and it will execute it extremely well with all the efficiency of the hardwired matrix multiply units coming from the tensor cores. Whereas in the TPU, you have that efficiency, but you don't have the flexibility. You can only do what that, you know, one unit has been, has been, you know, designed and hardwired to do. Now, the advantage of that is it's about the same energy efficiency, right? So when you're not using the other features

• 17:10Speaker 03

  of the GPU, you're not paying for them. They don't burn any energy, but they are sitting there using up die areas. So the TPU costs a little bit less to manufacture because you don't have all of that general-purpose processor sitting around it. But what you give up for that is the flexibility of being able to support new deep learning algorithms as they come out. Because if it, if it, those algorithms don't match what the TPU is hardwired for, it can't do it.

• 17:34Daniel Whitenack?

  Yeah, and as we've, as we've seen, the, the industry isn't moving very fast at new, new neural network architectures, right?

• 17:42Speaker 03

  They're coming up every day. I mean, it's hard to keep up with with all the papers on archive.

• 17:46Daniel Whitenack?

  Yeah, it is definitely. We try a little bit on this show, but we're all constantly falling behind.

• 17:52Chris Benson?

  So, quick follow up. In that case, based on the fact that you have the tensor cores in the GPUs, it's unlikely that Nvidia then would likely go to, you know, some sort of ASIC architecture or something else like that, since you've essentially have already accounted for that value in your GPU architectures. Is that a fair statement?

• 18:10Speaker 03

  Actually, not. We actually have our own ASIC-like architecture as well, in that we have something called the Nvidia Deep Learning Accelerator, the NVDLA, which we've actually open sourced. If you go to NVDLA.org, you'll see our webpage where you can download the RTL and the programming tools and everything else for what is actually a very efficient, hardwired neural network accelerator. And we use the NVDLA ourselves in our Xavir chip, which is the system on a chip that we have for our self-driving cars. The Xavir has a number of ARM cores of our own design. It has basically a tenth of a Volta GPU. It's, you know, 512 CUDA cores rather than 5120. And then it has the NVDLA as well as a computer vision accelerator.

• 18:54Speaker 03

  Because in embedded processors on the edge, that area efficiency is important. We don't want to give up the di area for doing, you know, deep learning entirely on the GPU. Now, there's still an awful lot of GPU performance on Xavir. It's, you know, over 10 terops on the CUDA cores. But there's also another 20 terops on the deep learning accelerators. So you wind up being able to support very efficiently, you know, large numbers of inference tasks on that. So we're actually doing it both ways. For the embedded applications, we have a hardware deep learning accelerator. For both inference and training in the data center, after considering all options, we have decided it's just much better

• 19:39Speaker 03

  to put the efficient tensor cores onto a programmable engine rather than building a hardware accelerator.

• 19:45Daniel Whitenack?

  So you've mentioned, and this is a great lead in, you mentioned kind of a variety of fronts on which Nvidia is working. And you've also mentioned a desire that you guys have to keep things programmable and easy to interface with and customize. One of the things that I've definitely seen is that Nvidia is definitely making contributions, not only on the hardware side, but on kind of on the front of helping users be able to interface with all sorts of these new types of hardware. For example, I see, you know, like Nvidia Docker and I see things related to Kubernetes

• 20:27Daniel Whitenack?

  and Nvidia working to kind of help people both program their hardware, but also access and manage and orchestrate things. I was wondering if you could, if there's anything you want to highlight on that side and mention, you know, where the different areas that you see Nvidia working on that are really exciting, maybe not on the hardware side, but maybe on the orchestration or software side.

• 20:52Speaker 03

  Yeah, no, we actually do research on deep learning that spans the gamut from, you know, fundamental deep learning algorithms and models, training methods, to tools that make it easier for people to use deep learning all the way up to the hardware. The stuff that I'm actually most excited about is some of the work on fundamental models and algorithms. We, for example, right now have the world's best neural network for doing optical flow, which is a really nice hybrid of classical computer vision and deep learning, because we've applied a lot of what's been learned over 30 years of doing optical flow the old way, but then have built that around a deep learning approach and we get the best of both worlds. We also have done enormous amount of research on generative adversarial networks.

• 21:35Speaker 03

  We developed a method of, we're the first people to train high resolution generative networks. In the past, you know, basically you just had too many free variables. If you tried to train a GAN to build a high resolution image, it would just get confused and never converge. We applied curricular learning where we trained the GAN first to do low resolution images. Images. Once it's mastered that, we then increase the resolution progressively. We call it progressive GAN. And we're very successfully able to generate high resolution images. This has been applied to numerous tasks. We've also been able to build coupled GANs where we can use them to transfer style.

• 22:15Speaker 03

  So for example, if we have a bunch of images in daylight, good weather, we can change those to images at night or images in the rain or images in the snow. And this lets us augment data sets for self-driving cars. We can also use these GANs to generate medical data sets, being able to, you know, take, you know, for example, brain images and tumor images and combine them in various ways to build larger training sets than you could get by just using the raw data. And then a combination of the real data and these synthetic images winds up giving you better accuracy than the one alone. So that works for exciting. We also have a number of tools. You mentioned our Docker platforms. We also have a tool called TensorRT,

• 22:56Speaker 03

  which optimizes neural networks for inference to we get much more efficient execution on our GPUs than if you simply naively mapped the networks on there. And so across the board, we've been trying to build the whole ecosystem so that somebody who has a problem can and draw from our collection of algorithms, they can draw from our tools, and then ultimately run it on our hardware and get a complete solution for their problem.

• 23:20Daniel Whitenack?

  How do you, how do you keep all of those wheels turning as the, as the VP of Research? There's a lot of different areas spanning all the way from hardware to software to tooling to AI research. I'm sure it's exciting, but a lot going on.

• 23:36Speaker 03

  Yeah, so I fortunately don't have to keep them all turning myself. I'm responsible for NVIDIA Research, which is an organization of about 200 people. And we do research on topics ranging from circuit design to AI algorithms. And, you know, basically what we do is we hire really smart people and then we try to enable them to take all the obstacles out of their way, get them excited about the important problems. And, and the, you know, the objective of NVIDIA Research is, is to do two things. One is to do research. There are a lot of Corporate research labs that actually don't do research. They wind up really doing development because they get pulled in too close to your various product groups. You know, they're the product groups always wind up having some fire to put out. And so they'll pull the researchers onto the short term development work, put the latest fire out.

• 24:19Speaker 03

  They wind up not, not really doing fundamental research. So our goal is to do that fundamental research and, and we succeed in that as evidenced by publishing, you know, lots of papers at leading conferences like, you know, NIPS and ICLR and ICML and CDPR. Um, then the, the other goal is to make sure that that research is beneficial to Nvidia, that it makes a difference for the company. And again, that's another failure mode of industrial research labs. Many of them publish lots of great papers, do lots of great research, and it has absolutely no impact on their parent company. Um, I think I'd have trouble convincing Jensen to continue, uh, running the research lab if we didn't, um, have many successes, but we do. So for example, the, the ray tracing cores in Turing were originally an Nvidia research project. KuDNN, as I mentioned, um, came out of research.

• 25:03Speaker 03

  We are applying deep learning to graphics. We, we demonstrated with Turing something called deep learning, um, super sampling, which basically, basically anti-alias is an up, uh, samples an image, um, using neural networks. It does it in a temporally stable way. Our DGX two, which includes NVSwitch, NVSwitch started as a project in Nvidia Research, as did NVLink on which, um, the switch is based. So, um, we have a long track record of taking kind of crazy ideas, you know, maturing them within Nvidia Research and then getting the product groups to, you know, embrace and ultimately put them into future GPUs and, um, software and, and systems products that we produce.

• 25:40Chris Benson?

  So Bill, as we come back out of break, uh, I wanted to ask you, uh, kind of back out just a little bit because we've gone down some, some amazing paths. I know Daniel and I have learned so much already on the show from you, but I wanted to, to put a little context around some of that and kind of get a sense as you've told us about all of these amazing technologies. What is Nvidia's vision kind of for the, for the future of AI? And, and as you've talked about some of the parts of your AI platform, you know, how are you utilizing that platform strategically to realize that? And what kind of investments are you expecting Nvidia to make going forward?

• 26:15Speaker 03

  So the short answer for the future of AI is continued rapid innovation. I expect to continue to have to stay up late every night reading papers on archive. And even then not be able to keep up with what's going on. But if you look at how that rapid innovation is happening, I think it's along several different axes. The first axis, I think, is breadth of applications. I think we've only begun to scratch the surface of how AI is affecting our daily lives, how we do business, how we entertain ourselves, how we practice our professions. And I expect more applications of AI to be occurring every day. And those applications to present unique demands, the type of models we need, how we curate training data, how we train the networks with that data and so on.

• 26:59Speaker 03

  The next axis, I would say, is one of scale, scale of both model size and data sets. We've seen this in areas like computer vision, in speech recognition, in machine translation, where over time, people collect larger data sets to have the capacity to learn those data sets. They build larger models that really raises the bar for the performance you need to train those models on those large data sets in a reasonable amount of time. And then finally, the axis that's probably most exciting to me is coming up with new models and new methods that basically increase the capability of deep learning to be more than just perception, to basically give it more cognitive ability, to have it be able to reason about things,

• 27:44Speaker 03

  to have longer term memories, to operate and interact with environments. A lot of the work in reinforcement learning, we find very exciting along that axis. So seeing AI, this constant innovation along all three of these axes, our goal with our platform is to evolve to meet these needs, to meet the needs of newer applications, to meet the needs of larger scale and more capable models and methods. And there's a couple of ways we need to do that. One is to continue to raise the bar on performance. To train larger models on larger data sets requires more performance. And Moore's law is dead. We're not getting any more performance out of process technology. So it requires us to innovate with our architecture, with our Circuit designs to do that. And we've done that generation to generation.

• 28:29Speaker 03

  If you look at the performance from Kepler, where we started working on deep learning, to Maxwell and Pascal, Volta, and now Turing, we've been able to really increase by large multiples, deep learning performance on each subsequent generation in the absence of really any help from process technology. And we expect to continue doing that. The next thing we need to do is we need to make it easier to program so that people who are not experts in AI, but are other experts in their domain, can easily cultivate a data set, acquire the right models and train them. And we do that through our tools. We support every framework. We have TensorRT to make it easy to map your applications

• 29:10Speaker 03

  onto inference platforms. And then we also have training programs. We have a deep learning institute where we basically take people who are application experts and train them so that they can apply deep learning to their application. And then the final way we want our platforms to evolve is to remain flexible. The deep learning world is changing every day. And so we don't want to hardwire too much in and not be able to support the latest idea. In fact, we think it would inhibit people coming up with the latest idea if the platform that everybody is using was too rigid. We want to make it a very flexible platform so that people can continue to experiment and develop new methods.

• 29:44Daniel Whitenack?

  Yeah. So in light of that, I'd be really interested to hear from your perspective, how ideas at Nvidia actually advance from research to reality, in particularly in light of what you just said, in light of that you want to make things easier for people to program, easier for people to interface with, application people to interface with, while at the same time pushing performance forward and keeping flexible. It definitely seems like it might be hard to balance those things. But as you've already mentioned, there's been a lot of great things that you guys have come out with that do balance that really well. So I was wondering if from that perspective, how you see things advancing from research to reality in Nvidia?

• 30:28Speaker 03

  Yeah, so that's a good question and one that I'm very excited about because it's kind of my job to make sure those things advance. So not all ideas start in NVIDIA Research. Many ideas start in the product groups. Many ideas start with application engineers who work with the customers and see the need. But for the ideas that do start in NVIDIA Research, which is an organization of about 200 people, individual researchers generally just start experimenting with things, come up with a good idea. And then the goal is to find a way for that idea to have impact on the company. And so we try to make sure everybody, when they come up with an idea, identifies both a champion and a consumer, who are often the same person in the product groups for that technology. And as they develop the technology further, they get some indication about, gee, does

• 31:12Speaker 03

  the champion care about this technology? Can they, will it make their product better? And if it doesn't, it's often an indication they should drop the idea. In fact, to me, one of the keys of good research is to kill things quickly. Most research projects actually don't go anywhere. And there's nothing wrong with coming up with research ideas that don't work. What's wrong is spending a lot of resources on them before you give up on the ones that don't work. And so we try to kill the ideas that either aren't going to work or aren't going to have impact on the company pretty quickly. But the ones that are going to have impact on the company, one thing that's really great about NVIDIA is it's a company where it's like a big startup. There's no politics, there's no not invented here. So if there's a good idea, the product groups don't care that it came out of research. They say, that's a great idea, we want that. And very often, they'll grab things out of our hands before we even think we're done with them.

• 31:57Speaker 03

  NVSwitch was a great example of that. We wanted to actually complete a prototype in research. We didn't get the chance. They grabbed it, made it a product before we had the chance to do that. And it's really about people. The people come up with the concept are communicating with the people who will turn it into reality. And then, once it sort of jumps over to that side, it becomes more of an engineering endeavor, less of a research endeavor, where people have to hit goals, things have to work, they have to be verified. But the whole process works. And ultimately, we're able to very quickly go from concept to delivering very polished, very reliable products to our end customers.

• 32:31Chris Benson?

  So I would like to take you into a particular use case. I know when I was at GTC in March, Jensen was on stage doing his keynote and we had all walked in looking at the amazing autonomous vehicles that you guys had in the lobby and stuff. And he made a comment that really struck me and I was just wanting to get your thoughts on it. He said, "Everything that moves will be autonomous." And in that presentation, he went way beyond just cars. He was talking about literally everything, whether it be on the land, sea, or air. And so, obviously, that would include GPUs and maybe other specialized processors that you guys put into those vehicles. But what other things are you doing to realize that vision, considering how cool it is to the rest of us?

• 33:15Speaker 03

  That's a great question. So, one thing we're doing in Nvidia Research is we're actively pursuing both autonomous vehicles and robotics. And in fact, autonomous vehicles are a special case, in many ways, an easy case of robotics, in that all they really have to do is navigate around and not hit anything. Robots actually have a much harder task in that they have to manipulate. They have to pick things up and insert bolts into nuts. They have to hit things, but hit things in a controlled way, so that they can actually manipulate the world in a way that they desire. And so, I've recently started a robotics research lab at Nvidia. It's in Seattle. We hired Dieter Fox from the University of Washington to lead that lab. And robots are just a great example of how deep learning is changing the world, because

• 33:58Speaker 03

  historically, robots have been very accurate positioning machines. If you look at how they've actually been applied in the world, auto manufacturers use them on their lines to do spot welding and to spray paint, but they're not responding to the environment. They simply have been programmed to very accurately move an actuator to a position repeatedly, over and over again, do exactly the same thing. With deep learning, we're able to actually give robots perception and the ability to interact with the environment, so that they can respond to a part not being in the right place, adjust, manipulate, pick that part up, move it around. They can perhaps even work with people, working as a team, where the robot and the person are interacting together by using deep learning to provide them with both sensory abilities and also

• 34:41Speaker 03

  Through reinforcement learning, the ability to reason and choose actions for a given state that they find themselves in. And so our goal from this is by doing this fundamental research in robotics is to basically learn how to build future platforms that will be the brains for all of the world's robots, just like we want to build the platform that's going to be the brains for all of the world's autonomous vehicles. Hopefully, this research will ultimately lead to that platform, not just the hardware, but the various layers of software and ultimately the fundamental methods that those future robots and autonomous vehicles will be using.

• 35:15Daniel Whitenack?

  So, Bill, we've kind of transitioned into talking about use cases and you've mentioned a lot about about robots and other things kind of at, quote unquote, the edge. I was wondering if you could give us a little bit of a perspective on, you know, what you moving forward, what you see as the as the edge and how how neural network, both training and inference, will be kind of spread across, you know, centralized compute in the cloud or on premise and on on edge devices and what those edge devices might might look like.

• 35:49Speaker 03

  That's a good question. So I see deep learning is happening in sort of three ways. So the first is training, which by large takes place in the cloud. And the reason why you want it to take place in the cloud is that, first of all, you need to have a large data set. You need to have someplace where you can store terabytes of data, maybe even even, you know, more than that. And, you know, you really want to do that in a centralized location. You also, if you're gathering training data, say, from a fleet of autonomous vehicles, you want them all to learn from each other's experiences, right? So you want to gather all that data, collate it one place, curate the data to basically discard the stuff that's not very interesting, keep the stuff that is, and then train one network on all of the data. So training really wants to happen in the cloud.

• 36:33Speaker 03

  It requires a large data set. It has a large memory footprint, has unique requirements, requires FP16. And then there's inference. And inference happens in both the edge and the cloud. I think most people, if you can do inference in the cloud, would prefer to do it there. There's an economy of scale. You can also share resources. You know, if you have a task where you're not doing inference constantly, but on demand, then you don't need to have a resource tied up all the time. You can only, you know, you can share it, use it when you need it. Somebody else can use it when you don't need it. So it's just more efficient to do inference in the cloud. But there are cases where you can't do inference in the cloud. And you know, an autonomous vehicle is a great example. First of all, you may have latency requirements, right? If your camera sees, you know, the, you know, kid running into the street,

• 37:18Speaker 03

  you can't afford the latency to send that image to the cloud, do the inference there and send the braking command back. You need to have a very tight loop that, that commands the car to stop. You also may not be connected or you may have bandwidth limits. So for example, people who have networks of surveillance cameras are producing just too much data to send all of it to the cloud. They need to do some data reduction, at least locally, have some local inference that filters the data and then send only the interesting data to the cloud for, for further processing. And then finally, there may be privacy constraints that, that limit your ability to send stuff up to the cloud. You may want to handle things locally to avoid sharing data that you don't want to share. So I think there are a lot of reasons why you want to do inference in these embedded devices. Almost no reason why I think you would want to do training there. And in, in the case where, where you are doing inference

• 38:03Speaker 03

  in the embedded devices, that often has very strong energy efficiency constraints. They may be battery operated. They may, you know, need to run for a long period of time without being, being recharged. And so the efficiency demands are even higher than, than for inference in the cloud.

• 38:17Chris Benson?

  Yeah, I've actually run into that myself in terms of those, the, the battery constraints, doing inferencing on mobile devices. You know, we, we've covered so much ground. If you are a software developer or maybe a data scientist who's doing software development and, and the engineering and, and you're looking at all of these things that we have been talking about, kind of from an app dev perspective, you know, from training and the hardware, working on the edge, the different tools, CUDA, you name it. What are the things that, the necessary skills that people should be thinking about? So many people are kind of self-training themselves into this. And there is, there is so much for a person who's just trying to get into AI to learn.

• 38:59Chris Benson?

  How would you structure that if somebody is trying to, to self-train them into this field?

• 39:03Speaker 03

  Well, I think actually what you need to know to be successful in AI falls into two categories. One is basic knowledge and the other is very practical how to information. For the basic knowledge, I think what's most important is having a really strong background in mathematics and particularly in statistics and probability theory because that's what all of AI is based on. It's, you know, you're basically, you know, doing statistical estimation of a number of things. And then the practical side of it is knowing how to use the tools that are available, whatever your favorite framework is, whether it's PyTorch or whether it's TensorFlow, having the practical knowledge to, you know, get a model, get a data set and run the tools to train it.

• 39:41Chris Benson?

  So since you mentioned that, I'm just curious because Daniel and I have used different tools. Do you have any personal favorites that you like to use? Not suggesting anything that you say is the right thing that everybody should do, but we always like to find out what people's preferences are.

• 39:56Speaker 03

  I don't really have any strong preferences. I have to confess that I actually don't do that much coding myself anymore. And the people I work with often migrate to one or another for different reasons. A lot of people use PyTorch because they like to sort of work from the Python base. Many people use TensorFlow. I think it is probably the most popular framework overall these days.

• 40:15Daniel Whitenack?

  Yeah, and I'm sure a lot of the frameworks that your team uses and also the tools that they generate and the research that they generate, I'm sure a lot of that uses open source tools like you've already mentioned. Are there any things that you'd like to highlight that, you know, Nvidia is kind of doing on the open source front that maybe our listeners could go and check out and potentially start playing around with?

• 40:39Speaker 03

  So one thing I'll highlight actually is our deep learning accelerator. If you listeners go to NVDLA.org, if they actually want to play with hardware for deep learning, they can download the RTL for that accelerator, customize it to their needs, include it into either an FPGA or an ASIC of their own design. We also open source a lot of software that comes out of our research. So, you know, for example, our work on progressive generative adversarial networks, progressive GANs, our work on networks that we use for optical flow, our work on denoising, all of those networks have been open sourced. So people can very easily replicate our results and apply those new methods that we've developed to their own problems.

• 41:17Daniel Whitenack?

  Awesome. Yeah, that's, that's super helpful and we'll make sure and include some, some links in our show notes to that. As we wrap up here and get to the end of our conversation, once again, I really appreciate all of the perspective on these different, um, different things. It was really helpful for me, I know. Um, I was wondering if you have any parting thoughts or kind of inspiring thoughts for the listeners? Um, assuming that our listeners are kind of either already in or getting into the, the AI field and kind of trying to find their place and find, you know, what people are working on. Do you have any parting thoughts for them or encouragements?

• 41:53Speaker 03

  Yeah, I think it's just a very exciting time to be working in AI because there are so many new developments happening every day. It's a, it's never a dull place. Um, it's in fact, so much stuff happening that's hard to keep up. As a hardware engineer, I think it's also very rewarding to know that this whole revolution in deep learning has been enabled by hardware. All of the algorithms, you know, convolutional nets, uh, multi-layer perceptrons, training them using stochastic gradient descent and backpropagation. All of that has been around since the 1980s, since I first started playing with, with neural networks. But it wasn't until we had GPUs that it was really practical. GPUs basically were the spark that ignited the, the revolution. They are the three ingredients were the algorithms, the large data sets, those were both there, but then you needed the, the GPUs to make it work.

• 42:38Speaker 03

  It wasn't, you know, for computer vision, it wasn't until AlexNet in 2012, where using, you know, GPUs, he was able to train a network to, win the ImageNet competition, that deep learning really took off. So I think GPUs are what ignited this. And I think GPUs are still really the platform of choice because with the tensor cores, they provide the efficiency of special purpose units, but without the inflexibility of a hardwired ASIC like a TPU. So you get the best of both worlds. Um, you can program in CUDA, but get the efficiency of a tensor core.

• 43:08Chris Benson?

  Well, thank you very much, Bill. Uh, for me, I have learned so much on this episode that I'm probably gonna have to go back and, uh, listen to it a couple of times to take in everything that you've taught us today. It's been really packed with incredible information. So thank you very, very much for coming on.

• 43:22Speaker 03

  Oh, it's my pleasure. Thank you.

• 43:24Chris Benson?

  And, uh, with that, we'll look forward to our next episode. I hope our listeners got as much out of it as Daniel and I did. Daniel, you doing good?

• 43:31Daniel Whitenack?

  Yeah, I've got.

• 43:32Chris Benson?

  Is your head going to explode yet?

• 43:33Daniel Whitenack?

  I've got, I've got a bunch of websites pulled up that I, I'm gonna, gonna start reading afterwards. So, uh, so it was a great time and we'll talk to you again next week.

• 43:42Chris Benson?

  Great. Thank you very much, Bill.

• 43:47Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When

• 44:29Speaker 00

  you go there, pop in your email address. Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.