Deep Reinforcement Learning

While attending the NVIDIA GPU Technology Conference in Silicon Valley, Chris met up with Adam Stooke, a speaker and PhD student at UC Berkeley who is doing groundbreaking work in large-scale deep reinforcement learning and robotics. Adam took Chris on a tour of deep reinforcement learning - explaining what it is, how it works, and why it’s one of the hottest technologies in artificial intelligence! Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $100 credit. Learn more at do.co/changelog https://do.co/changelog. O'Reilly Open Source Software Conference https://oreil.ly/2UurGS2 – OSCON has been ground zero for the open source community for 20 years. This year they’ve expanded to become a “software development conference” — because in 2019, software development IS open source. The program covers everything from open source, AI, infrastructure, blockchain, edge computing, architecture, and emerging languages. Use the code CHANGELOG20 to get 20% off Bronze, Silver, and Gold passes. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. GitPrime https://www.pluralsight.com/product/flow – Download GitPrime’s 20 Patterns book https://resources.gitprime.com/books/20-patterns/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog(q1), a field guide to help engineering managers recognize achievement, spot bottlenecks, and debug development processes with data. Featuring: Adam Stooke – GitHub https://github.com/astooke, LinkedIn https://www.linkedin.com/in/adam-stooke-06bb6923, X https://x.com/stookemonChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: NVIDIA GTC: Doing More with More: Recent Achievements in Large-Scale Deep Reinforcement Learning https://gputechconf2019.smarteventscloud.com/connect/sessionDetail.ww? SESSION_ID=277128Accelerated Methods for Deep Reinforcement Learning https://arxiv.org/abs/1803.02811Accelerated Methods for Deep Reinforcement Learning - code https://github.com/astooke/accel_rlOpenAI Five - Dota 2 https://openai.com/fiveAlphaStar: Mastering the Real-Time Strategy Game StarCraft II https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-iiNVIDIA Graduate Fellowship https://www.nvidia.com/en-us/research/graduate-fellowships/2018Crunchbase https://www.crunchbase.com/person/adam-stookeThe Hertz Foundation https://hertzfoundation.org/fellows/fellow-profile/11174/Adam-StookeUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:46Chris Benson?

  Welcome to the Practical AI Podcast. This is Chris Benson. We are the podcast that brings AI to you in a practical, productive, and accessible way. I am at Nvidia's GPU Technology Conference in March 2019. And we have a lot of world-class people in the AI space here. And Normally, I would have my co-host, Daniel Whitenack, joining me. He was not able to make it this time, but I have the pleasure of introducing to you Adam Stook. Adam is a graduate student who is studying robotics and deep learning at UC Berkeley,

• 2:27Chris Benson?

  and he's really focused on scaling deep reinforcement learning and hierarchical reinforcement learning. And so it's a pleasure to have you on the show, Adam. Welcome.

• 2:37Speaker 02

  All right. Thank you, Chris. It's a pleasure to be here.

• 2:39Chris Benson?

  So, I wanted to note, and we'll dive into it in a few minutes, that you had done a talk called "Doing More with More Recent Achievements in Large-Scale Deep Reinforcement Learning." But before we dive into the meat of it, could you tell me a little bit about your background? Kind of how you got here? How you discovered deep learning and reinforcement learning? And what your story is?

• 3:01Speaker 02

  Sure. I've been working and studying deep reinforcement learning for about four years now as a PhD student at UC Berkeley, under my professor, Peter Rebeal. Previous to that, I had studied physics and mathematics in undergrad. Actually, did a master's in physics here at UC Berkeley. And then I was in the Air Force at the time, so I was pulled away on active duty. I was very lucky to land a position at the Air Force Research Lab in Albuquerque, New Mexico. So I kept my technical chops up pretty well during this period. And yeah, about four years ago, my time in the Air Force kind of came to a natural end, and I came back to school and was looking for what's hot, what's next. And yeah, the robots caught my eye.

• 3:45Chris Benson?

  So I gotta start off by saying a couple of things. Connections here. First of all, Daniel is gonna be completely jealous that he's not here, because he is a physicist who also came into the AI space as well. So he's gonna be going, "Ah, gosh, why did you take it, Chris?" So, sorry, Daniel, on that. And number two, being in the Air Force, I worked for Lockheed Martin in the AI space, and we do a lot of work with the Air Force. So, anyway, I guess, so as you started, were you already interested in deep learning reinforcement learning when you were in the Air Force? Or did that come along sometime thereafter?

• 4:22Speaker 02

  Uh, no, this came along after, actually, only once I returned to graduate school. What I did get out of my time in the Air Force was just kind of by accident. It became a first exposure to me to a lot of programming and modeling and simulation in order to make informed decisions about technologies that we were going to invest in and try to fly in space.

• 4:43Chris Benson?

  What kind of tools were you using while you were doing that?

• 4:45Speaker 02

  In that case, I was programming pretty much entirely in MATLAB and a little bit of Excel. Why not?

• 4:52Chris Benson?

  Yep, but that's actually, a lot of people I know get into it from that. So that's totally valid.

• 4:57Speaker 02

  And so, yeah, that kind of gave me the first impression of like, wow, this is really powerful. If you can craft a good simulation and run this program in the computer, it can tell you a lot of important things about what's going to happen in the real world. And that was the first thing that kind of turned me on to this idea. So that when I came back to school, it was a kind of a natural thing to look to a computer science department.

• 5:18Chris Benson?

  Yeah, so tell me a little bit about that process. You're out of the Air Force, you're getting into school. How did you discover it? What specifics did you get into? How did you find your way eventually to reinforcement learning?

• 5:29Speaker 02

  Uh, yes, so I came back to school and I rejoined the physics department at Berkeley, where I'm actually still technically a member.

• 5:37Chris Benson?

  Okay.

• 5:37Speaker 02

  But after about a semester and a summer of kind of floating around to different research groups and finding a lot of really, really deep and interesting projects going on, but not really getting the personal feeling like that I was finding a place where I wanted to be continuing the work after graduating, you know, doing the PhD in physics still has a really strong draw on me. But I started to look to other things to do after graduation. And then eventually I decided, well, if I'm going to do something different after graduating, why don't I get ready to do something different before graduating? And so I started looking a little broader and found the robotics work happening kind of across the street on the campus there.

• 6:17Chris Benson?

  Okay.

• 6:17Speaker 02

  And a computer science. I'm very lucky to be at such a powerhouse research university like Berkeley.

• 6:22Chris Benson?

  Absolutely.

• 6:22Speaker 02

  With a hand in so many, leading in so many different fields. So, yeah, I just kind of went across the street and knocked on the door and said, hey, hey, can I play? And a year or two of basically re-accomplishing an entire set of graduate coursework and projects later, professor finally says, yeah, you know what, Adam? Okay, you can stay, you can do this with us.

• 6:45Chris Benson?

  He made you earn your chops there, didn't he?

• 6:46Speaker 02

  Which is code for, I can see you're not going away, so I might as well.

• 6:51Chris Benson?

  Yeah, persistence there, you know, and intelligence and all, but you're sticking with it there. So, um, so they were a robotic shop at this point that you're moving into?

• 6:59Speaker 02

  That's correct. Yeah, it's a robotic shop. Um, and at the same time, uh, it's, it's a lot about applying, applying deep learning to robotics.

• 7:07Chris Benson?

  Okay.

• 7:07Speaker 02

  Um, so kind of my personal path since then has actually been, uh, much heavier on the, on the deep learning side, the deep reinforcement learning in particular.

• 7:15Chris Benson?

  And before we even go on, we've mentioned, uh, reinforcement learning and deep reinforcement learning several times already. Could you tell us how you would interpret those in, in terms of what they are for anyone that doesn't know what that is?

• 7:26Speaker 02

  Sure. Yeah. Let's start with reinforcement learning, which has, um, a little bit of a longer, uh, history and kind of pedigree behind it. It's basically a kind of a, a formal way of doing learning by trial and error. Um, so you've got some sort of task that you're, you're trying to figure out how to do. Um, you know what are the different possible, uh, you know, small actions that you can do in order to try to achieve that task, but you don't necessarily know ahead of time what the task is, uh, or how to do it. So you just explore, you try things, you, the key thing is that you get a signal back from the, from the task. That's what we call a reward. Um, when you receive a positive reward, it says, oh, you did a good thing. So do that more. When you receive

• 8:10Speaker 02

  a negative reward, it says, oh, you did a bad thing. So do that less. Um, and yeah, based on queuing off of those signals, basically again, through trial and error gradually over time.

• 8:18Chris Benson?

  Make an adjustment.

• 8:19Speaker 02

  You make an adjustment, you try again, uh, and then eventually you can learn to do some, some pretty sophisticated tasks through this. So, so there's a whole, um, yeah, mathematical formalism kind of like built up around this that we don't necessarily need to go into the depths of, of all, of all the terms with that, but it's pretty well established. Um, and a more recent, uh, development is deep reinforcement learning, which simply refers to the application of deep learning to reinforcement learning. Uh, deep learning, uh, meaning for all practical purposes, using deep neural networks.

• 8:49Chris Benson?

  Sure. Um, and, and at a high level, without diving too deep on this, can you talk a little bit about how you're applying deep learning to integrate that into reinforcement learning? Is there, uh, is there a, a high enough level to, for, for kind of a, an understanding of what that integration is without diving too far into the math?

• 9:07Speaker 02

  Sure, yeah, we can do that.

• 9:08Chris Benson?

  Okay.

• 9:09Speaker 02

  So, one of the key components that you have in a reinforcement learning system is the learning, what you would call the learning agent itself. So, the learning agent is interacting with the environment, which contains the task. And the learning agent is where the deep learning comes in, because the learning agent is going to be somehow making use of a deep neural network to express its decision-making.

• 9:35Chris Benson?

  Sure. And so, this gives performance advantages over what you could previously do with just reinforcement learning, given the fact that there are so many more options available in terms of adjustments that you can make going forward, I assume.

• 9:47Speaker 02

  That's exactly right, yes. So, in terms of the interfaces between the environment and the agent, so the environment provides some sort of information to the agent that the agent gets to look at and use, and we're making its decision of what to do next. And it turns out that deep neural networks basically open up a whole new class of environments of much more complicated sets of information that the agent is able to digest in order to figure out what to do next.

• 10:16Chris Benson?

  Sure. So, I guess, so what, as you got into this environment, you're now, they've accepted you into the fold. What did you do? What captured your imagination as you got into some of your first projects with the group?

• 10:29Speaker 02

  Yeah, I think early on, the projects that I kind of queued in on the most were some early work out of DeepMind on learning to play Atari games from scratch, using only the screen image

• 10:42Chris Benson?

  Yep.

• 10:43Speaker 02

  as the input to the learning agent. So, in a very similar style to the way that a human would actually operate to play the game.

• 10:51Chris Benson?

  So, I remember reading about that. Can you tell us a little bit about, you know, what that was and how it inspired you to move forward?

• 10:57Speaker 02

  Sure, yeah, so this is about five or five or more years ago by now, maybe was the first result of, again, playing Atari from only the screen image. It made a big splash when it came out.

• 11:10Chris Benson?

  It sure did.

• 11:10Speaker 02

  There was a Nature article about it, cover of Nature even.

• 11:13Chris Benson?

  Yeah.

• 11:14Speaker 02

  So this went a lot more, published a lot more broadly than the usual, like, learning-only conferences and this sort of thing. And then very shortly after that, after the initial successes there with the first algorithm, then another paper comes out with the second learning algorithm that's a little bit different, but is also getting good results, maybe running a little bit faster. And then over the next several years, kind of in rapid fire, these teams that came out with the original ones are making little enhancements here, little enhancements there, piling them on. And before you know it, these learning algorithms are just completely destroying these Atari games way, way beyond

• 11:51Chris Benson?

  Oh, yes.

• 11:52Speaker 02

  human level. And it really is just a springboard for lots of more sophisticated and more interesting games, for instance, to come after that.

• 12:02Chris Benson?

  So, you know, obviously, that's been a while since the Atari news that came out in Nature and stuff. Where did that take you at that point? As you're getting into this, what captured your interest? What did you choose to do? What activities from, in terms of research, did you want to engage in? What was that journey like? Because a lot of us out here who are very interested may not have had that experience, and just very curious about how that evolved to where you are today with the talks that you're giving at this point.

• 12:32Speaker 02

  Yeah, I think it actually ended up in combination with a certain course that I was taking on parallel computing. It actually led me down the path of scaling up implementations of deep reinforcement learning. So taking an algorithm, okay, this algorithm is, like, shown to work, and here's how we run it. Maybe we're running it on a small computer, and it takes five or 10 days to do one learning run and to see the thing go from losing every point in Pong to winning every point in Pong. Say, okay, this is exciting. This is working. This is something, we've got, like, something real to go on here. Now, let's refine it and scale it up and adjust it to make much better use of the computers that are available

• 13:15Speaker 02

  today. And, I mean, long story short, that has a lot to do with why I'm here at GTC.

• 13:19Chris Benson?

  Yep, well, we're working our way there.

• 13:24Chris Benson?

  So you mentioned already the compute being a huge limitation. And obviously, we are at Nvidia GTC where it's all about compute in so many things. So I guess, you know, what bridged you, you know, today you did this talk doing more with more recent achievements in large-scale deep reinforcement learning. What kinds of things are you covering in that? And, you know, what was the evolution to get to that point? And what were the, what were some of the milestones that you went through along the way to get to where you're at now?

• 13:54Speaker 02

  Sure. I think some of the early steps started out with just taking a kind of in-house algorithm, reinforcement learning algorithm that we had, and identifying kind of the key pieces in a way that the algorithm could be broken down to operate in parallel over more compute resources. So instead of just one CPU core running the computation or whatever, maybe you can use all the CPU cores in the machine, 16 of them, running in parallel in a smart way that gets you 16 times faster.

• 14:27Chris Benson?

  Yes.

• 14:28Speaker 02

  You know, to the answer. And so this was kind of how I started, was just with one example, kind of algorithm like this. And just a lot of practice of figuring out how to set that up, how to, how to set that up in a computer and get it to run and actually get the efficiency out.

• 14:45Chris Benson?

  And you hadn't even gotten to the GPU world yet.

• 14:48Speaker 02

  Hadn't even gotten to the GPU world, no, no. And then, let's see, with this, I actually took a little, once, once we had this running up and running smoothly, took a little detour through, through actually an Intel Knights Landing development kit that we bought. So a CPU-based architecture, but with maybe 60 or 70 cores that we could successfully parallelize across. But ended up finding in the end that, no, really.

• 15:16Chris Benson?

  It wasn't quite up to what you needed, was it?

• 15:18Speaker 02

  Wasn't quite up to what we needed, and really, really, there's no way around it. Like, the GPU was, was the way to go.

• 15:23Chris Benson?

  So, so at what point, you know, how long ago was that where you realized you needed to go GPUs and you started grabbing those because you realized that you were, you were limiting yourself until you could speed up your training?

• 15:35Speaker 02

  Yeah, I think by now, this was maybe, maybe about two years ago.

• 15:39Chris Benson?

  Yep.

• 15:40Speaker 02

  Or so. So it was, it was a bit of a journey. Yeah, starting, starting on the GPUs about two years ago, and then about one year ago, kind of wrapping up the project and more or less having it in shape.

• 15:50Chris Benson?

  So one of the things I was thinking as you were taking me through this, what was, what were you trying to achieve with some of these projects? As you're, as you're doing these experiments and, and carrying it forward, what was your end goal in mind? What were you trying to get to?

• 16:04Speaker 02

  Yeah, so the main, the main project that I was working on over this period was, was really just, man, we had a need for speed. We're just, we're just gonna explore these, these deep learning, these deep reinforcement learning algorithms and see how fast, how fast can we learn one Atari game?

• 16:22Chris Benson?

  Yeah.

• 16:22Speaker 02

  And, and see, see what kind of records we can't set and otherwise blow out of the water. So it turns out that a lot of the same techniques for parallelism that were useful in the CPU setting, the same principles applied in the GPU setting.

• 16:36Chris Benson?

  Sure.

• 16:36Speaker 02

  For using multiple GPUs inside of one computer to run the algorithm. So about the same time that I was setting down that path anyway, we were very lucky enough to, to have Nvidia come by and, and donate actually a DGX1 as part of their university research support.

• 16:55Chris Benson?

  That's a great donation, you know? That's the kind of donation I want right there.

• 16:59Speaker 02

  So yeah, this is a really fortunate moment for us and good, and good timing for me. As, as about the time I was getting into this anyway, my advisor says, hey, hey, Adam, we've got this DGX1 and, uh, well, here you go.

• 17:12Chris Benson?

  Go, go, go. Go to it.

• 17:14Speaker 02

  Go to it, exactly.

• 17:15Chris Benson?

  Here's the first AI supercomputer, have at it.

• 17:17Speaker 02

  Exactly. And so I couldn't, couldn't say no to that. I mean, didn't want to say no to that.

• 17:21Chris Benson?

  Of course not.

• 17:22Speaker 02

  Couldn't say no to that.

• 17:22Chris Benson?

  That's Christmas right there.

• 17:23Speaker 02

  And then couldn't stop once I, once I, once I started either. So.

• 17:28Chris Benson?

  I'm guessing you've, it was speeding up your process a little bit there, you know, to go from CPUs, uh, straight, and you had, it sounds like you didn't have an in-between. Did you go straight from CPU all the way to using DGX1? Just like that? Or was there an intermediate steps along the way?

• 17:44Speaker 02

  I know, that's right. We pretty much just went straight to, straight to the DGX1.

• 17:48Chris Benson?

  Wow. That's like stepping out, that's like stepping out of this old jalopy car and getting right into the Lamborghini and just going.

• 17:56Speaker 02

  That's it, that's it. I don't, yeah.

• 18:01Chris Benson?

  So I'm just curious, you know, it's non-trivial to, you have to kind of, to learn how to use a DGX1. They suddenly plopped one into your lap and said, "Hey, make use of this thing." And, and, you know, what was the, were you already familiar with, with containerization and Kubernetes and things? Or was there a learning curve for you to, to get ready to use the, the box? Or what was that like to make that transition?

• 18:26Speaker 02

  There was a little bit of a learning curve. And so at this point, I was already pretty familiar with the deep learning, the particular deep learning library that we were using, which at this time was Theano.

• 18:36Chris Benson?

  Mm-hmm.

• 18:37Speaker 02

  And luckily enough, about the same time, Theano released support for Nickel, so Nvidia's collective communication library.

• 18:47Chris Benson?

  Yep.

• 18:47Speaker 02

  Which is for direct interconnects between the GPUs within the DGX1 for very fast operations. Very fast communication, sorry. And so, yeah, the learning curve was mainly around, okay, I already am familiar with this deep learning library, but let's see how to plug it into use for multiple GPUs. This was a pretty new thing at that time, about two years ago. And so there wasn't really built-in support for Theano, for, from Theano for multi-GPU use. There was some kind of preliminary stabs at that. But what I ended up doing was kind of writing, writing our own libraries for the, for the parallel processing aspects.

• 19:25Chris Benson?

  Sure.

• 19:25Speaker 02

  And then, and then, yeah, again, luckily having the Theano guys expose an interface to this Nickel, this communications library, to tie the multiple GPUs together in the most efficient way possible, based on the Nvidia hardware that was inside.

• 19:39Chris Benson?

  Got you. And I guess, were you already in the middle of a particular project when the DGX1 came into, into the picture? Or was that, did you kind of pick it up as you picked up a new project to move forward?

• 19:51Speaker 02

  No, I think, looking back on it, I hadn't really thought about this before, but it was just incredibly fortunate timing, because I was already on this project, had already been tinkering around with the Knights Landing for some time, and was ready to move on for that, and move to GPUs. And so, no, it just, like, the timing couldn't have been better.

• 20:09Chris Benson?

  It would just happen. That sounds fantastic. So, I guess, as you're making this transition, how are you, so you now have this DGX-1 that's been gifted to you, and they're saying, "Go do stuff with it." What's going through your head? What kind of projects are you thinking about? You're now into deep reinforcement learning, I assume, at this point, and you have the power to drive that forward. What were you thinking, and what kinds of projects have you taken on over the last couple of years to take advantage of that?

• 20:38Speaker 02

  Yeah, so, I mean, again, a lot of the work at the beginning was just scaling out the reinforcement learning itself, taking existing algorithms and showing, discovering that they can be scaled up to run on the entire system, so that we could use all eight GPUs and all 40 CPU cores within a DGX-1 to learn a single Atari game, and get basically linear speed-ups with that. So, instead of taking 10 or 15 hours to master Pong, you know, we're getting it to, like, four minutes.

• 21:09Chris Benson?

  Yep.

• 21:10Speaker 02

  Or so.

• 21:11Chris Benson?

  Which was... That probably helped your productivity a little bit.

• 21:13Speaker 02

  Helps the productivity.

• 21:14Chris Benson?

  Go get a cup of coffee, and you come back, and it's done.

• 21:16Speaker 02

  Come back, and it's done, and then you can iterate and try the next thing. And that was actually, I think, a pretty interesting finding, because one of the key techniques to scaling up to using multiple GPUs that we also see across a lot of other scaling efforts, is increasing the training batch size. So, how many in this, there might be a couple of points where we need to stop back here and make some definitions, but when training these neural networks, we often use this algorithm called stochastic gradient descent.

• 21:48Chris Benson?

  And can you define that for the audience loosely?

• 21:52Speaker 02

  Sure can, sure can. So, basically, you're gonna have some way that you want to change the outputs, change the behavior of the neural net, which is... This decision making function inside your reinforcement learning agent. And you're going to change it based on experience that you have in the game. But instead of making updates on all the possible experience that you could gather, you you'll gather like a little bit of experience at a time and then make a small adjustment. And then gather a little bit of experience at a time and make a small adjustment.

• 22:29Speaker 00

  This episode is brought to you by O'Reilly Open Source Conference in Portland, Oregon, July 15 through 18. We'll be there, by the way. As you know, OSGC has been ground zero for the open source community for 20 years. And this year, they're expanding to become a software development conference. Because in 2019, software development is open source. At OSGC, you get to see what's shaping the future of software development. The program covers everything from open source, AI, infrastructure, blockchain, edge computing, architecture, and emerging languages. Hear from industry leaders like Holden Caro, Rupert Dottre, Julian Simon, and Alison McCully. Learn more and register at oscon.com/changelog. Prices start at just $925 when you register before April 19.

• 23:13Speaker 00

  After that, the price is going to go up. Plus, you can use our code CHANGELOG20 to get 20% off your bronze, silver, or gold passes. Once again, our code is CHANGELOG20. And head to oscon.com/changelog to learn more and register.

• 23:44Chris Benson?

  Okay, so having defined stochastic gradient descent, take us forward on that.

• 23:49Speaker 02

  Okay, so the next step into making stochastic gradient descent run more efficiently on a GPU, which is itself a highly parallel computing platform, is one way to do this is to increase the training batch size. So increase the amount of experience that you gather and use together each time you're going to make a slight adjustment to this decision making function. And that gives you full Utilization of this GPU, which is kind of like a fat and wide computing pipe.

• 24:16Chris Benson?

  Yep.

• 24:17Speaker 02

  And if you want to make full use of eight GPUs, then you need to have an eight times bigger training batch size in your algorithm, because you need to fill up all eight of those GPUs in order to run them efficiently. So it's a pretty interesting finding that we found that we were able to scale up the training batch size, even in a game as simple as Pong, all the way to the point where we're making efficient use of eight GPUs in the full machine and we're getting good linear scaling, where you're learning the game basically eight times as fast when using eight GPUs.

• 24:48Chris Benson?

  So I know at some point Nvidia is working on some of the same problems, you know, as they're learning to get the parallelism of the capacity out there, so that you can take advantage of all the GPUs. Were you still using the code that you wrote to handle that? Or have you switched over to some of the stuff that Nvidia was producing? I had the same experience working at a prior employer, where some of the things that we needed to do was out ahead of any release and stuff. So we had a kind of a similar problem. Have you just stuck with the code that you wrote to accomplish that?

• 25:19Speaker 02

  Yeah, so far for new projects that I've been working on, that aren't focused on scaling RL, but doing other learning experiments. Yeah, exactly. We're using the same codebase that we established before during the scale-up project. So far, the new projects we've been running, it's interesting, you end up running many different experiments in order to see what is working and what is not working. So in the end, an efficient way to run is actually to use only a single GPU, but maybe stack multiple different experiments running on the same computer, each one on a separate GPU. And so we had kind of like baked that into our codebase anywhere, where it was flexible. You could either throw all eight GPUs at one problem, or have each of the eight GPUs running a separate problem, and you have like many different settings that you're trying

• 26:04Speaker 02

  to test anyway. And so either way, in the end, you get a good turnaround of results.

• 26:08Chris Benson?

  Gotcha. So now that we've kind of gotten up to more or less, you know, the more recent past, and you're now using DGX1, and you have these great tools available to you.

• 26:21Chris Benson?

  You know, to kind of come full circle back to your talk today.

• 26:26Chris Benson?

  As you, as you talked about, and I quote, recent achievements in large scale deep reinforcement learning. What things did you cover in your talk? What kinds of things is useful? So, you know, some of our listeners out there are using DGX ones, DGX twos, and other equipment from other companies as well. What, what are the learnings that you found that are going to help us along there in terms of recent achievements?

• 26:48Speaker 02

  Sure. So, most of the projects that I talked about earlier today were to do with large scale research projects happening at other organizations such as like Google DeepMind in London and OpenAI here in San Francisco. Some really, really impressive recent achievements coming out of those places.

• 27:07Chris Benson?

  Can you share some of those achievements from each that you talked about?

• 27:10Speaker 02

  Certainly can. Yeah. So, kind of building on this like foundations that happened under the Atari experiments over the last several years. Now, there's new results coming out in other domains to include Dota 2, StarCraft 2, a version of Capture the Flag that's in an actual like first person action, you know, 3D game style.

• 27:29Chris Benson?

  Yep.

• 27:30Speaker 02

  So, it's really exciting to see a lot of more like sophisticated video games that are actually interesting and still maybe like addictive for humans to play today. That are now being, being tackled and, and solved by, by these learning agents.

• 27:43Chris Benson?

  Yeah, the, the StarCraft 2 work really captured the, my imagination and those of some friends of mine. We were talking about that just in terms of the, the, just the, this field is leaping forward so fast. As you talked about, you know, going from Pong, going to, to, to managing games to that level with, with this technology and doing that. What, I guess, tell us a little bit more about, you know, DeepMind and OpenAI and how that's influenced you.

• 28:12Speaker 02

  Yeah, sure. So I was fortunate enough to do an internship, research internship at DeepMind over the last fall. So I had some firsthand exposure to some of their techniques and their

• 28:23Chris Benson?

  Very cool.

• 28:23Speaker 02

  working methods and such. I'm not going to say anything that isn't publicly

• 28:28Chris Benson?

  Of course not. We would never ask you that.

• 28:33Speaker 02

  So I gotta be careful about that. But yeah, no, it was a wonderful experience to see the whole organization from top to bottom. Amazing resources in terms of people, amazing resources in terms of compute. And it's no wonder, once you're inside the door, it's no wonder the amazing things that are coming out.

• 28:51Chris Benson?

  So of the things that are publicly known now, what was the coolest thing that you experienced there while you were doing that internship that you can share with us?

• 29:02Speaker 02

  Yeah, I think the thing that was the coolest for me, again, was kind of the combination of these two factors I just saw, the people and the compute resources. And both of those organized well in the same place. So I actually was sitting next to much of the StarCraft II team.

• 29:16Chris Benson?

  Really?

• 29:17Speaker 02

  When they were in the heat of their development last fall.

• 29:19Chris Benson?

  Oh, that's cool.

• 29:20Speaker 02

  In the run-up to these like professional matches. And so just like, I wasn't personally involved on that particular project, but I just, you know, heard a lot of background chatter.

• 29:31Chris Benson?

  Looking over their shoulder.

• 29:32Speaker 02

  Saw the very intense meetings going on. Saw people scrambling to get presentations together for the meetings with their latest developments and learning curves and everything. Saw people, you know, debating about how much compute they were going to use, whether they were going to like bring down, you know, all of Google Cloud or something like that.

• 29:47Chris Benson?

  Oh, wow.

• 29:50Chris Benson?

  That would be a bad moment there, you know?

• 29:52Speaker 02

  Would be a bad moment, yeah. So it was just really amazing to see this great organization of really interested and generally excited like researchers working together in this team environment that was like very, very, I don't know, it's like professional and eager.

• 30:11Chris Benson?

  Yeah. So what about OpenAI? You mentioned that a moment ago. What are some of the things that they have been doing that have inspired you?

• 30:19Speaker 02

  Yeah, and OpenAI also has a very exciting project going on with a slightly different game, Dota 2, which I think is even more popular and widespread than StarCraft 2. I think I read something on their blog about maybe the annual winnings of esports contests in Dota 2 is like $40 million a year or so.

• 30:40Chris Benson?

  Wow.

• 30:40Speaker 02

  Yeah, so this game is happening.

• 30:42Chris Benson?

  Some real money there.

• 30:42Speaker 02

  This game is happening, there's real money. And yeah, likewise, they have like a totally independent branch, a very different approach, and their own run to train an agent to play at a professional level in a very complicated environment such as these games.

• 31:01Chris Benson?

  So, it sounds like pretty amazing experiences to be exposed to that. Where do you see yourself going? Is there stuff that you're working on now that you can share? Or things that you have in mind for the future? And, you know, kind of what you want to do specifically?

• 31:18Speaker 02

  Yeah, there's a couple of projects that I've been working on recently that are kind of, I think are interesting. One of them actually is working towards real-world applications of reinforcement learning. So it's very exciting, all this development that is happening in these video games. But obviously, we're not developing reinforcement learning to play the games better.

• 31:41Chris Benson?

  Well, we could be, but we're not.

• 31:43Speaker 02

  We could be, but we're learning to play the games better to develop reinforcement learning, maybe for other things. So I think another really exciting result that came out recently is in job scheduling, which is like managing parallel computing resources. So I'll give a shout out to another group, Hungzi Mao from MIT, is the first author on a paper that came out in the fall to do with reinforcement learning for job scheduling on data clusters under the Apache Spark setup.

• 32:17Chris Benson?

  And we'll include a link to that in the show notes as well for people to go reference.

• 32:21Speaker 02

  Great. And yeah, this is really, really exciting work to see reinforcement learning applied to a real world problem like this, scheduling hundreds or thousands of CPUs, basically, under diverse workloads from, from multiple users. And they, they did a really nice job of laying out, okay, here's a couple of heuristic job scheduling algorithms that you might have, you know, for deciding which computing tasks need to run on which system, which CPU and like when. They, they have some really nice like heuristics that they, that they use, and then also some more advanced, but recent, like handcrafted algorithms. And then they show deep reinforcement learning, of course, taking the cake and blowing them all out of the water. It's a really beautiful thing to see.

• 33:02Chris Benson?

  So, so where, where do you think we're going next in terms of applying deep reinforcement learning to robotics, you know, this field? Both of those individually are moving so fast now. What do you, you know, do a little fortune telling for us and, and lay out what you think the, what, what you think we're going to see in the, in the near term, maybe even the longer term? Be a little speculative.

• 33:26Speaker 02

  Okay, sure. I mean, I think we're already seeing interesting robotics results coming out of like OpenAI, for instance. They had a blog post recently on learning dexterity, where they had a robotic hand which could manipulate a cube in very, very human-like looking ways, trained entirely on reinforcement learning, using basically the same algorithm and setup as they use for Dota 2, which is really cool to see this, this techniques kind of crossing into new applications. But I think definitely robotics is, is a very, very ripe field for, for application, where obviously there's so many things that are going to, that could be enabled, where like, not only is the hardware getting there and in place, but man, it's really hard to, it's just really hard to like hand program these

• 34:11Speaker 02

  things, to do any, to do interesting activities in new environments and new places that they haven't seen before. And like, learning is, is definitely going to be the way to, to get behaviors out of robots that generalize to new scenarios that will let them roam free in the world and be, be useful and, and safe.

• 34:27Chris Benson?

  So, so do you, do you dare to put any, any timeframes around when we'll see different levels of that out there? Or am I, am I getting too out there?

• 34:39Chris Benson?

  The look on his face, by the way, you can't see it listening to this, but he was like, "Oh, God, I just poured gas on the fire right there."

• 34:47Speaker 02

  Maybe what we can do is I'll just record five years, 10 years, 15 years, and then we can go back later depending on which one was right.

• 34:53Chris Benson?

  There you go.

• 34:53Speaker 02

  And just edit it in. How about that?

• 34:54Chris Benson?

  Fair enough. I had to try there. So, a lot of people that listen are kind of getting into it. You know, we have a lot of people that are practitioners, but we also have a lot of people out there that are in the technology world, academic world, business world, who are trying to figure out how they're going to move into this space. And someone who is ramped up, done that, understands what's necessary to be productive in the world, what kind of advice do you have for people in a couple of different scenarios? If they're in a business and they may have a data science team or something, or a group of developers,

• 35:32Chris Benson?

  and they're wanting to kind of retrain in reinforcement learning, deep reinforcement learning, and maybe even robotics that might be an application that they're interested in, what would you, first of all, for that kind of person already out there in the real world, so to speak, what should they be thinking about? How should they start making a transition to take advantage of these technologies at this point?

• 35:56Speaker 02

  Yeah, I think one possible way to approach this, taking on a new field like this that I think could be productive, is to take the time to practice it and do it yourself. So, there's starting to be more and more implementations of these deep reinforcement learning algorithms available out there on the internet. GitHub, for instance, I'm sure is full of them. I mean, I just released the code for Excel, Excel RL. My project for scaling up on the DGX-1 is now out on GitHub.

• 36:26Chris Benson?

  And of course, we'll add a link to that into the show notes there.

• 36:29Speaker 02

  Okay, so this is one such example. But really taking the time to, okay, you know, read off of those, but implement it yourself. Make yourself rewrite it from scratch. And get all the way to the point where you're recreating the learning curve that is published in the literature. Because a lot of what happens with this work is, you know... There's like so many programming concepts that are not too difficult to understand mainly and get in place, but there can be lots of little bugs and lots of little gotchas. And maybe you didn't tune this hyperparameter just right, or maybe there's a detail that's a little bit obscured in the paper and maybe isn't as revealed as clearly, you know, as could have been. And so you need to go back and reference someone else's implementation

• 37:11Speaker 02

  to say, oh, that number needs to be .25 instead of 1.25. And suddenly, suddenly it works now. Oh, this is new. So make yourself go through all the stages, basically, from scratch to reproducing the learning curve that you see in the literature. And then you'll know that you're doing the full stack.

• 37:28Chris Benson?

  Gotcha. And I'm going to ask for one other use case that's probably some commonality between them. But if you are a student, maybe in high school now, and you're looking at trying to figure out what you're going to do as you go into your first university experience, and, you know, what would you advise that, you know, 16, 17, 18-year-old kid who's ready to say, hey, I think this might be what I want to do? And they have the advantage of going through schooling and taking advantage, you know, they're not out there in the world yet. What would you advise them to do? What kind of track should they be thinking about?

• 38:04Speaker 02

  Yeah, definitely taking as many of the computer science and programming classes as you can. Being a good programmer is definitely a plus in this field. But if you're really trying to get into learning research, it's actually not always fully necessary. Like, there's a lot of things to know about learning theory and such that you could also get involved in a lot of mathematics, a lot of statistics. So yeah, I would say, yeah, try to take, early on, take a broad approach, because a different sub-area within this whole, like, learning, field of learning might grab your attention. And before you know it, you've gone a year or two deep into a certain sub-topic, and then you realize, oh, this is the part that I really want to do.

• 38:48Chris Benson?

  Okay. Well, this has been fascinating. Thank you so much for coming on. I guess as people want to reach out to you, make contact with you, you know, where are you in your social media? How would you, how do you like to interact with people out there in the world?

• 39:02Speaker 02

  Yeah, sure. That's easy. You can find me either on Facebook or old fashioned email can do the trick as well.

• 39:10Chris Benson?

  All right. We'll include those as well. So, thank you very much. This was a great conversation. I really enjoyed it. You really made the thought of getting into deep reinforcement learning accessible. And thanks for doing that and sharing with us. And I appreciate the time.

• 39:26Speaker 02

  Good. Yeah. Invite anyone who wants to to come in and jump on the field. It's exciting times.

• 39:30Chris Benson?

  Will do. Thank you very much.

• 39:31Speaker 02

  Thank you.

• 39:34Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor. Go on iTunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do here at changelog because of rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers at linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop

• 40:15Speaker 00

  in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 40:35Speaker 00

  Winner winner chicken dinner. You have won again. Yes. That's why you listen all the way to the end of the shows because we give you previews of what's coming up. And as you may have guessed, we have another preview of our upcoming show called Brain Science. This podcast is for the curious. It explores the inner workings of the human brain to understand behavior change, how about formation, mental health, and the human condition. This show is hosted by myself, Adam Stokoviac, and my good friend, Maryard Reese, a doctor in clinical psychology. It's brain science applied. Not just how does the brain work, but how do we apply what we know about the brain to better our lives? Here we go.

• 41:13Speaker 00

  My wife and I have learned this, this concept of goodwill, right?

• 41:17Speaker 03

  Yeah.

• 41:18Speaker 00

  I can take your feedback or your criticisms in a different light if, if I know that you have goodwill for me.

• 41:26Speaker 03

  Yep.

• 41:26Speaker 00

  Meaning that you're not trying to harm me, that you are for me, not against me. And sometimes change, as we all know, is painful and can be painful. So sometimes the necessary feedback and or criticism that can influence that change can also be painful. But I can accept it differently if I know.

• 41:43Speaker 03

  Right.

• 41:44Speaker 00

  That she or they or whomever is in the scenario with me has goodwill for me, you know? Whereas if you know that they're not for you, then you obviously take it a whole different way. And that's, that's an okay thing. But we often are, you know, in relationship with people that are giving us crucial feedback and we need to have that kind of, that lens, like it was significant in our marriage to understand, hey, I know there are times when you give me feedback, I am not happy about it, but, but I know you have goodwill for me. So therefore I calm down, I listen, I, you know, I take that in and I process it, whatever. But I take it in a different way because I know that she's for me and not against me.

• 42:27Speaker 03

  Yep. One of the key things when it comes to change is a sense of openness. And even relationally, like of going, I need to be able to see some, how somebody else responds or how they're feeling as based on their perspective of what they're going through and not just my perspective of their perspective. And so this goodwill is like, I believe that we're on the same side and that you're not trying to make it harder for me. But so I can understand if I were sitting where you were sitting, had the background that you had, why you would have taken it in that way. And then I can provide an opportunity to clarify or create more connection, even when it doesn't feel good.

• 43:11Speaker 03

  And I honestly think this is so much of what's missing in people's relationships. If I look at relational interactions through the notion of conditioning, wherein I get a sort of hit of dopamine, feel-good feelings because I went to a person, I had a conversation that didn't necessarily feel good, but there was openness on both parties to hear one another's perspective, that it actually then reinforces like, oh, when I go and I have this exchange with people, I feel better. So now I'm going to go and engage with other people and get the feedback, even if I might not like the feedback, because now I'm buffered,

• 43:55Speaker 03

  and I'm not alone in this, and somebody else sees my world.

• 44:01Speaker 00

  That's a preview of Brain Science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us at [email protected]. In the subject line, put in all caps, "Brain Science", with a couple bangs if you're really excited. You can also subscribe to our master feed to get all of our shows in one single feed. Head to changelog.com/master, or search in your podcast app for Changelaw Master, you'll find it. Subscribe, get all of our shows, and even those that only hit the master feed. Again, changelog.com/master.