GANs, RL, and transfer learning oh my!

Daniel and Chris explore three potentially confusing topics - generative adversarial networks (GANs), deep reinforcement learning (DRL), and transfer learning. Are these types of neural network architectures? Are they something different? How are they used? Well, If you have ever wondered how AI can be creative, wished you understood how robots get their smarts, or were impressed at how some AI practitioners conquer big challenges quickly, then this is your episode! Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. Discover.bot https://bit.ly/2OWMyw2 – A digital space for bot developers and enthusiasts of all skill levels to learn from one another, share stories, and move the bot conversation forward. Want to learn more about building bots? Get started with their Guide to Bot Building Frameworks https://bit.ly/2Vxk7qO. GitPrime https://www.pluralsight.com/product/flow – Download GitPrime’s 20 Patterns book https://resources.gitprime.com/books/20-patterns/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog(q1), a field guide to help engineering managers recognize achievement, spot bottlenecks, and debug development processes with data. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: SUBSCRIBE ~> Brain Science https://changelog.com/brainscience — For the curious! We’re exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. RL (Reinforcement learning) Good overview article https://skymind.ai/wiki/deep-reinforcement-learningHuman-level performance in Quake III–Capture the Flag https://twitter.com/DeepMindAI/status/1134158019133136896Practical AI episode about Deep Reinforcement Learning https://changelog.com/practicalai/40Practical AI episode about OpenAI, reinforcement learning, robots, and safety https://changelog.com/practicalai/14PyTorch RL tutorial https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.htmlGANs (Generative Adversarial Networks) Good overview article https://skymind.ai/wiki/generative-adversarial-network-ganThis is not a person website https://thispersondoesnotexist.com/Christie’s AI art auction https://www.christies.com/features/A-collaboration-between-two-artists-one-human-one-a-machine-9332-1.aspxOpenAI Generative models https://openai.com/blog/generative-models/TensorFlow GAN tutorial https://www.tensorflow.org/alpha/tutorials/generative/dcganTransfer learning Good overview article https://blog.fastforwardlabs.com/2018/09/17/deep-learning-is-easy-an-introduction-to-transfer-learning.htmlForbes article on Google AutoML https://www.forbes.com/sites/kalevleetaru/2019/05/28/googles-automl-and-bigquery-ml-the-rise-of-one-click-hyperscale-machine-learning/#6380aca32895Practical AI episode on BERT https://changelog.com/practicalai/22Practical AI episode on GPT-2 https://changelog.com/practicalai/32How to build a State-of-the-Art Conversational AI with Transfer Learning https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313 -NAACL workshop on transfer learning for NLP https://twitter.com/seb_ruder/status/1135223959828537344Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode Cloud servers. Head to linode.com/changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, CI/CD, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front-end web servers that need to be full-duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/changelog. Again, do.co/changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and tag with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:47Chris Benson?

  Welcome to another fully connected episode where Daniel and I will keep you fully connected with everything that's happening in the AI community. Take some time to discuss the latest AI news and we'll dig into learning resources to help you level up your machine learning game. My name is Chris Benson. I am the Chief Strategist for Artificial Intelligence, Hyperformous Computing and AI Ethics at Lockheed Martin. And with me is my co-host, Daniel Whitenack, who is a data scientist at S I L International. How's it going today, Daniel?

• 2:13Daniel Whitenack?

  It's, it's going pretty good. It's been a good week. How about with you, Chris?

• 2:18Chris Benson?

  It's been good. Just the usual busy stuff. I am excited about today's episode. This was, we were talking about it prior to getting on air and you had some great ideas. You want to, you want to go ahead and talk about why we're doing what we're doing?

• 2:34Daniel Whitenack?

  Yeah, sure. So some of the listeners might know that I do industry trainings in AI and other things for companies and sometimes at conferences and that sort of thing. And one of the frequent questions that comes up during those trainings and just kind of conversations about AI in general are questions about kind of the difference between like an AI model. So you might think about having like a convolutional neural net, you know, model or image detection model, like something like that, like a model. Is that the same or different from like things like reinforcement learning, GANs, transfer learning?

• 3:16Daniel Whitenack?

  Are like GANs and reinforcement learning and transfer learning, are those like types of models? Like you have a transfer learning model? Or are they like different sorts of things than models that, you know, like then specific architectures that might be associated with like specific neural units, like recurrent neural networks or convolutional neural networks? So my thought today is that maybe we could just kind of go through a few of these like methodologies or approaches, maybe we can call them, that aren't maybe like models themselves, but are connected to the AI world in some way. So I'm suggesting that maybe we kind of talk through what, what reinforcement learning

• 3:58Daniel Whitenack?

  is, what GANs are and what transfer learning is. How does that sound?

• 4:04Chris Benson?

  Great idea as far as I'm concerned and it's the kind of thing that that we hear in feedback a lot because as people come into the podcast, they're coming for, you know, some people are already experts in the field. A lot of people coming in are brand new and they're trying to understand what the field is about and there's so much to learn these days and it's evolving so. So rapidly that I thought this was a great idea to just kind of go through each of these and just identify or define what each one is. And basically how it works and allow people to kind of get up to speed on those a little bit faster.

• 4:38Daniel Whitenack?

  Yeah, and as we go through these, we can maybe just give a little bit of a sense of what they are, but also give. Some places that they've shown up in the news recently and in the AI news or news in general. And then some learning resources for each one, if you want to kind of get started in reinforcement learning or one of these other things, some, some. Some links that we'll for sure put in our show notes so that you can follow up on those things and start learning them practically. Remember, we're all about. Practicality here at this podcast, so I want to make sure and include those links as well. So which, which one of these you want to get started with, Chris?

• 5:18Chris Benson?

  You want to dive into reinforcement learning upfront?

• 5:21Daniel Whitenack?

  Sure. So again, we're kind of thinking about approaches or methodologies that involve AI models, but might be kind of. Slightly different than a single one end to end model. So with reinforcement learning, where, where have you seen? First of all, where have you seen reinforcement learning being applied, Chris?

• 5:45Chris Benson?

  I think the thing that made reinforcement learning big was its application to simulation and to robotics. And it's, it's, you know, reinforcement learning has been kind of a core technique for, for doing simulation robotics for a long time. And then. In recent years, deep reinforcement learning, which we'll dive into in a few minutes here, has really come in and revolutionized that process itself. But in real life, at a previous employer, I, we were working on a large, multi skilled team with different people specializing. And we had reinforcement learning specialists on the team that were focused on doing robotics. So. I mean, it's definitely real world stuff. It's not just academic and, and it works and that's why they're doing that.

• 6:29Daniel Whitenack?

  Cool. Yeah. So you mentioned that there's kind of this thing reinforcement learning and then there's like deep reinforcement learning and that's kind of where part of this AI or neural network stuff gets plugged in. But in reinforcement learning, one of the main pieces of reinforcement learning is this thing called an agent and this agent takes action. So like in your case, Chris, with the with the robots, you know, what, what, do you remember kind of what this sort of agent or these actions were that the agent was taking?

• 7:05Chris Benson?

  Sure. So you'll have different software components within the robot and they may be integrated with different types of models and they each have a particular job and and for simplicity's sake, let's just say it's about moving the robot around its environment. And so initially, you have to have an algorithm where that the agent is going to use to make decisions based on what's happening to it in the environment that it's operating in. And the way you do that is every time the agent actually takes an action, that changes the relationship it has within the environment, which is called state. And so that might have been, it could have been a good action

• 7:47Chris Benson?

  that's kind of going toward what you're trying to train it for, or maybe not such a good action. And the way that is determined by the person that's training the model is by offering a reward for for the appropriate actions being taken. So you're essentially, you could kind of think of it as, since you know, we're always talking about pets and stuff like that, treating a dog for doing the right thing with positive reinforcement training for those of us who have pets. Same kind of idea is you want to let the agent know, hey, that was good. You get bonus points for this, you know, for doing the right thing, or we're going to pull back something if you don't. And so that's kind of the basic idea. You go through that iteration many, many times to try to get

• 8:30Chris Benson?

  your robot or your simulation, could be a video game, could be whatever, to start behaving in the way that it has been most rewarded along the way.

• 8:39Daniel Whitenack?

  All right, cool. So let me, let me try to, I'm kind of trying to parse through in my mind some of the things that you said, which was really good. So there's this first thing that's called an agent. And that agent can take action. So let's say in a very simple scenario with a robot, maybe the robot can only do two things. It can move left or it can move right. So it, that agent has to determine if it, if based on some external factors.

• 9:08Daniel Whitenack?

  So the environment and its current state, or maybe where it's at, or which way it's facing, if it's to move left or if it's to move right. So it takes in some inputs from that environment, and it's supposed to determine if it moves right or left. Now, I think what, what this, so the agent employs what's called a policy to determine that next action. So let's say that the robot is in this place with these coordinates, and maybe there's other external factors or something. And so it's got a current state, it's somewhere in its environment, and that policy is, is to determine that, that next action based on the current state, whether it's maybe move left, move right,

• 9:53Daniel Whitenack?

  do this, do that. So, um, yeah, I think that there's the, so I think so far, I'm kind of trying to count up the things that reinforcement learning involves in my mind. And we've got the agent, we've got the policy, we've got the state, and then we've got the environment. Now, you mentioned, you mentioned the reward. So, uh, so the reward also is kind of how the model gets feedback. Is that right?

• 10:18Chris Benson?

  Yeah, it's the feedback loop, and, and the purpose of the reward is to shape the policy. So your, your policy is, is being evolved so that at the end of your training, the policy, uh, hopefully always does the right thing that you're training towards. And you're, you're essentially giving it little bumps with the reward to get it there. And so you're trying to shape the policy, which is the strategy that the robot is using to move around. And so, um, there are many different ways of doing that. There are lots of different algorithms that have been used over the years. And, um, you know, and one of those which we'll talk about, uh, has moved into what's called deep reinforcement learning.

• 10:56Daniel Whitenack?

  Yeah, so in my mind, if I'm thinking about this, I kind of see this loop where the agent takes actions. And then at some point in the feedback loop, the environment or so the environment infuses a reward or feedback into the into the agent. Now, you know, one of the things that people ask me kind of is when they're trying to figure out this reinforcement learning thing, they kind of get the idea of, okay, you know, you can give a dog treats and kind of help train it. So this idea of training kind of makes sense. But but they have a hard time picturing kind of where the neural network fits in this in this scenario or where the model fits in this scenario.

• 11:41Daniel Whitenack?

  So one example of that might be like if if our robot has a camera, right? And it's looking at its environment, or maybe it's looking at a simulation. One thing it could do is like image recognition, right? And then based on that image recognition, it could determine whether to move left or or move right or or something like that. So instead of like the image coming in and then the output just being like this object is in this image or not. Then in this scenario, the image would come into the model. And the output of the model would be like the action like left or right or something, something like that.

• 12:23Daniel Whitenack?

  So there's still this kind of neural network model there, but it's tied into this feedback loop where the the output is is that is the action is, am I am I representing that correctly, Chris?

• 12:34Chris Benson?

  I think that's a very good explanation. So a lot of times what the reinforcement learning is acting on, maybe, you know, so maybe a camera, the camera images coming in. And so the type and is a little side note, the type of neural network that is most often used for that. It's called a convolutional neural network or a CNN. And we've had several episodes where we've talked about that, including one that was a deep dive on the technology early on. And so Typically, when we've talked about those, we'll, we'll talk about the convolutional neural network, basically classifying what it sees, essentially putting a label on it with a percentage.

• 13:14Chris Benson?

  If, you know, I'm, I'm, I'm looking at something, is that a horse? Is it a cow? Is it a dog? And there's some level of a percentage of confidence that is being assigned to those, traditionally with CNNs. The difference when you put it in with this particular approach, with reinforcement learning, is you're, you're talking about influencing the policy. So what you really need is the output of that convolutional neural network is what action should I take for my next action? And that's, that way it feeds into how the, the reinforcement learning algorithm is trying to, to do that reward to, to change the policy over time on how the, the model is acting on the environment.

• 13:54Daniel Whitenack?

  That's a great point. So, um, you know, you mentioned the convolutional neural network, but you know, people could see that this reinforcement learning algorithm or approach is, is really just that. It's a, it's an algorithm or approach where within that approach, you could apply a convolutional neural network in your agent to kind of learn a certain policy to take in images and output actions. But people use reinforcement learning for all sorts of other things. And, you know, the, that approach is kind of independent of the specific kind of model that comes in. So, um, you know, you could perfectly well use other type, other architectures of neural networks, you know, recurrent and

• 14:38Daniel Whitenack?

  other things, um, within your agent. But this reinforcement learning loop or approach would kind of still, still be there. That would still be kind of an RL approach to maybe a different sort of problem.

• 14:52Chris Benson?

  Yeah. And, and you raise a really great point there. And we've kind of alluded to it several times. And that is, um, and, and just to kind of back out of the, the specific RL reinforcement learning focus conversation. We're talking about different approaches that have different algorithms or architectures. And when you set aside all these buzzwords, they are, each one is trying to solve a particular class of problem, whether it's, whether we were talking about CNNs looking at images and trying to solve that and reinforcement learning, being able to have an agent take actions that are rewarded to get to the right policy to act in your environment.

• 15:33Chris Benson?

  We've talked about several others and the point is you can use a lot of these together. And so to avoid confusion is if you're working on a particular problem and you might be in reinforcement learning and say, well, it's images that I'm acting, that I need to act on in this case, you would stick a CNN there. And that is just one possibility of how you would combine different types of architectures or algorithms in deep learning to get where you want to go. So it's not always the case that one architecture, one algorithm gets you where you want to go. You may, a little bit like Legos, you may plug some of these together. Just wanted to clarify that in case there was any confusion.

• 16:12Daniel Whitenack?

  Yeah, I appreciate that. So maybe before we move on to the next thing, let's maybe just think about, okay, where is reinforcement learning showing up in kind of AI news? And what are some learning resources that people can can look into if they're trying to learn reinforcement learning, if this has piqued their interest? So one of the things that I've seen in the news recently, so very recently are, you know, and people have probably seen related things on Twitter or wherever, is DeepMind released a reinforcement learning approach to, that resulted in human level performance in a video game, Quake 3.

• 16:55Daniel Whitenack?

  And so this is pretty cool where, you know, a lot of these reinforcement learning techniques have been applied to kind of fun things like, like video games and things like that. If you're, if you're more interested in reinforcement learning, we've actually had a couple of episodes, so episode 14 and episode 40 of. Practical AI that talk about certain applications of reinforcement learning with a little bit more explanation. And as well, one of the learning resources that I found that looked really good on this front is there's actually an official PyTorch tutorial on reinforcement learning. And we'll make sure and link that in our show notes.

• 17:37Daniel Whitenack?

  If you want to go ahead and dig a little bit deeper into reinforcement learning and actually try some things out on your own.

• 17:51Speaker 00

  This episode is brought to you by discover dot bot. Learn everything there is to know about bots at discover dot bot slash practical AI. Discover that bot was built by Amazon Registry Services as an online community for bot creators and makers of all skill levels to learn from one another, to share stories, and they regularly publish guides and resources to answer questions like how to set up payments to your bot, how to stop shopping cart abandonment, what KPIs are worth measuring, how to write an engaging chat bot dialogue. You can even register dot bot domains there. Learn more and explore this huge library of bot resources at discover dot bot slash practical AI. Again, discover dot bot slash practical AI.

• 18:39Chris Benson?

  So Daniel, now that we've covered reinforcement learning, what do you say we dive into again generative adversarial networks?

• 18:48Daniel Whitenack?

  Yeah, that sounds, sounds good. So this sounds kind of, kind of scary adversarial things, Chris.

• 18:56Chris Benson?

  Are you gonna talk about terminators now? Are we, are we about to all die?

• 19:01Daniel Whitenack?

  Yeah, is it, uh, yeah, I don't know. How adversarial are these, are these networks, Chris?

• 19:08Chris Benson?

  Well, I'd say that, that they're adversarial with each other, which is the whole point, which is why they're calling, calling it that. It is a, it is a really, really interesting innovation that came about in 2014. Where a, one of the The very famous figures in this space, whose name is Ian Goodfellow, was put out a research paper with several of his colleagues and about generative adversarial networks. And what that is, is basically you have two different types of neural network architectures designed to work together,

• 19:50Chris Benson?

  or more specifically, against each other to try to get where you want to go. It's a way of often creating outputs that are creative from this type of technology. Pretty interesting stuff. I've seen it used for lots of different use cases, too.

• 20:09Daniel Whitenack?

  Yeah, you mentioned kind of the creative element of this. You know, one of the places I think this has received a lot of attention for good and bad, and in some ways, is like in generating images. So, you know, there's kind of examples of creative uses, like generating specific artwork, or generating things in the style of certain other things. There's also examples of generating kind of pictures of fake people and all of these things. So this all involves this kind of generative element of GANs, or generative adversarial networks. So you mentioned that there's kind of two elements of this methodology, Chris.

• 20:52Daniel Whitenack?

  So there's obviously some sort of generative element of this, which is what people call the generator of this approach. What's the other thing that's involved here?

• 21:04Chris Benson?

  So you have the generator, and then you have, which is the one part of this combined architecture. And then the other side, the other algorithm, is the discriminator. And so essentially, the generative architecture, or model in this case, that's being trained.

• 21:21Chris Benson?

  Is is creating outputs that are inputs for the discriminator. And the input on the discriminator side, it's essentially trying to to classify which ones are real and which ones are fake. And it has those mixed in with the ground truth data set. So that if you're trying to create images and you're and and this might be something that's completely new, the discriminator has access to a data set that has a bunch of real images that are the the ground truth that you're training against. It is it is that baseline data set. And the generator is is also looking at those, but it's creating images that are meant to

• 22:04Chris Benson?

  look like whatever it is that the that the data set is representing. And so it might be, I'm just making this up, might be cats, since we like to talk about cats on the internet. And so you might have a bunch of images in the actual data set of cats. And then the generator is trying to create new images of cats and slide that in with the ground truth data sets. And it's up to the discriminator to determine which ones are real and which ones are not and put a percentage on that. And so there's this feedback loop between the two to where the discriminator is making its choices and giving that feedback to the generator. And in turn, the generator is is learning from what the discriminator is able to do,

• 22:46Chris Benson?

  right or wrong, and produce more and better images. So it's a neat thing where this the adversarial side is that these two models are literally trying to beat each other. One analogy could be a policeman against a counterfeiter with the generator being the counterfeiter and the discriminator being the policeman. And they're each trying to do their thing and get better and better at it. And by doing that, they they both get better.

• 23:13Daniel Whitenack?

  Yeah, I've also heard the analogy kind of being the generator is the artist and the discriminator is the art critic, trying to examine the output of the generator. So similar to, so in some ways similar to reinforcement learning, there's this kind of overall scaffolding in which, in this case, two models are interacting. So there's more going on here than just kind of one end-to-end model. There's a couple of things happening here and there's this loop between the generator and the discriminator. Now, each of these pieces, so the generator itself and the discriminator, each of them

• 23:59Daniel Whitenack?

  could be a single neural network. So the generator might be a neural network that takes in, for example, some random inputs and generates an image on the output, like an art image or something like that. So its input might be, you know, some kind of random input like that and the output might be what you're trying to generate. The discriminator, on the other hand, it's taking in a whole bunch of images and it's kind of like a classifier. So it may just be another type of neural network that is trained to be a classifier,

• 24:40Daniel Whitenack?

  to classify as like, you know, human-generated or computer-generated or good art or bad art or, you know, something like that. So it's a classifier that classifies that set of images. So you've kind of got two "models" here. And that's where the neural networks are fitting in here. Of course, there are specific types of generative models that work particularly well in this framework. For the image case, DCGAN is fairly popular. There's an OpenAI article that we'll for sure link in our...

• 25:23Daniel Whitenack?

  In our show notes that kind of describes some of the generative models that are used in in GANs. But maybe as we kind of look a little bit more at GANs, maybe we can talk about where they've been showing up in the in the news. So where have you been seeing GAN show up recently in in AI news or news in general, Chris?

• 25:47Chris Benson?

  Well, one of the things that we have talked about on a couple of previous episodes was that there was a portrait that was created by a GAN that Christie's Auction House sold at auction for $432,000. And it really, nobody, including the people selling it, were expecting that. And that was, it was for a unique and original piece of artwork that a GAN created. But, you know, and that really, suddenly, it really shook that industry, you know, because it was it was one of those instances that nobody saw coming. But we're also seeing it in all sorts of other places, creating original music. We've talked about that in the past.

• 26:28Chris Benson?

  It's, I know that Ian Goodfellow uses it in the security industry, which is completely different. And so there are, there are so many different use cases where you want some sense of originality or creativity to to play into it. And and using GANs to actually generate the stuff is regardless of what the medium is, is is becoming a better and better option for doing that.

• 26:53Daniel Whitenack?

  Yeah, I know one thing that I've seen, but kind of even people that have, so like my, my brother in law, who isn't involved in the in the AI industry at all. I mean, he's kind of interested in in tech things, but not really a programmer or anything like that. He even showed me this one website. So people are probably familiar with this, that it's thispersondoesnotexist.com. Have you seen this, Chris?

• 27:18Chris Benson?

  I have. And it's gotten better and better over time.

• 27:21Daniel Whitenack?

  Yeah, it has gotten better over time and this website, if you're not familiar with it, you can go there and basically all it shows you is a picture of a person. But and it looks, you know, exactly like a real person. So it's kind of, you know, it takes you off guard when you realize that this person does not exist. In other words, this picture of this person, which looks, you know, real in every way, is a picture of someone that is completely generated. So everything about that that picture is generated using this this type of methodology. And of course, that's really interesting and kind of amusing in certain ways.

• 28:03Daniel Whitenack?

  But also, it's kind of, you know, concerning in other ways. Like, of course, you know, everyone is concerned with fake news and fake content on the internet now. So there's definitely a concern with these around if what you're looking at is actually is actually real or not. I remember talking on one episode, I forget which one, about, you know, there's actually people out there that will create a fake persona, fake picture for you for Instagram to be kind of your company's influencer, you know, on the internet. So there's a question here of like, you know, how real are the things that we're interacting with?

• 28:45Chris Benson?

  Yeah, so it's interesting. One of the responsibilities I've taken on at Lockheed Martin is contributing to developing AI ethics and figure out, you know, not just about what we do, but about how we react to what's happening in the world. And there are obviously bad actors out there. And so one of the things, you know, GANs are so powerful. And as a quote, Facebook's AI Research Director is very well known in the industry, Jan LeCun, and he referred to GANs famously as the most interesting idea in the last 10 years in machine learning. And obviously Ian Goodfellow and his partners that were working on this

• 29:27Chris Benson?

  are among the brightest minds in the field. So there's so much. Potential for the use of GANs, both wonderful, interesting and some bad use cases as well. So it's, it's, it's the advent of GANs has changed the conversation in terms of AI safety and AI ethics and, and how these technologies can and should be used.

• 29:51Daniel Whitenack?

  Yep. So if, if people are interested in diving a little bit more into GANs, there's definitely some good resources out there. So for, for reinforcement learning, we, we mentioned there's a PyTorch tutorial and there's a bunch of other tutorials out there for that. But there is a really great TensorFlow tutorial for GANs. So we'll make sure and link that in the, in the show notes. Actually, if you, if you go to that tutorial, they have some nice, some nice pictures as well, talking about the generator and the discriminator and cat images and, and all of those good things. But then they walk you through all of the code with Keras and TensorFlow to, to actually create this, this

• 30:36Daniel Whitenack?

  GAN. And they have a link to kind of pop that up in a Google Colab notebook so that you can go ahead and, and get started with GANs.

• 30:53Speaker 00

  Well, hello there, listeners of Practically I. How are you? This is Adam Stokoviac. If you haven't heard yet, we're launching a new show called Brain Science. It's a podcast for the curious. Are you curious? Because if so, we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and what it means to be human. It's brain science applied, not just how does the brain work, but how do we apply what we know about the brain that can transform our lives? Learn more about the show and subscribe at changelog.com/brainscience. Until then, here's a preview of episode one, where we talk about the fundamentals of being human.

• 31:34Speaker 03

  We're also all designed to be in relationship. We are fundamentally hardwired to have social groups and, and this sense of attachment. And because I'm sort of a geek when it comes to research, what researchers have found is that attachment, which that's what we label how we relate and connect with others, attachment is 100% learned.

• 31:59Speaker 03

  Which means our genetics don't actually contribute to how we learn to stay in proximity with other people. And with that, that we all develop ways to manage the threat of the loss of a relationship. But nobody gets to opt out of going, I need to be in a relationship with others. I mean, think about it within the context of the prison system. Like, why is it that the punishment for prisoners when they don't fall in line is isolation?

• 32:29Speaker 00

  Hmm, yeah, that's true.

• 32:31Speaker 03

  Right? That wouldn't be significant if in some way that doesn't actually harm our brain.

• 32:36Speaker 00

  It's almost like we need to have that echo from another human being to let us know that we

• 32:40Speaker 03

  Yeah.

• 32:41Speaker 00

  were there, or we're alive, or just some sort of feedback loop. I'm not really sure how to describe that.

• 32:46Speaker 03

  Well, it really is this sense of being with, right? Like, I can't fight battles on my friends' behalfs or on my kids' behalfs, right? But the simple fact that I know of what's going on makes a difference. Because I would contend it's sort of like, I help them hold that weight emotionally. And so that actually leads me into the third thing. And the third thing that I would say in regards to the fundamentals of being human is that we all struggle.

• 33:16Speaker 00

  Oh, yes.

• 33:17Speaker 03

  Right?

• 33:18Speaker 00

  Big time.

• 33:19Speaker 03

  And that, you know, we don't always get to pick the way in which we struggle, but we all struggle.

• 33:26Speaker 00

  Well, if you like what you hear, you should go to changelog.com/brainscience. The show is not out yet, so don't get too excited, but you can subscribe and be notified as soon as the show launches. Once again, changelog.com/brainscience.

• 33:59Daniel Whitenack?

  Okay, lastly, so we talked about reinforcement learning, we've talked about GANs. Let's go ahead and jump into this last thing that I hear people asking about, which is transfer learning. And we've certainly touched on this in previous episodes, but we haven't kind of put it in context like we're putting in context these other things. So, transfer learning is another one of these kind of methodologies or approaches that's used in AI by AI practitioners to do a bunch of different things. But transfer learning isn't kind of a model in and of itself. It's another one of these approaches. And I would say

• 34:41Daniel Whitenack?

  in comparison to GANs and reinforcement learning, it's actually one that I've leveraged pretty heavily in my own work. I haven't touched as much, or reinforcement learning and GANs haven't touched my life as much as transfer learning has. I think transfer learning is something that pretty much all AI practitioners should be familiar with and utilize heavily. What do you think, Chris?

• 35:08Chris Benson?

  Well, I would say that pretty much all AI practitioners have utilized it whether they realized it or not.

• 35:14Daniel Whitenack?

  Yeah, that's probably true.

• 35:15Chris Benson?

  If not before, certainly when they were learning how to do this and they were initially going through and creating their first models, they were almost certainly using transfer learning, even if they didn't realize it. It's kind of the secret weapon of kind of getting yourself going. And it's probably almost always used in certain types of use cases, such as computer vision. And as we get into defining what it is, it'll become apparent why.

• 35:43Daniel Whitenack?

  Yeah, and it's definitely impacted the natural language processing or NLP community very heavily. And there's been a lot of efforts in that direction recently. I know on one of our very first episodes, we had the guys from Machinebox on the episode.

• 36:03Chris Benson?

  Yeah, that was episode two.

• 36:05Daniel Whitenack?

  Was that the first? Oh, that was episode two, the first one with guests.

• 36:08Chris Benson?

  It was the first one with guests, that's correct.

• 36:10Daniel Whitenack?

  Yeah, so Machinebox has this really great service that you can spin up that will do facial recognition. And really, all you have to give it is like one or two images of a person's face and it automatically kind of updates the model and does really great facial recognition. And of course, it's not just utilizing one or two images and training a whole neural network on two images. That just wouldn't work. So there's something else being leveraged under the hood. And as Chris mentioned, in that computer vision context or NLP, a lot of times that that thing is transfer learning.

• 36:51Daniel Whitenack?

  So at a high level, what, how do you think about transfer learning, Chris?

• 36:57Chris Benson?

  So the way I think about it is when you're, when you're creating a model, you, it's, it's not, you don't just go and do it and it's done. It is an iterative process. And so, you know, kind of going back to the basics of what deep learning is, what a deep neural network is, is you have a series of layers and each of those layers is responsible for generalizing something, understanding something, and they tend to build on themselves. So in the context to make it real of computer vision, you may have a deep neural network and the early layers are

• 37:37Chris Benson?

  You know, there to recognize just simple things like lines and or corners and things like that. And you tend to build those features up to where now it recognizes after it combines some of those together, what lips look like, or what an eye looks like. And then, you know, you go up a little bit and it starts to recognize how you put those different features together and make it a face and, you know, and then then a full head. And and so each one builds upon the other. So the really cool thing about this is, let's say that you need to go recognize something and maybe some of those baseline features like recognizing at the lowest level, recognizing lines and curves and such, obviously, in every image, you're going to do that. So if you have a model that's really good at doing that already,

• 38:21Chris Benson?

  taking that and if it's getting close to human recognition, or maybe animals or certain common objects, you can move higher up the stack. And then, at whatever point, the purpose of that preexisting model might diverge from yours, you can take those layers that that were consistent with what you're trying to achieve and build upon those. And since they were built with a with a general data set that is different from the data that you're about to train it on, your new model is more likely to generalize better as well, since you have a more diverse data set by definition, since you have pulled in a partially trained model from somebody else's data set. And so it's kind of like we're all standing on the shoulders of giants. You build upon what other people have built and you get it,

• 39:05Chris Benson?

  you can take that preexisting model that might work really well up to a point, and then you take your specific data with your specific data set, a set of images about something that you care about, and then fill that out. And you end up being able to get a very robust model that does something very useful with much, much less training, and it's a lot less brittle, since it has a broader data set to base it on.

• 39:30Daniel Whitenack?

  It's kind of like, you know, most programmers don't write every single line of code from scratch when they're, when they're creating a new application. There's a lot of copy and paste that goes on because they've done this before. They've done that before. They've created this sort of service and it just needs to be kind of slightly different this time around. And so they, they never start from scratch. They kind of copy a bunch of things over. It's very similar here in the sense that like, yes, you can, in many cases, take a model that has not been trained on any data yet and train it to do a certain task. Let's say that we want to translate text from English to Hindi.

• 40:15Daniel Whitenack?

  So what we do is we get a parallel corpus. So we've got a bunch of examples of English phrases and then a bunch of examples of the corresponding Hindi translation of that. And we train a model on English to Hindi so that when we put in an English phrase, what we get out is the corresponding Hindi translation. So that's kind of like training from scratch. That would be like, you know, creating every, in my, in my analogy, creating every line of code from scratch. We're, we're initializing all of the weights and the biases of our model, all of the parameters of our model from scratch. So from some random seed or all starting at zero or whatever that initialization is. But we're, we're using that English to Hindi corpus to train all of those parameters

• 41:01Daniel Whitenack?

  of the model from scratch. Whereas now, let's say after we've done that, we don't want, later on in our work, we don't want a model that is trained to translate English to Hindi, but we want English to Urdu, which is a related language. And, uh, you know, this means that we could do one of two things. We could either get another huge corpus of like English to Urdu data and train from scratch again. Or we could leverage the knowledge that we already that we already created in that English to Hindi model. So we could take that model and all of the weights and parameters that we trained for English to Hindi.

• 41:42Daniel Whitenack?

  And then we could just kind of slightly modify it or fine tune it by retraining those on the new data set. Maybe a smaller amount of English to Urdu language. So this has been widely used in NLP because in a lot of cases, maybe you want to take a pre-trained model that's very general. So it's it's applied to maybe do translation for all domains. And you want to really fine tune that for a specific domain of of text or of of some content. And so what you'll do is you'll fine tune or slightly modify that on this new data set. So there's kind of this initial pre-trained model.

• 42:26Daniel Whitenack?

  And then there's the fine tuning of that pre-trained model on on a new data set. So it could be on a new data set, or you might fine tune it by kind of adding additional layers to it as well.

• 42:38Chris Benson?

  So to kind of bring this back full circle on that, if if any of our listeners have taken classes from, you know, maybe Nvidia's Deep Learning Institute, or maybe Coursera, whatever, on specific things like NLP or computer vision or something. Chances are in that class, one of the things you did when you started creating those the models for your class was they would have you go in and select an architecture to base that on. And that itself is transfer learning. And you're going to find libraries of these models that are pre-trained that you can build upon in all the common frameworks out there. TensorFlow has them, PyTorch has them. It's it's it is truly the most common way, certainly to get started or to do or to build upon.

• 43:24Chris Benson?

  I have, in my own experience, I have more often than not seen people use transfer learning in their work than than start from scratch and try to build things completely further ground up. You would have to do that if there is not the right type of model that you can build upon. This is, this is normal stuff. This is what we do. And I thought your analogy, Daniel, for in terms of using libraries, if you're a programmer is, you know, you're truly using lots and lots of code that other people have built. Maybe a lot of that's open source, maybe some of it's proprietary, but you're still using those APIs to build whatever thing you're building, whatever application you're building. And, and it's exactly, that's a fantastic analogy you gave on, on matching it up to, to transfer learning in ML.

• 44:07Daniel Whitenack?

  And, and another thing is like, in a lot of cases, you may just not have access to the data that you need. So for example, you know, you may not have access to the huge number of face images that someone else has trained a model, a facial recognition model on. So they might have, you know, 200 gigabytes of, of images that they trained their model on, and you only have a handful. But that doesn't mean that you're kind of totally out of luck, right? Because a lot of people have released these sort of pre-trained models for facial recognition and, and other things to where like, like we were talking earlier with Machine Box.

• 44:47Daniel Whitenack?

  You might just be able to utilize that pre-trained model and update it with a couple of new images or a handful of new images. And you know, that kind of removes the burden on you to gather all of these large sets of data, maintain them, update them over time. Run really long jobs to train these models using GPUs, which is really expensive. And so it can also be kind of a operational and cost-saving strategy, strategy as well.

• 45:17Chris Benson?

  Absolutely.

• 45:18Daniel Whitenack?

  So I've seen transfer learning in the news recently in, in a few different places. One, one of the places, as I was searching around in, in preparation for this, for this episode, I saw a recent article from, from Forbes about Google's Auto ML. It mentions actually transfer learning in, in that article, which, you know, I thought was, was reasonably technical for, for Forbes. But, uh, but yeah, they, they talk about how Google's Auto ML services are using transfer learning, leveraging. Transfer learning to allow people to create these sort of customized models, maybe for translation for their specific domain like

• 45:58Daniel Whitenack?

  for law or for medicine or for, you know, customer service or something like that. So it's definitely being utilized in a lot of production services. Where else have, have you seen transfer learning kind of

• 46:15Daniel Whitenack?

  come into, you know, recent news or recent releases of things, Chris?

• 46:20Chris Benson?

  Well, we've, we've had several episodes that made reference. You know, some of the, the algorithms that we've talked about were BERT in episode 22. We talked about GPT, yeah, GPT two in episode 32. And, and those are models that you can build upon as well. And, and, and that's really, I think it's really important to note that this is kind of the standard way you start thinking about a problem is you go out and, and, and look and see if there is something out there that makes sense to build upon. And it's almost the, the, the route into machine learning today. And so a lot of these great institutions are, are in fact, building things that all of us can then take it thereafter

• 47:03Chris Benson?

  in the tool of whichever one we want to use and, and apply that.

• 47:06Daniel Whitenack?

  Yeah, I think the, the sort of BERT and GPT two and other large scale language models are good examples. So for example, with like a, as we talked about BERT or GPT, GPT two and other episodes, you can basically take that pre-trained model. And in a lot of cases, how you would quote unquote, fine tune it is by adding a layer that would do named entity recognition or adding a layer that would do text classification or something. And keeping all of that knowledge from the BERT or GPT two embeddings on the front end of your model. So you're kind of only adding or, or changing it a little bit, but you're kind of leveraging all of that knowledge that Google or OpenAI has, has already built into it for you.

• 47:50Daniel Whitenack?

  So a couple of things that I've seen even over the past couple of days, if you're looking to get hands-on with transfer learning, there have been a couple of resources that have been published that are, I think, are really great. So one of those is blog posts from the Hugging Face team. So if you remember on episode 35, we had Clem from Hugging Face on. He had some really interesting and fun stuff to talk about, but their team has released this tutorial on how to build a state-of-the-art conversational AI with transfer learning. And I think that builds on some of these large-scale language models.

• 48:34Daniel Whitenack?

  And then even today, I just saw there was an NAACL workshop on transfer learning. So that's the computational linguistics conference that's happening, I think, even right now up in Minnesota. And there was a workshop there and they released all of the code and Colab notebooks and information, I think, slides from that tutorial. So we'll make sure and link that in the show notes as well if you want to get hands-on with transfer learning. But yeah, I think that, so talking through these things with you, Chris,

• 49:14Daniel Whitenack?

  has definitely helped categorize some of these major components of AI methodologies in my mind. I hope it has for you as well.

• 49:24Chris Benson?

  It definitely has. I hope that we get feedback from our listeners. That certainly, I know when we were talking about doing this before recording this episode, we were hoping that there might be some of the confusion out there that we could alleviate. And we would love to hear back from people through changelog.com/community or on our LinkedIn group, which we invite people to join as well. You can find it out, you can search for Practical AI Podcast on LinkedIn and do that. But we'd love your feedback to know if these were helpful. Are there other specific questions we left unanswered? And are there other topics that you would like us to cover in future shows?

• 50:06Daniel Whitenack?

  Awesome. Well, thanks for, thanks for talking through these things with me, Chris, and I'll look forward to hearing from our listeners out there of how they're using these techniques. And if we messed anything up or misspoke, or if there's additional great resources that you know about on this front, please, please reach out and we will, we will talk with you again soon.

• 50:30Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our errors before our users do here at changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers at linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whiteknack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our

• 51:13Speaker 00

  weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.