Making GANs practical

GANs are at the center of AI hype. However, they are also starting to be extremely practical and be used to develop solutions to real problems. Jakub Langr and Vladimir Bok join us for a deep dive into GANs and their application. We discuss the basics of GANs, their various flavors, and open research problems. Sponsors: DigitalOcean https://do.co/changelog – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Jakub Langr – Website http://jakublangr.com/, GitHub https://github.com/jakubLangr, LinkedIn https://www.linkedin.com/in/jakublangr, X https://x.com/langrjakubVladimir Bok – Website https://vladimirbok.com/, GitHub https://github.com/vbok, LinkedIn https://www.linkedin.com/in/vladimir-bok, X https://x.com/vladimirbokChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Ian Goodfellow NeurIPS paper 2014 https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdfGAN TTS https://arxiv.org/pdf/1909.11646.pdfBooks “GANs in Action” by Jakub Langr and Vladimir Bok https://www.manning.com/books/gans-in-actionUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at forty bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/changelog. Again, do.co/changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AIFM. And now into the show.

• 1:47Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International. And I'm joined as always by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are you doing, Chris? Doing well today. How's it going, Daniel? It's going pretty good. I've got, you know, models training. And messy data to work with. So as good as any day could be, I guess.

• 2:14Chris Benson?

  What more could you ask for?

• 2:16Daniel Whitenack?

  What more could you ask for? I guess in certain scenarios, you might ask for interesting models, which is what we've got to talk about today. On previous episodes, I know we've mentioned GANs a few times and we've talked about some of the specifics, but not a whole show devoted to them. And so we thought we'd dig into this topic a little bit more. And one of the ways that we thought we could do that was to get some experts and that's what we've done. So we've brought in Jacob Lunger and Vlad Bock, who are the authors of the book GANs in Action. And they're going to help us parse through all of all things GAN. So welcome, Jacob and Vlad.

• 2:57Speaker 03

  Thank you.

• 2:58Daniel Whitenack?

  Yeah, thank you for having us. So before we begin and jump into GANs specifically, let's jump into each of your backgrounds and hear about how you ended up where you're at. So maybe Jacob, could you start things out?

• 3:10Speaker 03

  Sure. So I was sort of working in machine learning since about 2013. Obviously, back then, GANs weren't a thing, but I sort of fell in love with the field. And I was sort of firstly curious about all the latest and greatest things going on in research. And as I was sort of following conferences and like the researchers that I really respect and admire, I came across this thing called generative adversarial networks sometime in 2015, I believe. So that was quite soon after the original paper by Ian Goodfellow came out. And I just sort of fell in love with the technology and the whole idea that I'm sure we'll get into later, just kind of really made

• 3:52Speaker 03

  sense to me. So I was from that point on really curious, but only sort of, you know, as a part time hobby. And then over time, things sort of started picking up. So I started writing more blogs, blog posts about it. And then eventually, you know, Manning approached me with a book offer. So I think that was where it really started to take a more coherent form. And I started dedicating more and more time to it, to now where it's taken over all of my life, you know, I'm working full time and with GANs and, you know, doing these types of, you know.

• 4:26Speaker 03

  Communication and outreach type of thing, which I think, you know, is just a testament to how far GANs as a field have gone. So it's kind of ramped up a little slowly, but you know, now it's kind of really, you know, took over every aspect, which is great because I think I really got to see the field from its inception, which is not something you see every day.

• 4:46Daniel Whitenack?

  Yeah, it's been a pretty quick ride, I guess. So you're saying like 2015 was around the time that, you know, Ian Goodfellow came out with with a paper and and that stuff kind of started getting momentum. Is that right?

• 4:58Speaker 03

  Yeah, so I think the original paper was presented at then NIPS 2014. And then I think, you know, then you saw like a slow trickle of papers, which kind of eventually turned into like an avalanche by like 2016, 2017.

• 5:14Daniel Whitenack?

  So Vlad, I was wondering if you could give us a little intro about yourself as well?

• 5:18Speaker 04

  Yeah, totally. I studied computer science, so machine learning was always one of my interests, both personally and professionally. And after college, after a brief stint at a Y Combinator startup where I worked as a data scientist, I joined Microsoft. And Microsoft has an arm called Microsoft Research, which is essentially it's like an R and D division. It's effectively the Bell Labs of our time. And then I got involved with a research project where we used GANs along with my team. And it was just fascinating to see the margin by which data generative tasks, this technique has exceeded everything

• 6:00Speaker 04

  else that used to be the state of art. So it was truly this like stepwise improvement that is rarely seen in, or it used to be rarely seen in machine learning. And from there, I stayed involved with the field.

• 6:13Daniel Whitenack?

  And now are you working, you know, in a practical sense, day to day with this technology?

• 6:19Speaker 04

  A little bit here and there, although I must say that when it comes to practical applications of most vast measures of machine learning and deep learning techniques, it's still very much in the supervised machine learning area. And less on generative tasks. So when it comes to my day-to-day job, then GANs are almost no involvement.

• 6:39Daniel Whitenack?

  Gotcha, yeah. And maybe we can get into some of those practicalities a little bit later, but to kick things off into GAN world, and I'll let you guys choose who wants to take on certain questions, but maybe one of you could just give us a brief, like, what makes a GAN a GAN? How is it different than what we might think of when we think of a "normal neural network" or a, you know, "normal machine learning model," whatever that is?

• 7:07Speaker 03

  Sure, I could give that a whirl. So I think the two areas that I would highlight is that, first of all, from like very high level, GANs generally live in the unsupervised world, which Vlad alluded to already, that there's this distinction between supervised and unsupervised. So most GANs exist in a world where you don't actually need any training labels, which is where the supervision comes in. Rather, the most generative models learn from the data itself. So all it needs is just raw data and it manages to reconstruct it. So, you know, the prototypical example being human faces, and just by feeding GAN

• 7:48Speaker 03

  a whole bunch of human faces, it will eventually learn to reconstruct completely novel faces that are not in the original training set. So that's, I think, idea number one, is that it's unsupervised. And then idea number two is, so the model itself effectively uses most of the time two neural networks that compete against each other, one being called the generator, one being called the discriminator. And the generator is almost like an amateur painter or something like that. And it tries to basically take some inspiration, generate a new image, and then the discriminator is like an art critic who says, you know, "This is a good picture," or "This is not a good picture." And through the back-and-forth process,

• 8:31Speaker 03

  they both get better over time at generating and then telling apart real from fake. So at the end, you can have a generator that's pretty good at producing realistic images. So that's kind of like the high-level summary. Anything to add, Vlad?

• 8:48Speaker 04

  Yeah, totally. I think it's a great intuitive description. I would just add some of the technical details in which GANs differ from traditional neural networks. And a core of it is the training procedure, where a traditional neural network is effectively an optimization. You have a very complex loss space, in which you are trying to minimize some loss function, which is the objective. So essentially, there is some measure of an error, or how far the neural network is from its objective. And then the training procedure just uses calculus to minimize that objective. With GANs, since there are two neural networks, the training process can be better described

• 9:30Speaker 04

  as a game, rather than an optimization, which has far-reaching implications on the training process itself and as the outcome of the network.

• 9:39Daniel Whitenack?

  So, yeah, in this game, and maybe this is related to Jacob's discussion of being unsupervised as well, is the idea that, like, you have these, let's take the faces example. So you have real faces, or real pictures of faces, and essentially, those are kind of labeled in the sense that those are real faces. And then anything that comes out of a generator model trying to generate real faces, or trying to generate realistic faces, those are kind of automatically labeled as fake. So there's kind of this fact that, you know, everything you pump in is real?

• 10:19Speaker 04

  Yeah, absolutely. So actually, there is an active debate among researchers in classifying GANs. Because when it comes to the training process, then exactly as you alluded to, they can be seen as a supervised machine learning. Because you do have implied labels in real or fake. And there are also some GAN models that actually have explicit labels as well. But even the traditional core GAN, there are implied labels. However, when it comes to the training setup, then a more correct description is unsupervised, because there is no need for humans to instinctively label this data. So usually the bottleneck in machine learning is access to a large dataset with labels.

• 11:03Speaker 04

  And GANs and the GAN paradigm helps to solve for that problem by essentially having the labels implied. There is no need for a human to go image by image and label which one is fake and which one isn't. Because that comes from the very nature of the setup of the GAN model. When we look at it from a practical standpoint, then they can be seen as a unsupervised model. Although when we look at the particulars of the training process, then we are measuring a distance to an explicit label. So from that standpoint, the training can be seen as supervised in a way. So they are definitely in this gray area between supervised, unsupervised.

• 11:46Speaker 04

  And the Google researcher Francois Chollet, who developed the cross Python programming deep learning library. He refers to some techniques like this as self-supervised as well.

• 11:57Daniel Whitenack?

  So you mentioned that there's these two models that are involved. And it makes sense to me kind of now what you're talking about in terms of unsupervised and how to think about that. I guess maybe in training each of those models, does each one still have the idea of loss? In terms of what it's trying to generate? Or are they completely different? You know, the training of each model individual, is it completely different in some way?

• 12:25Speaker 03

  Well, I was just not 100% sure that I understand the question. But obviously, the two networks have different inputs, right? So for one of them, it is the latent vector, right? So one of them is just kind of like some sort of typically a random sample from some somewhere between like 100 and 500 dimensional vector. And then it's just like a, you know, the same way that any latent space would be. For those who are familiar, it's kind of relatively meaningless vector. Though there are some qualifications to what do you mean by meaningless specifically in this case?

• 13:05Speaker 03

  But it's just a vector of random numbers and then uses a process to basically get to transpose convolutions or deconvolutions to a size of the image, right? So it reshapes that vector through learned transformations. Something that looks like an image ideally coming from, you know, the data, original data distribution. And so of course, like that is how you, you know, evaluate it as well. And the way that it gets the feedback is through the discriminator, which then tries to effectively, like we discussed, label images that the generator produces, right? And every time the generator manages to fool the discriminator,

• 13:45Speaker 03

  it gets a lower penalty than the discriminator who gets penalized more because it got something wrong. So it's this iterative process, but of course, like there, you know, in the original formulation, like Vlad lead to, it's, it could be thought of as a game. So the two networks are basically set directly against each other. It's a zero sum game, right? One has to lose in order for the other one to win. Um, but it turns out that even that works relatively well. But in practice, people use more complicated loss functions that just have better like numerical properties. But the original formulation kind of gave it some solid theoretical grounding. So people were sort of more willing to accept why that works.

• 14:30Speaker 03

  And then just kind of use these numerical tricks to make the training more stable or give it some other nice properties. And we can talk about some of the more advanced variants later. But that's basically what the generator is doing. And then the discriminator is basically just a classification algorithm, right? So, you know, you can think of it as a real or fake detector like we discussed. So the training process there is more natural to what most people are used to thinking.

• 14:56Chris Benson?

  So I guess could we turn to maybe a couple of examples and talk a little bit about, you know, what makes GANs kind of useful or interesting for certain tasks? And what are kind of the range of tasks that you could use them on? And, and kind of if you could maybe throw out a couple of different examples about how GANs would be implemented to solve a particular problem more conceptually and stuff to give people a sense of how to fit this concept into their thinking as they're, as they're learning this topic?

• 15:28Speaker 04

  Absolutely. So GANs are, as their name suggests, generative adversarial networks. They are well suited to generative tasks, which is where you generate synthetic, yet realistic looking piece of data. And GANs have been particularly well suited to generating fake imagery.

• 15:48Speaker 04

  So you may have seen in media that there are fake images of human faces that are at a photorealistic quality. Or even fake videos of statements by famous people that were never made, and yet the video footage looks as if a Hollywood studio made it. But it is something that researchers were just able to produce using this technique. And a great way to think about it is in contrast to what machines used to be good at until GANs came along. So machine learning and later on, deep learning is excellent at uncovering patterns in existing data

• 16:30Speaker 04

  and then using that insight to unsupervised machine learning tasks such as regression or classification. So for instance, there has been huge advances in machines that were taking an image as an input and then categorizing it to the correct label. So you have an image and then the machine tells you it's a dog. Or you have another image and it tells you it's a cat. What used to be extremely difficult until Ian Goodfellow came along with this innovation was doing the reverse of this process. Essentially, taking the label dog, feeding it into a neural network and having an image of a dog being produced at the end of it. I mean, this is overly simplifying it, especially when it comes to internal workings of it.

• 17:15Speaker 04

  But conceptually, it's essentially instead of having a computer classifying something, it's having a computer create something. So it's a, philosophically, it's like a level of imagination or creativity that the machine would have. Yeah, this has been something that was extremely difficult to do because it's extremely hard for a human to define what a realistic image of a dog is. And it's also extremely difficult to capture mathematically.

• 17:46Speaker 04

  So the other generative tasks that are other generative models that came before GANs, they were usually trained by recreating the same image it was fed in, that it was fed, that's essentially like taking an image, compressing it into a representation and then recreating the image itself. And then you can tweak the internal representation to produce a fake image that is somewhat similar to the original one. What GANs were able to do is that the generator itself is not learning explicitly by something that a human programmer would define or a researcher would define. There is no explicit loss function for the generator. What is happening is that we have another neural network,

• 18:30Speaker 04

  which is the simple classifier, which is the discriminator, that helps the generator generate something that looks realistic. So effectively, we have another model that helps us do the teaching. And what's really fascinating is that what Ian Goodfellow accomplished is taking something that machines used to be, are, do that, which is classification, right? Like taking an image and saying it's real or fake, and using that insight to help machines achieve something that used to be very difficult for them, which is generating realistic data.

• 19:10Speaker 03

  So maybe to just quickly add on to that, I think in terms of your original question, right? Like around the application side of things. My perhaps slightly sort of less, you know, something like, I guess this is an informal podcast, so some researchers I know would probably object to this, but...

• 19:28Daniel Whitenack?

  All is safe here.

• 19:30Speaker 03

  Yeah, exactly. So I think, I think it's a reasonable approximation to how to think about GANs, is to think as the, like, basically first learned general purpose generative framework, right? So GANs have been sort of applied, as we discussed, in images and video, but also people might not know that they have been successfully applied in like tabular and highly structured data, natural language processing, audio. I've seen papers in... Network theory and and graph applications, obviously lots of artistic applications. You know, even some defenses against adversarial examples, though that is still far from being a solved problem. So obviously, GANs have had like sort of managed to replicate so much interesting data.

• 20:12Speaker 03

  But in terms of like the business applications, that they tend to be non trivial, right? Like it it you need to really think where does this technique make sense? Uh it can be extremely powerful. And you know, unlike with classifications and and sort of decision theory and things like that. We have not had the ability to recreate, you know, realistic data for that long. So it is not always immediately obvious to uh how to, you know, apply it in a business process or something like that. And the reality is that that this is a technology that's like literally four or five years old. So I think it'll take a while till there will be like some sort of, you know, mass adoption.

• 20:55Speaker 03

  But I think there's lots of interesting things in the the fact that you can apply this uh technique in all these different domains. But you know, the applications of like how to use it, you know, sometimes will be. I I think one of the earliest practical applications I remember was in something in dentistry. Where people tried to actually propose, which is another area that I haven't even mentioned. Proposed like a three D mesh of the crown, I think that they were trying to fix. So basically to create art artificial crown that would uh fit into the patient's mouth and and on the with the rest of the teeth. Using GANs in the three D space. Um I'm not sure, when I was doing research on this, I'm not sure how far it actually went.

• 21:37Speaker 03

  Because there was a lot of like, I think twenty sixteen, twenty seventeen, there was a lot of talk about it. And people like love to mention that example, but I have not seen that much follow up. So I'm not hundred percent sure what happened there. But uh you you know, there's definitely lots of really interesting applications. You just need to think about like, how do I use this strength, right? Rather than like very obviously applying it to to to some sort of business process where you have like. Okay, you know, I need to make a decision here. I'll just train a deep neural network to do that instead.

• 22:11Chris Benson?

  So I've noticed over the course of the conversation that we have actually mentioned, we've referenced Ian Goodfellow half a dozen times. And I thought it might, it might be worthwhile to ask you guys if you could just kind of tell us who Ian Goodfellow is. Obviously, he's, he's a, a pretty big deal in this industry. And with you guys working on GANs, if you just kind of give us a little bit of a background on who he is and what he did, and why it's significant to the conversation.

• 22:37Speaker 04

  Yeah, so Ian, why he's so important is that, you know, he's the single person who invented this technique. Of course, there were other his colleagues that are on the paper. But when it comes to receiving the credit for the invention of generative adversarial networks, it's him. He was a PhD student at the University of Montreal, where in 2014, at the end of the academic year, he went out drinking with some of his friends. And they were discussing the, somewhat I alluded to at the, at the beginning of this dialogue, how difficult it is to have machines synthesize photorealistic imagery, which used to be like from a research standpoint,

• 23:18Speaker 04

  an intractable problem. And Ian came up with the idea of the two dual link neural networks, which he then, after returning home from the pub, coded up. And later that year, he and his colleagues have published a paper that truly started this field. And then of course, there was the, the research community took it up. And there were since then huge innovations made, both on the original GAN, GAN model, both when it came to the complexity of the neural, of the model itself, as well as its application, the use of labels during the training, both the generator discriminator, or one of those. So the, the field has advanced

• 24:00Speaker 04

  considerably in just a few short years that, that it has been around. But yeah, and again, he is so prominent because he is credited with his invention.

• 24:12Daniel Whitenack?

  Yeah, I think, and just to note, we're often asked about learning resources. He is one of the three primary authors of the Deep Learning textbook, which we have referenced on this show many times. And though he has worked for various organizations in the past, I believe he's currently with Apple at this point.

• 24:28Speaker 04

  Yeah, that's correct. Yeah, he was at Google, OpenAI, and now, most recently, he joined Apple, which, very excited to see what we will come up with, because I'm fairly certain that the technology that is powering the emoji that you can manipulate with your face on, like with an iMessage, the underlying technology for that would be GANs. So when we have seen a lot of the flashy applications of creative image processing and image editing, like the applications that make you older, like the Face App, or the applications that make you look younger, like the baby filter on Snapchat, tend to have GANs as the underlying technology.

• 25:11Speaker 04

  There are also other photo editing pieces of software that have very advanced features that are also using GANs as the underlying technology. So when it comes to the creative applications, when it comes to like the immediate, like the commercial use cases, then image editing is where GANs have shined. But I think that's only scratching the surface of what will be ultimately possible with GANs in particular, and also the research directions that this techniques has opened up.

• 25:41Daniel Whitenack?

  So to kind of summarize, and I'll make an attempt at this, and you can tell me if it's a good summary, but to kind of summarize, it seems like that GANs, as opposed to other kind of "normal" models that people might envision, is that there's actually two models, the generator and the discriminator, and they feed back to one another's. And one is trying to generate something, whatever that might be, based on some randomness. And then one is trying to differentiate between the generated version and some gold standard or real version. Would that be a good overall summary?

• 26:18Speaker 04

  Yeah, functionally, yes.

• 26:21Daniel Whitenack?

  Okay, and I was wondering, like, for those two models, I guess, I had a couple of questions, but one of them is, like, let's say we take the example of the faces or something like that. In terms of the mechanism feedback between the two, is it that, like, when you are generating things, you generate, like, a whole bunch of, you know, fake faces to mix in with the real faces and then try to, you know, discriminate with the classifier model or retrain or update it? Or is it a sort of one-at-a-time thing, like, you generate one face and then add that in? What's the sort of balance that happens there and the considerations you have to take into account?

• 27:02Speaker 03

  So, if I get your question correctly, I think it's about the training and how, you know, how to balance the two networks' learning regime. And I think, you know, you kind of hit the nail on the head here because that tends to be one of the most challenging aspects of GANs is the training part, because even though it might sometimes seem like magic, it's obviously, you know, driven by, you know, real algorithms. So to nail that dynamic can be very challenging. And, you know, in my day-to-day work or, you know, even over the course, it's just like playing around with new research papers and their code. That tends to be one of the biggest challenges. People have proposed, you know, literally, I don't think it's an exaggeration, there's

• 27:46Speaker 03

  literally hundreds of papers and thousands written on just the training dynamic alone. And it's obviously, like, quite a bit of challenge to get that exactly right. There's, like, techniques that keep on popping up over and over again, people in, and, you know, many of those would be, like, covered in the book, but to feel this ever-evolving, right? It's, there's new things coming out, you know, every major conference, there is, like, at least, you know, five to 10, like, new proposals on how to improve this training dynamic. And some of them take off and, you know, start to be incorporated by more papers, others sort of, uh, might have been good but, like, fade into obscurity through some, like, you know, the...

• 28:30Speaker 03

  The pseudo random process of academic discovery, that's kind of my take on it. But yeah, I think in general, like I've I've noticed that, you know, having a solid starting architecture, like close to something that you know has worked. And look at the data set that you're applying it to. Because, you know, a lot of the academic work tends to work on fairly standard data sets. If you're applying it on something else, tends to be very different. So you need to think about the data set as well as the network and the architecture. Which I think just kind of talks about one of the differences between academia and industry. Where the industry problems tend to revolve much more about the data set than and thinking about

• 29:14Speaker 03

  the sort of, as Karpstein calls it, like the data programming.

• 29:18Daniel Whitenack?

  So is that this part of the reason maybe why, um, so Vlad mentioned that, uh, when he was giving his sort of introduction, that there is still a bit of a struggle to kind of make the transition from GANs to their application in sort of day to day data science work in a widespread manner. Do you think most of that is because there is still a lot of fuzziness around the best way to approach training? Or what do you think's factoring in there? What are some of the? Is that the main challenge? Or are there other things kind of preventing that?

• 29:52Speaker 03

  I mean, I personally think that there's quite a few challenges. I think training is definitely one of them, but I think realistically, I mean, even if you look at the state of our field more broadly, like not that many companies are successfully deploying deep learning models, even supervised on a regular basis. So I I think that, you know, obviously the infrastructure and the support and the business thinking about the whole machine learning space is maturing. But I think, you know, generally, I think the reasons for why GANs have generally been applied mostly as like specialized startups or very specialized business units that have someone with a lot of GAN experience is for obviously the training difficulty, but also

• 30:36Speaker 03

  like having the right set of.

• 30:40Speaker 03

  Business sort of incentives or not even incentives, maybe more more like intuitions around how to apply GANs successfully. I think there's a lot of really, really good applications for them. But you generally need someone who can, you know, sort of tell you where where that extra effort is worth it because it will be somewhat challenging because of the training dynamics and other things to to deploy. So I think you need to have someone who can like sort of guide you through what makes sense in this situation, but also like someone who can like pick out the the right tool for the job. So, you know, even machine learning broadly is like still relatively novel though to us and may not feel that way.

• 31:25Speaker 03

  And I think businesses are still trying to catch up.

• 31:29Chris Benson?

  So I'm curious, as you know, as we've been talking about this and we've kind of gone into some into some depth about how generators and discriminators work. And I'm wondering, are there other models, you know, either other types of neural network models or other machine learning models outside the neural net space that you could use as a generator or discriminator?

• 31:51Speaker 04

  It's a great question. So the discriminator itself, if you actually isolate it from the GAN model, then that's just a classifier in most of the incarnations of the GAN architecture. So these are two separate neural networks that can effectively function independently. When it comes to generative tasks, then there have been other models that were used for that purpose. You have the image in particular, you have both restricted Boltz machines and auto encoders that were used for this purpose. But when it comes to image generation, and this applies to both static imagery like photos or to video footage, then GANs are indisputably the state of art for for those type of tasks.

• 32:37Daniel Whitenack?

  So we've kind of gone over the the basic generator discriminator, the interplay between the two and the specific models. I know that you talk about a few more advanced types of GANs in your in your book, though. Is there a whole, I guess this is a whole research area, and there's probably a lot of different types of GANs, but are there some more advanced types of GANs that are starting to filter into maybe a little bit wider spread usage?

• 33:06Speaker 04

  Yeah, Jacob, do you want to talk about CycleGAN? And I can then talk about some of my supervised paradigms.

• 33:12Speaker 03

  Sure. So I think, yeah, I think like Vlad said, I think CycleGAN is definitely one of the more prominent examples. I think when you talk specifically about the different cases and different applications, I think, for example, BigGAN is very popular with sort of as an artistic tool. So there is a tool you can check out made by one of my friends, Joel, who wrote GAN reader, I think it's GANreader.app or something like that. And you can sort of create new like combinations of images and people have entire like art collection, just of generative art that have been happening for quite a few years now. I was very surprised at how quickly like GANs became popular among the sort of artistic

• 33:56Speaker 03

  community. There's like lots of like digital artists using GANs as their primary tool of choice. So there's BigGAN, CycleGAN, and like I think there's been recently, well, a couple of startups popped up using StyleGAN as well as a way to generate like stock images of very high resolution faces and a bunch more, but I don't want to steal all of the material from Vlad, so.

• 34:25Chris Benson?

  I was wondering, but as y'all are talking about this and you're kind of going through these different versions, if you could, I recognize you don't want to go do a deep dive necessarily on all of them, but if you could give us a little bit of explanation with kind of each of the titles that you just went through, so that those of us who are not as familiar with them can kind of categorize them in our own thinking as we try to absorb the material.

• 34:48Daniel Whitenack?

  Yeah, I think maybe you mentioned CycleGAN and StyleGAN, I think, was it?

• 34:52Chris Benson?

  Yep, I heard both is.

• 34:54Daniel Whitenack?

  Is there like a quick, we'll really test your summarization skills, like a quick one-sentence description of each of those?

• 35:01Speaker 03

  So I think the way to think of CycleGAN on a very high level is, you have two domains and you're trying to basically always map one domain into the other. And it's very surprising how broadly that transfers. So for instance, satellite images to something that looks like Google Maps or night to day. Right? These could be different domains. And you can then have a generative framework that can translate, you know, something that is a day image into a night image and back again. So you can kind of have these two domains and you can have one generator for each translation, basically, if that makes sense.

• 35:36Chris Benson?

  That does. That's a good one. I like that kind of hands-on aspect because as I'm trying to follow what you're saying, that gives me a tangible example. Do you have something similar for the StyleGAN?

• 35:47Speaker 03

  So StyleGAN, I think most of the advancement there is in the algorithmic,

• 35:57Speaker 03

  from the algorithmic perspective. I think one of the things that StyleGAN had, one of the major innovations, just a big one, was the fact that you sort of keep, you don't source from the latent space just at the beginning, but you keep adding information throughout the generative process. So that means you can influence different levels of features and have much finer level of control. So because you keep re-add, like adding sort of sources of information and inspiration, quote unquote, throughout the generative process at different layers of resolution, you can sort of more finely tune, you know, like the big, like broad aspects of the face or very tiny details.

• 36:41Speaker 03

  And you have a more granular control than just the initial vector. You know, if you just want to change the hair a little bit, you can do that. Or if you just, if you want to change like the whole face or ethnicity or, you know, gender or these big, you can do that also. And you can kind of sort of have a better way of expressing what type of image you'd like to get.

• 37:13Speaker 05

  This episode is brought to you by Brave. Big news from the Brave team. Version 1.0 is official. That means our favorite open source, privacy-focused, blazing-fast browser is ready for primetime. Their brand-new iOS app landed just in time for the announcement, and the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to chainslog.com, hitting the triangle icon in the upper right-hand corner, and flipping us a tip.

• 38:00Daniel Whitenack?

  So Vlad, I think maybe you had some other input as well in terms of maybe advanced or specific different kinds of GANs that people are pursuing now in a sort of wider sense?

• 38:13Speaker 04

  Yeah, sure. Yeah, I can talk to the conditional GAN. So in a regular GAN, you have the data set of, let's say, real images of human faces that the generator over the course of the training iterations learned to mimic. But similar to what Chekov was mentioning about style GAN, there is no way in the classic GAN paradigm to control what type of image would get generated. So once the generator gets trained on a data set of real human faces, at any given time when you feed it a random vector, that's the latent vector that Chekov was mentioning earlier, it would spit out a face. But the researcher would have no control over whether the face is a man,

• 38:57Speaker 04

  or a female, or a child, or let alone more fine-tuned features than like a human with glasses, or somebody with long hair or short hair, and so on. What the conditional GAN allowed to do, which is one of the early innovations that was since then fine-tuned by the research community, was to introduce labels during the training process, which allowed the discriminator to not only recognize whether an image is real or fake, but also whether it matches the given label. So in the example of human faces, it receives an image.

• 39:36Speaker 04

  It receives, it's told whether it's a real or fake image, but it's also told what gender it is. So therefore, for the generator to be successful at fooling the discriminator, it needs to produce images that are not only realistic looking, but also ones that match the label. And the magic of it is that once you have the generator properly trained, you can then pass it the latent space and the label that you want to produce, such as, you know, I want an image of a child. And it would, given, you know, sufficient training data set and given it's properly trained, it would then produce a fake example matching the label of your choice.

• 40:21Daniel Whitenack?

  And on the discriminator side, like in that, in that case, would it just be a matter of like adding a feature to the input of that classifier? That would be like, you know, whatever it is, gender or ethnicity, like, like was mentioned. Is that just another feature that gets factored into the discriminator?

• 40:38Speaker 04

  Yeah, exactly. There's different implementations of this can be done on a technical level, but broadly speaking, you're absolutely right. It's essentially training classification that isn't only binary, as in real or fake, but it is one that is taking into account also the, the correct label. And what's really great for the conditional GAN program is that the additional information that the training process is conditioned on can be arbitrary. So it can be a description. It can be, or it can be a single label, or it can be a description. So there are also GAN models that can take in a set of tags or even like a word description and then produce an image that is matching the description.

• 41:22Speaker 04

  So you can, for instance, feed it a description, say like, bird sitting on a branch. And if properly trained and given sufficient data, then the generator would produce a fake image matching the description.

• 41:37Speaker 04

  You know, of course, our imagination can go wild, but I could see this having tremendous practical applications, especially in spaces like animation, where currently you need a lot of effort by human animators to create, let's say, characters in a game or characters in a Pixar movie. But with GANs, you can greatly optimize this creative workflow by having the trained model essentially aiding the human animator. Or if these techniques get advanced enough, you can imagine digital worlds, be it in VR or even like a regular PC game, which self-create.

• 42:20Speaker 04

  So you can essentially going to be an infinite world where characters get generated on the fly, new terrains get generated on the fly, and are going to be extremely believable without the need for input for a human animator or even programmer.

• 42:36Chris Benson?

  Gotcha. So I'm curious, there is so much research going on right now in GANs. It seems to have really exploded in terms of just the so many people and organizations are interested in this and trying to level up. What are some of the biggest open questions that are still out there that people are trying to address right now? Where do you see the top researchers really focusing?

• 43:01Speaker 03

  So just to like put my perspective, and I think definitely keen to hear what Vlad thinks about this as well. There's the training question that we've already alluded to. And I think the other big area is sort of these more complex data sets that I think are only kind of getting started. The whole audio synthesis. I think the first papers that I remember seeing doing using like audio were maybe at ICLR this year, ICLR, just like one of the big conferences. Maybe there was something before, but that was like the first time I saw like good attempts at doing that. And I think just last month or two months ago, DeepMind released GAN TTS. So.

• 43:46Speaker 03

  There's just an example of a vertical where I would expect more things to happen. And so I definitely think that there's like a lot of scope in these um sort of non-visual types of data as definitely at least in research. I'm not so sure about production like in the next year, but like research definitely. And the third area I'll mention is just having a GAN being incorporated in some bigger process. I think a lot of the time where I've seen some of the more successful applications of GANs that have actually gotten deployed were things that it was a sort of supporting process either on the training side or some sort of post-processing side for a larger like machine learning

• 44:30Speaker 03

  pipeline. So I think GANs have a lot to offer if you know how to like, you know, add it into your algorithm, whether that's as a domain adaptation algorithm or just some sort of, you know, better sort of tabular data like generator or anonymization tool, that sort of thing. So there's definitely a lot of like scope for them to be incorporated as like one of the pieces in a bigger puzzle.

• 44:58Daniel Whitenack?

  Awesome. And Vlad, did you have anything to add there? Or I don't want to cause any any friction between the two of you, but if you if you disagree or have any thoughts, what are your thoughts here?

• 45:09Speaker 04

  Yeah, no, I still think it's like the GANs are great that there are countless research directions and different people can get excited about the different opportunities. Therefore, me, it's primarily the ability to leverage the internal representations that the GAN model learns along the way to succeed at the generative task.

• 45:30Daniel Whitenack?

  So this would be a similar idea to what people talk about in in sort of like, I guess in the NLP case, we're talking about like word embeddings and things like that.

• 45:40Speaker 04

  Exactly.

• 45:41Daniel Whitenack?

  Is it a similar idea?

• 45:42Speaker 04

  Yeah, precisely. Word embeddings is a perfect example. So you may have heard of the simple arithmetic that we can perform on word embeddings to demonstrate that the machines or the neural networks develop a very complex internal understanding of the semantics of the human language. So for instance, you can take the embedding or the vector that describes the word king, you can subtract like pure arithmetic the word man. And then you can add the word woman. And the resulting vector is going to be very close to the vector for the word queen. Which again, we take king, which is male royalty.

• 46:21Speaker 04

  You subtract the word from men, you add woman, and then you end up with female royalty, which is queen. And what GANs have been demonstrated to do is that you can perform the same kind of arithmetic on images. And it's really fascinating. So when you take the example that was published in 2015. So very early on in the short history that GANs have been around. You have an image of a man with sunglasses. You subtract an image of a man, you add an image of a female. And the outgoing image is a female with sunglasses. So you can perform arithmetic, and this is completely unsupervised.

• 47:05Speaker 04

  This is just based on the internal representations of this extremely complex space that is images. That you can then perform again like something intuitive is arithmetic. And then the computer, without being taught what the correct answer is, would come up with an answer that a human would, based on our intuitive understanding of what "arithmetic on images" should produce.

• 47:29Chris Benson?

  Just as a quick interjection, with that arithmetic as you're describing it, I mean, doing that on imagery, on video, I mean, that is a deep fake at the end of the day. Just to kind of tie two terms together, or am I wrong?

• 47:41Speaker 04

  Uh, yes, it works on the same principle. I'm not sure how deep fake works. Deep fake might actually work more on the principle of a cycle GAN that Jacob was talking about previously. But deep fakes, vast majority of them, are based in one way or another on GANs. And it is exactly, it's essentially modifying an image in a believable way so that it looks like something else. And this is also at the core of the applications that I alluded to earlier, as in the Face App, which you take a selfie and you can immediately make yourself look older. So it's translating yourself into an older version of yourself, and things like that.

• 48:22Chris Benson?

  Yeah, I can do without the older version because that's already happening quick enough anyway.

• 48:26Speaker 04

  Yeah, exactly. Who would, right? I just like, just wait a little bit. But what's fascinating is that the older version is like the perfect example of something where machines just cannot have the data. Because you would have to literally wait 50 years for people to get older and have images of them, so that you can train it in an appropriate supervised way. As in, like, bad examples of like, this is what this person looks like young, and this is what this person looks like when they are 70 years old. That's effectively impossible to get unless you get clever, which these generative models have been extremely helpful with.

• 49:05Daniel Whitenack?

  Yeah, since Chris went there, I didn't go there. I don't want to say anything bad about GANs. But since Chris brought up the idea of deep fakes and all that, that people are concerned about. Thanks for explaining that, you know, how GANs are particularly well-suited for this sort of task, because they develop this deep understanding of a deep representation of these images and features and different things that are important. You know, people put a lot of focus on that deep fake stuff. And you've mentioned a bunch of other applications. But from your perspective, I guess, do you think that there are a lot of good examples, of positive examples of GAN usage out there?

• 49:48Daniel Whitenack?

  Or are you concerned at all that the deep fake stuff? And, you know, obviously, that's what gets retweeted a lot on Twitter and blogs and all of that. Are you concerned that that's kind of overwhelming the attention around GANs? When there's a lot of good uses of them?

• 50:03Speaker 03

  So, yeah, so I would like to kind of set a couple of things straight. So first of all, like, so this is a super important topic, right? Like, and I think there's a reason we like, so it's part of the motivation why we wanted to include at least like a short ethics section in the book. But because obviously, we understand at the same time that people don't, that it's not a book about ethics, right? But I also want to highlight one thing, which is that the original deep fake algorithm and quite a few of the more successful are not actually GAN-based. There are other techniques. I just want to like equate the two terms. So, you know, GANs are one of the ways to

• 50:46Speaker 03

  Do deep fakes and uh uh in fact like so there's a London based startup that uh. I know some of the researchers from the Pulse and Thesea, which kind of uses this, uses a mixture of. Several techniques, including GANs, uh to as a way to sort of reanimate faces. And sort of do a more realistic dubbing. So effectively, when you have a V that's in. English, uh and then you wanna port it over to, I don't know, Chinese market, Russian market, or, you know. Like any other country, if just if you dub, obviously you can sometimes see that that's not what the people are saying. So they have kind of been been applying this in like the movie space to basically.

• 51:30Speaker 03

  Make uh improve the experience for moviegoers. And I think, you know, that there are obviously. Many positive applications even of deep fakes, but as overall as a technology, whether that's a net positive or. A negative, I don't know. What I would say, though, is that I think, you know. Photoshop and other manipulation tools like After Effects have been around for at least a decade. Maybe and uh realistically, you know, and when you look at. Where people can do a lot of damage with misinformation, it starts with already with something much simpler, right? Like things like articles or text, right? The G P T two and from Open A I and that sort of thing, like.

• 52:16Speaker 03

  There's a lot of stuff in between and I think. It's a really important debate around like, do we wanna, how do we wanna approach it? Because of course, like you can't uninvent any, any, any piece of technology and uh you could.

• 52:35Speaker 03

  Even guardrails that you set around them, I think are potentially problematic. Because you set those guardrails in almost any way, there will be some edge cases that will fall outside. And then you'll use cases where or or cases where people will just. Get fooled by by something because normally they're protected. So I think the typical example being like the platforms trying to. Policing, I think it's it's a very long and complicated debate about, you know, what. The boundaries around that should be. So I'll leave it fairly high level, but that's my take on it.

• 53:15Daniel Whitenack?

  Yeah, I think that's a great take and I think it gives people a lot to think about. Of course, with a lot of things with AI, I think there's important ethics things to consider. But at the same time, I don't think it's worth writing off specific techniques as, you know, blacklisted or something because they could be used in a certain way. I remember, I think, Chris, maybe you remember, but I'm pretty sure at some point during our conversations, it was brought up that, you know, GANs are, there are quite a few really, really positive uses of them, like in a social good sense as well in terms of, I think that there was one I saw where they were generating, like augmenting tumor imagery

• 53:58Daniel Whitenack?

  data sets to help actually train tumor detection algorithms because that data is obviously, there's privacy concerns, but it's also fairly scarce and, and hard to supervise that sort of thing. So I think that there are, you know, and I'm sure you guys have better insight into this, but there are a lot of positive use cases, right?

• 54:19Chris Benson?

  You know, just before you guys jump in, I remember when we talked about deep fakes some episodes back, we had a lot of people in the audience come back at us and say, you know, hey, guys, I recognize you were being a little bit, you know, dark in that and we had acknowledged that, but a lot of people came back talking about good uses and I just wanted to kind of call that out and that, sorry Vlad, I think you were starting to talk.

• 54:40Speaker 04

  Yeah, no, I was saying absolutely, like medical space, I think it's a great example to bring up where can augmented data sets can be tremendously useful in unlocking diagnostic applications where previously there just wasn't a sufficient enough data set.

• 54:57Daniel Whitenack?

  Awesome. Well, we've had a lot of great conversation here. I know I've learned a lot and obviously we will link your book in our show notes so people can find it, GANs in Action, and that I'm sure is a great place to start in terms of getting to know about these subjects in a lot more detail. But I was wondering also if, you know, maybe to close this out, if you both have any insight into, you know, where's the best? Place for people to jump into this subject in terms of maybe like certain frameworks that are easier to work with GANs than others? Or maybe it's like certain example tutorials or

• 55:38Daniel Whitenack?

  problems or tasks that would be a good place to start and jump in and experiment and get hands on. What are your thoughts for our listeners that are interested in that?

• 55:48Speaker 04

  Yeah, so we might be a little biased, yeah, here, but yeah, we might be a little biased, but I'll let Jacob comment.

• 55:53Speaker 03

  Oh, yeah. Well, so I think I'll try to provide like one example for each, right? So obviously, like, I think keeping up to date with what's latest, I think Twitter and following the right researcher is the best way. In terms of like frameworks that I that are easier generally, I mean, I have not had extensive experience with TensorFlow 2.0, but I find PyTorch to be a lot easier to work with, especially for GANs, because there's a lot of complicated things that are going on and and it's easier to kind of dive into the internals. So I generally kind of tend to prefer PyTorch over over TensorFlow these days. And I think, you know, I think especially with the latest improvements that Facebook's making into PyTorch, I think

• 56:38Speaker 03

  there's a lot of things that are picking up, especially amongst the research community, which is kind of where I live right now, like in the bridge between like the research and industry. So towards it's good. And I would say that I think in terms of the the types of networks that are most tried and tested, I think like DCGAN, CycleGAN, yeah, those would be like the like StyleGAN, obviously, actually is that those would be like the three bases that I think are really important to understand. But of course, like it's a massive field, so I think having like a comprehensive resource, you know, like like our book is it could be really useful as a as a sort of map to get these get your head wrapped around it because

• 57:22Speaker 03

  there's much more than I can I can say in this case. But yeah, hopefully that's a good.

• 57:29Daniel Whitenack?

  Awesome. That's great. I think it gives a lot of good perspective because people want to start poking around and I encourage people to get the book and dig in to this subject and work on good positive examples that can, you know, help demonstrate this technology. We certainly thank both of you. Thank you for taking time out of your busy schedules to help us parse through this subject, which is a complicated one and there's a lot there. So I'm glad that we had you both on here to help us parse through it. I really appreciate it and hope we can meet sometime. Yeah, no, absolutely. Thank you for having us.

• 58:03Speaker 00

  Thank you so much. All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor. Go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear change law because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in

• 58:47Speaker 00

  your email address. Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.