GIPHY's celebrity detector

GIPHY’s head of R&D, Nick Hasty, joins us to discuss their recently released celebrity detector project. He gives us all of the details about that project, but he also tells us about GIPHY’s origins, AI in general at GIPHY, and more! Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $100 credit. Learn more at do.co/changelog https://do.co/changelog. Discover.bot https://bit.ly/2OWMyw2 – A digital space for bot developers and enthusiasts of all skill levels to learn from one another, share stories, and move the bot conversation forward. Want to learn more about building bots? Get started with their Guide to Bot Building Frameworks https://bit.ly/2Vxk7qO. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. GitPrime https://www.pluralsight.com/product/flow – Download GitPrime’s 20 Patterns book https://resources.gitprime.com/books/20-patterns/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog(q1), a field guide to help engineering managers recognize achievement, spot bottlenecks, and debug development processes with data. Featuring: Nick Hasty – Website http://jnhasty.com/, LinkedIn https://www.linkedin.com/in/nickhasty, X https://x.com/jnhastyChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Blog post about the celeb detector project https://engineering.giphy.com/giphys-ai-can-identify-lil-yachty-can-yours/GIPHY Enters the Open Source Community with Celebrity Detector (readwrite article) https://readwrite.com/2019/03/09/giphy-enters-the-open-source-community-with-celebrity-detector/MTCNN method used in the celebrity detector project https://arxiv.org/ftp/arxiv/papers/1604/1604.02878.pdfGitHub repo https://github.com/Giphy/celeb-detection-oss/Demo page https://celebrity-detection.giphy.com/3D projection of celeb face embeddings https://celebrity-detection-projector.giphy.com/Complete list of all celebs in the model https://github.com/Giphy/celeb-detection-oss/blob/master/examples/resources/face_recognition/labels.csvUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:47Daniel Whitenack?

  This is Daniel White neck, one of the co-hosts of Practical AI. I'm a data scientist with S I L International. I'm joined today by my co-host, Chris, who is a chief AI strategist with Lockheed Martin RMS APA Innovations. How you doing, Chris?

• 2:04Speaker 00

  Doing great. How's it going today, Daniel?

• 2:05Daniel Whitenack?

  It's going great. We were just talking before the show. I'm dealing with a few allergy issues. I think you are too, but it's that time of year, I guess.

• 2:12Chris Benson?

  It is. So a couple of coughs here and there is a normal part of a podcast as far as I'm concerned.

• 2:17Daniel Whitenack?

  Yep. Well, are you up to date on your pop culture, Chris?

• 2:21Chris Benson?

  I'm trying. I'm looking forward to learning a whole lot more on this episode.

• 2:26Daniel Whitenack?

  So my wife and I have been listening or watching Jeopardy now that it's on Netflix during our dinners recently. And I, I fail often at the pop culture categories. So I'm really glad that we have Nick Hasty from Giphy with us today, who is an expert in such areas, but also an expert in AI to talk about some of the work they're doing in AI at Giphy and where that crosses with pop culture and other things. So welcome, Nick.

• 2:56Speaker 03

  Hi. Thanks for having me. It's great to be here. Yes. Extensive pop culture knowledge is a prerequisite here at Giphy.

• 3:03Daniel Whitenack?

  Yeah, I assume so. So hopefully I can, I can do a little bit better at Jeopardy after, after the show. Yeah. So Nick is the Director of Research and Development at Giphy. Nick, do you want to just give us a little bit of background about yourself, how you got into tech, how you got into AI and how you ended up at Giphy?

• 3:23Speaker 03

  Yeah, for sure. I'd be happy to. I've got kind of a varied background, to be honest. So I'm currently Director of R and D here at Giphy. We're in New York, New York, and I live in Brooklyn. So I grew up in Georgia, actually, and.

• 3:36Chris Benson?

  Ah, that's, that's where I'm at. Okay. Good for you. Georgia rocks.

• 3:40Speaker 03

  Can, Georgia. It's like about an hour and a half north of Atlanta.

• 3:43Chris Benson?

  I'm in Kennesaw. There, I know.

• 3:45Speaker 03

  No kidding.

• 3:46Chris Benson?

  And I grew up in Cobb County. So for those who don't know that the Atlanta metro area, I'm sorry, but you know, we're practically neighbors growing up.

• 3:52Speaker 03

  Yeah, totally. Yeah.

• 3:53Daniel Whitenack?

  I feel a little left out here, but I'll let you guys continue.

• 3:56Chris Benson?

  So sorry.

• 3:56Speaker 03

  Yeah, that's really funny. Well, it's a small world. Atlanta's doing good. So then, yeah, I went to University of Georgia's undergrad and I was actually an English major and always dabbled in computing in terms of like custom Doom mods and like music technology stuff. I was big into music and I actually didn't really get into, you know, web stuff and more Peter Science until grad school. When I went to an interactive telecommunications program at ITP called ITP. At NYU, it's kind of a art-technology school in the Tisch School of the Arts. And they kind of take a bunch of random people who are good at a lot of different stuff and throw technology at them, anything from algorithms and visualizations, data viz stuff,

• 4:38Speaker 03

  to like actual light, electrical engineering, just kind of see where you end up. So I was like historically very into like creative tech stuff and even like art proper. So after ITP, I started working for the New Museum here in New York. They have a new media art branch called Ryzone. And it's been around since the mid 90s, started off as a listserv. And it's now a website and museum and an archive and stuff where they house digital art. It's like an internet art organization. I was a director of technology there for a number of years. And I got a lot of my web chops there. And being at Ryzone, we were very invested in the GIF early on.

• 5:19Speaker 03

  It's because it was, I mean, the GIF is like a purely internet media. I guess you could say it's like something that was really born. I mean, it predates the internet, but I mean, it really came to its own.

• 5:29Chris Benson?

  Yeah, once upon a time, there were only GIFs and JPGs on the internet, you know.

• 5:33Speaker 03

  Yeah, totally. Yeah, like old.

• 5:36Daniel Whitenack?

  Well, it's also, it's like a primary communication technique now, almost, native to the internet.

• 5:42Speaker 03

  It's a really wild kind of trajectory that the GIF has had. It's been really interesting to see. So yeah, when I was at Ryzone, we would do like art shows where like the artists would only work in the GIF format. This is like the early 2000s, not early 2000s, this is like, well, they did stuff in the art shows like that since the 90s and the 2000s. And I started in the late 2000s there. And I've worked hard on their digital archives. So there's a lot of archive digital assets, including animated GIFs and those kinds of things. And after that, I started working, I was kind of like teaching myself Node and I was building a GIF maker. Tool and through school I was introduced to, through a schoolmate, I was introduced to Alex Chung, Giphy's founder. He was at Beta Works and he had just been pulled in as like they were,

• 6:27Speaker 03

  they were doing like a seed program for a number of different startup people. Alex is kind of like a serial entrepreneur. He's been in the game for many, many, many years and just has tremendous technical knowledge and just like business expertise. And he was looking for an engineer and to build a team and we met and talked and just really hit it off. And so they started working at Giphy the next day and this was Giphy's first week. So I was technically Giphy's first employee for a while. There were two founders, a guy named Jace Cook was the other founder, then Alex, and then I was working building the website and Alex was building the API. And we were just kind of cranking for a while and the team continued to grow. So yeah, I mean, we definitely, Giphy kind of took

• 7:08Speaker 03

  off like a rocket. It was very interesting. I think it was a really cool way that it worked out. It was like the time and the place, like you were just saying, Daniel, the GIF, it was coming into its own, I think with bandwidth increases and you know, it wasn't take like 10 minutes for a GIF to load, you know, in Tumblr, dumping GIFs everywhere. There was a whole way, I think on Tumblr, people really started kind of communicating with GIFs and using GIFs as reactions and using GIFs as a communications medium. You got like pop culture and all these kind of different things you can load into a GIF. And so it's like if a celebrity is saying hello, you know, you got all these kind of different layers of meaning that you can unpack. And it became kind of a, you know, really interesting way for which people to interact with each other online. And we kind

• 7:49Speaker 03

  of realized that. And our goal was to just, you know, since GIFs are so old and since they kind of work everywhere, I mean, at this time it was 2013. So like, there was still like contention around which video codec to use on the web and You know, somebody supported, you know, Firefox supported this, but Safari supported that. But like, no matter what, the GIF was supported everywhere. It was definitely an old format and, you know, overweight and kind of slow, but it's kind of universal ability to just load and play in anything, including like iMessage way, way back in the day.

• 8:23Chris Benson?

  So I guess at this point, with you guys being going for quite a few years at this point, what does Giphy research do now, now that you've kind of grown into a full organization? What are the typical types of things that your researchers are engaged in?

• 8:34Speaker 03

  Yeah, so fast forward now, I've kind of moved out of engineering and I more, I guess I'd say more, I'm like a product director person. So I run the R and D team. We are within the search division. So our earlier-

• 8:49Chris Benson?

  How big is Giphy in general, just to get a sense?

• 8:52Speaker 03

  Yeah, it's a good question. There's about 100 people. There's roughly 75 in New York. In LA, we have a, have our own content creation studio. So there's like artist animators out there. There's about 25 of those. Engineering wise, we're probably 30 to 40 at this point. Yeah.

• 9:08Chris Benson?

  That's a good sized organization. You're definitely past the, the early startup phase.

• 9:12Speaker 03

  Oh yeah, totally. It's been a wild ride to see that happen. There's a lot of people here now and it's awesome to have a lot of super talented people here taking care of the things that, you know, just like there were just, you could never deal with in the past. And just now we're just so much more mature and so much more able to handle everything that comes at us. So, you know, like I was saying, my background's more like, I like making things. And so I have been able to move into the more of the R and D, like special projects, see kind of research things, find fun stuff, you know, create prototypes for things and then other teams maybe, you know, find value in it and pick it up. But I mean, to be honest, our team, we're a good sized company, but we're not so big that we can just sit in the back and like mix chemicals and like play with our thumbs. Like we gotta

• 9:51Speaker 03

  ship stuff. So maybe one day, maybe one day. But yeah, so, so my team alongside the celebrity detector stuff, I mean, we've done, we were the kind of the first team to really use deep learning tools, like Word to Vec, we use this open source, we use that deep learning model like all over the place for recommendations and a couple of other things. So like on the Giphy website, if you go to the GIF detail page, which is like a single GIF, all the related GIFs down below, like our team kind of overhauled that and used Word to Vec to take user data to approximate relatedness in terms of how visitors visited the Giphy website in a session. We did a similar thing with our tag space. So like, we do like related tags and recommended

• 10:34Speaker 03

  searches using deep learning, Word Defect specifically. We are doing other kind of things now where we're building trending tools using, you know, various types of models, not all deep learning, some just, you know, machine learning, or some of them just kind of old school statistical algorithms, doing like language prediction stuff. We're doing head-to-head comparisons against like search results so that we can kind of get a better sense of how our search is performing. We had a spell of time in which we were doing a lot of image annotation. Obviously, Giphy has a whole lot of GIFs and, you know, we relied on our content team to annotate those for a long time. You know, we have like a

• 11:17Speaker 03

  team of people who are ultra mega pop culture wizards who can, you know, paddle off anybody in a GIF and then all the slubs and everything. And, you know, humans can only go so far when you reach the tens of millions in terms of your catalog.

• 11:31Daniel Whitenack?

  You know, it strikes me as why you're the company that can have like a pop culture wizard department.

• 11:36Speaker 03

  Oh, yeah, totally. I mean, we've got, I mean, it's, we could call it that. Our content development team is definitely, you know, people like, people whose jobs are to watch TV shows and cut GIFs out of them, our live GIFs. That's not the only thing they do. They do a lot of stuff, but like, we'll watch the live event. Like, they'll, they'll watch the Oscars and they will cut GIFs of the Oscars in real time and catch all those awesome moments of people's sassy looks or, you know, celebratory speeches and those kinds of things. And we'll cut those in real time. And we work with partners, movie studios, these kinds of, you know, various other people in the entertainment industry to live GIF events and things like that. So, yeah, it's very, it's very funny.

• 12:13Daniel Whitenack?

  So how many, I mean, you mentioned a lot of different things that you've done and are doing. Like, how many people are on this R and D team working with you to come up with with these projects?

• 12:25Speaker 03

  Yeah. So right now, there's five of us. We've grown a little bit. So we work with a A team in the Ukraine called Rails Reactor. And so I have a guy, Dmitry Voytek, Ihor Khrush are working with us right now. I've had a couple of other employees from them, but they are really amazing machine learning engineers and kind of the specialists in terms of like the algorithms and building out the systems and stuff like that. So we work with them and then Giphy CTO Anthony has, Anthony Johnson works on our team a lot too. He has a pretty extensive background in machine learning and deep learning. And so we're, right now we're tag teaming the team. It's been really awesome. We're like really cranking out some stuff.

• 13:04Chris Benson?

  So is it almost like a having between the, the two organizations working together, kind of almost having a sub-team of deep learning folks inside your R and D organization then?

• 13:14Speaker 03

  Exactly. Yeah, totally. It's like, um, you know, like I said, I've kind of moved into more product and vision and strategy and communication. I mean, I still will open up a Jupiter notebook and kind of like work on the data a little bit and kind of figure out like, you know, things around training data and sources and making decisions. But um, in terms of like the actual implementation, I don't do that as much anymore. Just, just it's cool.

• 13:38Daniel Whitenack?

  So with it being like, uh, I mean, obviously the things that you're building are filtering pretty quickly into the product and you're moving quick and you have, have a smaller team. Do you feel like that, I mean, you are R and D, but R and D in some organizations, like things take forever to get from R and D to product. Do you feel like that has, that's influenced how you approach AI development in one way or another?

• 14:05Speaker 03

  Yeah, I think, uh, you know, R and D initially was like me trying to end up in a really, really removed kind of fun playland, um, kind of hoping for that and thinking if I put the name first, then the reality will follow. But the truth is

• 14:19Daniel Whitenack?

  You can hope.

• 14:20Speaker 03

  Yeah. And not that I don't enjoy shipping in, but um, you know, with the velocity of Giphy would be cool. I was kind of hoping to like slow down and kind of like play a little bit. But truth be told, I mean, the work we did was like, had a lot of value. It was very obvious that it needed to be integrated into the company itself. So we ship and it's a goal of ours. And I think, and I think a lot of companies say, oh, we're going to spin up an R and D team. And then, you know, the team disappears and nothing ever shows up. And they're like, whoa, what are we throwing money into? You know, that can be kind of a gotcha. So we work hard to make sure that, you know, we're doing cool stuff and we're reading papers. And like, we're like looking at the latest stuff, but we're also then taking what's out there

• 15:02Speaker 03

  and applying it directly to Giphy so that there is a real value that can be proven and integrated. You know, I think my team's probably taken over four or five of the Giphy API endpoints. We don't do like the actual search algorithm, but we have provided a lot of metadata for the search that gets fed into the search engine. But I mean, like the Slack integration, we took that over a while ago and we redid it over last summer. That's one of Giphy's probably most well-known integrations is, you know, forward slash Giphy and then whatever, and then you get kind of a random gif. Historically, it was very random and we really liked the juxtaposition of the weirdness, you know, just kind of like the funniness, like it, I don't know. I mean, I think it-

• 15:44Daniel Whitenack?

  Yeah, I remember those days very well.

• 15:47Speaker 03

  Yeah. Our philosophy is work is boring. You probably don't want to be there. So maybe if you can just throw a little weirdness into the mix, you know, it can make things a little funner. But um, yeah, but since we've, we have taken over that and did, you know, applied some learning techniques to that to make it a little smarter and make it a little more on point and measured engagement, you know, it's been like 20 to 30% increases overall in terms of how people have used the integration itself. So like Slack and that endpoint, which is our translate endpoint, our related GIFs endpoint, related tag searches endpoint. Yeah, so point being like the things that we've done, we've been able to prove value in that. And then those things have been able to be incorporated by the company as a, as a whole.

• 16:30Daniel Whitenack?

  So the AI stuff that you're doing, do you typically have like a target value add, like product wise, when you're going into that? Or is it, do some things just start as like, oh, I wonder if we could do that with GIF?

• 16:43Speaker 03

  That's a good question, a little bit of both. You know, like being with the company for so long and not having the- Media constraints of having to ship very specific things. I kind of wanted to explore the periphery of Giphy's products and offerings that have kind of been more lackluster or kind of unexplored or whatever. Like, you know, our tag space, which was something that like, we know it hadn't really been maintained and it was very flat and it was just kind of, no one knew what to do with it. And then, you know, here comes Word2Vec. Let's see what happens when we throw the tags in Word2Vec and play around like, oh, wow, this is really cool. We can see interesting relationships between our tags that we never knew existed. Hey, these are actually better than our current implementation, you know, how we do related tags.

• 17:27Speaker 03

  Let's keep iterating on this product. And then it becomes like an actual, replaces the old method in which we were to do that. So it's kind of, that's been the approach. I mean, for the celeb detector itself, though, that was something that we've always had a need for tools that can annotate images. Like I was saying earlier, due to like, you know, just general human constraints around our content team, as obvious, you know, entertainment, celebrities, that's our bread and butter. So if we have a tool that can annotate these gifs as they come in through uploads and crawls and that kind of stuff and help us surface those things, that's a value add. But that being said, it was always something we wanted to open source. And it was always something we wanted to do to kind of demonstrate where Giphy is

• 18:10Speaker 03

  and what we're doing and kind of the different intersections we exist at. Because, I mean, people, I think, given Giphy's kind of distributed nature, like, oh, I know you're from Slack, or, oh, I see you in Facebook Messenger or Instagram Messenger, or, oh, I use stickers in Instagram, or whatever. People don't always know like what it is. It's kind of this amorphous thing. So this was kind of a flag in a way to say, hey, you know, we're doing these things, we're using these technologies, these are the things we're thinking about. And this is the data that we have to deal with. So like, the list of the celebrities that we trained in, those are taken from our top 50,000 celebs that people search for. So these are the celebrities that people actually care about. Like, there are celeb detection systems out there.

• 18:53Speaker 03

  I mean, I think Microsoft has one and that you can pay for. There's a couple of third-party hosts. But those services, while they may have a large number of classes in which they can provide, they didn't always hit the ones that we needed because... We tended to just be more on the cutting edge in terms of like what's going on in popular culture and celebrity culture. So, you know, the thesis was we can really demonstrate that we can do the tech, we can build something interesting and fun. We have a very unique set of data that is not necessarily handled by other people. So we can kind of, you know, put all our different ingredients into this like stew and then give it to people as a way to, if you're new to deep learning or if you're new to these kinds of things, we

• 19:37Speaker 03

  can open this up for people and gifts are fun and easy. I mean, that's a, it's a nice entry point. Like if people are slightly intimidated by machine learning or deep learning or any of these technologies, you know, gifts and celebrities is kind of like a nice spoonful of sugar to help people kind of dive in. I mean, I've always personally found that when I'm working with fun content or working on a fun project, I tend to learn the technology better that way as, as opposed to just reading a book or, you know, hammering stuff out in the class. I'm like, I want to build something and want to incorporate this fun thing and this fun thing. And, you know, if I end up needing like a, some deep learning algorithm or whatever to make it happen, that's cool. Yeah, that was kind of the thought behind that.

• 20:23Speaker 00

  This episode is brought to you by discover dot bot. Learn everything there is to know about bots at discover dot bot slash practically AI. Discover that bot was built by Amazon Registry Services as an online community for bot creators and makers of all skill levels to learn from one another, to share stories, and they regularly publish guides and resources to answer questions like how to set up payments to your bot, how to stop shopping cart abandonment, what KPIs are worth measuring, how to write an engaging chat bot dialogue. You can even register dot bot domains there. Learn more and explore this huge library of bot resources at discover dot bot slash practically AI. Again, discover dot bot slash practical AI.

• 21:13Chris Benson?

  So Nick, to take you back, we kind of leapt right into the full company, but I actually wanted to ask you a question about, you described when you had come into the company and you were having that conversation and the next day you were doing it. But even possibly before that, as you know it, how did the project itself come about? What was its genesis? And also, why were celebrities the thing that you moved to? Or was there anything before that?

• 21:38Speaker 03

  The project, you mean the celebrity picture or Giphy itself?

• 21:41Chris Benson?

  Maybe describe both of them, maybe kind of discriminate between the two and where they each were.

• 21:46Speaker 03

  Yeah, totally. So for Giphy itself, so Alex Chung, who is the founder, he built, he's just a perennial creator, like dude who just makes stuff all the time and has worked in the internet industry for a long time. And he kind of, he caught on the GIF thing and he built a GIF search engine that was just a website that he kind of built on his own and he launched it and it would immediately, he got some traction, I think it was like a press article and crashed and burned and because of its, just the influx of traffic. And so, BetaWorks, who, which is a New York City incubator slash venture capital firm, they knew Alex and they were introduced to him and Alex was like, I got this thing building.

• 22:27Speaker 03

  They're a cool company, they are very aware of culture and art and technology and were like, kind of got what Alex is up to. And they're like, let's-

• 22:37Chris Benson?

  Our episode with Hugging Face, Clem mentioned them as well.

• 22:41Speaker 03

  Okay, right on, yeah, totally. So they understood what Alex was trying to do and so they were like, they had a hackers and residents program and they brought Alex on and his, to build Giphy. And then, you know, Alex, within the first week of Alex working there, I was introduced. I was doing contract work and I had been working on a GIF stuff, I had been working on GIF stuff already at my previous job at Verizon and I was building a GIF creator because at the time in 2013, there was no good way to really make GIFs on the web. And I met Alex through mutual friend. And we just like really hit it off and I had already kind of been, you know, it was very evident that we would probably do good together. And so I just started working the next day. So that was 2013. Flash forward six years, I'm doing the R and D stuff.

• 23:26Speaker 03

  And, you know, the impetus was like, you know, Giphy is, we do a lot of different things. People know us in different ways, but let's really flex and demonstrate how the cool tech we're doing. And I think people may have an idea of like the volume and the gifts and we serve and like, kind of the scale of which we do. I mean, we serve a couple billion gifts every day. But I don't know if anyone, people know kind of like about the more interesting kind of projects that happen in-house and how working at Giphy is working at like, working with interesting technology at the nexus of culture and entertainment. So the celeb detector, you know, kind of embodied this. We could do this project, say, hey, you know, Giphy, we do deep learning. We're working hard, we've had this experiment kind of happening in the background.

• 24:11Speaker 03

  We use this tool in-house to annotate our gifts. But, you know, the real goal was to like, put it out there for people. Hopefully get some press. Hopefully people are interested in it and also be excited about the community itself. I mean, like any startup, you know, like we use tons and tons of open source projects. And Giphy would not exist categorically if it wasn't for, you know, awesome open source projects. So our success, we want to give some of it back and be able to demonstrate some of the things we've picked up and learned along the way and give something that, you know, people could play with. So yeah, that was where it came.

• 24:44Daniel Whitenack?

  Cool. Yeah. So I, I mean, that, that describes a lot of the motivation. I'm curious, you've talked about how the celeb project came about. I'm wondering, like, you talked about previously using like Word2Vec and other things for related gifts and all of that.

• 24:59Speaker 03

  Yep.

• 24:59Daniel Whitenack?

  In particular, on the celeb detector project, as you moved into that space, what were the kind of different things that you needed to learn tech-wise to be able to accomplish that, that you weren't doing before in your AI work?

• 25:13Speaker 03

  Yeah, it's a good, it's a good question. I mean, we hadn't done a lot of ConvNet stuff. A lot of the image recognition, object detection stuff, we had, we had used a lot of third parties for, you know, like API, like we're not going to build, if you need a generalized model for object detection and images, like. Not necessarily going to train your own for that. So we didn't necessarily need that in-house. But for the celebrity detection one, yeah, it was, um, this was like our first real extensive deep learning project. So there was a lot of experimentation in the background. I mean, first, you know, it was like just getting the data together against our searches. Like,

• 25:55Speaker 03

  we had a lot of images around some of these celebrities, some of them we didn't. So we had to like scour the web a little bit to build the image data set. And then making sure we had, you know, and this is probably like any real machine learning or deep learning project, but um, you know, making sure where the the labels were good. Like I had mentioned earlier, you know, Giphy's tag system was, has been going for a while and was kind of big and difficult to carry or difficult to manage in some ways. And so we used a figure eight, or it used to be crop flower. We used them to help us make sure that the labels that we had for the celebrities was really good. And then I think, um, since it is face detection, we did use previous existing models. So just finding the right one that we wanted to kind

• 26:38Speaker 03

  of do the, um, to build ours of, who ended up being the ResNet 50. I think that was, all that, you know, was pretty standardized. I think a lot of it kind of got a little more hairy when it became things like clustering and finding out how to, um, like group our faces together. So when our team used the center loss approach for face recognition, that was like a real, we, uh, in the blog post, I linked to the paper. When we used that approach, that kind of really, the results really dramatically increased. Um, that was like a big gateway.

• 27:10Daniel Whitenack?

  So, so when you're saying, I, I know in the blog post, which we'll link in the, in the show notes, by the way, so, so make sure and check that out. But, uh, you, you mentioned kind of the like face vectors, face detection, and then this clustering bit, which you just mentioned around like, uh, using center loss. So like, could you describe a little bit, for those that aren't familiar, like, what do you mean when you say like, these faces are kind of like encoded or, uh, vectorized? And then we're doing like clustering with center loss? What, what does that process really mean? What, what are you doing when you're doing that?

• 27:43Speaker 03

  Yeah. So, and again, I'm even more of the product person at this point. So I'm going to try to like, give my high level. I wasn't into weeds as much on that.

• 27:52Daniel Whitenack?

  Yeah, no worries. No worries.

• 27:54Speaker 03

  But yeah, when you have a face come in to the network, you make an embedding of it so that it becomes like a series of numbers or whatever.

• 28:00Daniel Whitenack?

  Yeah, so it's like face to numbers.

• 28:01Speaker 03

  Yeah, exactly. And so then for all the faces, they have a numerical representation. And what you want to do is be able to find similar faces and be able to match them. So if a new face comes in, when you translate that face into a group of numbers, you want to be able to check it against the previously existing groups of numbers that kind of live within the model to come up with a prediction. So for us, you know, there are pre-existing models, like I was saying, like the ResNet that can take an image and make it into a vector. And so that wasn't so hard, but finding the right way to take the vectors and cluster them and group them in such a way that when a new face vector came in, we could make an accurate prediction. That was definitely the biggest challenge for us.

• 28:46Daniel Whitenack?

  So you mentioned that you had selected ResNet 50. What kind of tools were using, was the team using to be able to do it? Was this TensorFlow or PyTorch or MXNet? Do you have any sense?

• 29:00Speaker 03

  Yeah, we, yeah, totally. So we went with PyTorch. We played with TensorFlow a little bit, but PyTorch was great. I mean, we've kind of fell in love with it. It's, I guess it's very Pythonic to say. TensorFlow in Python has its own kind of idioms and its own kind of ways in which it wants to be used. But PyTorch felt it was kind of easier to, it was just more malleable, we would say. And it was, it allowed us to iterate faster and allowed us to kind of experiment quicker. So yeah, we really fell in love with that and that's kind of become our go-to. And the code that we released is all in PyTorch.

• 29:32Daniel Whitenack?

  Cool. Yeah. So I'm kind of wondering as I'm thinking, like, you're building this face detector bit, which vectorizes the faces. You've got this clustering bit. And I'm just imagining like pop culture in general, like there's so much of, it's so dynamic, right? Like there's going to be new celebrities that are super popular in a super short period of time. So have you thought about it, like how that influences how you kind of manage and deploy the model in the sense of like, as new celebrities come onto the scene, how you're updating it over time? How like you're swapping out models in terms of like serializing them out of PyTorch and then swapping them out in a service or something? Like, how do you think about that sort of updating stuff over time?

• 30:15Speaker 03

  Yeah, for sure. I mean, we'll probably continue adding to the model based on the search queries that come in because it's like we want to be able to identify the celebrities that like our users are searching for. And that by its nature tends to be the more the newer, the cutting edge stuff. So we'll continue to, I'm not sure when the next time we will, because we've got other projects going on, but at some point in the future, we'll probably do another big sweep across our searches and see who we're missing.

• 30:42Chris Benson?

  Can I ask a follow up to that before you move on?

• 30:44Speaker 03

  Yeah.

• 30:44Chris Benson?

  I'm just curious. So when you're tying the business side, so a new celebrity comes on the scene and you guys are going through some sort of process about, hey, we gotta, we gotta add this in. And you have this deep learning model out there. How are you connecting the business side and the technical side together in a meaningful way to where you're able to get on top of it quickly and make it work and get it deployed out? What's the connection between the business side and the deep learning team?

• 31:11Speaker 03

  I would say in terms of like, you mean in terms of like making this value proposition to the business?

• 31:17Chris Benson?

  Yeah, like process, like how does it go from somebody who has nothing to do with deep learning is maybe watching the celebrity scene saying, oh, there's somebody else that we need to start paying attention to. How does that get into a task?

• 31:29Speaker 03

  Right.

• 31:30Chris Benson?

  Level action from the deep learning team? How does that move across there?

• 31:34Speaker 03

  Totally. I mean, I usually I'm in pretty good contact with like our content development team and they're kind of in charge of all the latest pop culture stuff. And I worked closely with them through this to get the, for them to go through the list of celebs to approve the different ones and make sure there's nothing bad or weird in there. And they can certainly, as things bubble up, they can hit us up and be like, hey, this person's really growing. Can we add them at some point? And we'll probably wait, obviously, because training is, it takes a little time, obviously, and we have to make sure we got the right data set. So as we hit a certain kind of set of new celebs, we'll probably just do one big training and then re-release the model after that. Hasn't been a huge ask so far. I mean,

• 32:18Speaker 03

  in terms of business value, I think this isn't something where we're like, hey, Giphy's got an API that we offer to everyone that's part of the Giphy API. So you can get celebrity predictions. It's not like, this isn't. The proposition was never deep learning as a service. It was always like, let's do this thing in-house, set it up in-house, and then also give it out to other people to kind of show how we do things in-house. That's why we included like some ops oriented code like Docker and those kinds of things to show how we're our philosophy now for deploying those kinds of things.

• 32:51Daniel Whitenack?

  Yeah, that's much appreciated, by the way.

• 32:53Speaker 03

  Yeah, we've actually been working with, for some of the latest stuff that we're doing, looking into Cube Flow and specifically Seldon. I don't know if you guys are familiar with that because we use Kubernetes here at Giphy for all our architecture and deployments and stuff. And Cube Flow seems pretty magical and awesome, especially Seldon, because it'll give you GRPC and REST APIs kind of out of the box and handles all the model management, all those kinds of things that historically have been, you know, they feel a little ad hoc. I think everyone kind of just does what they can, like, oh, put it on S3 and then download it or put it directly into, you know, the image or something. That's, that's kind of where we're moving towards. It seems like a sustainable and kind of a very efficient way to handle machine learning, deep learning deployments.

• 33:37Daniel Whitenack?

  Yeah, that community is moving forward really quickly. I know Chris and I are both big fans of things that are going on there. And in our last episode, so the one that will air prior to this one, we talked about some AI infrastructure and Seldon was mentioned there and Cube Flow too, and just a lot of the great things that are going on in the Kubernetes community. So I definitely encourage people to check that stuff out and check out that episode if you want more details there as well. Specifically, so you kind of moved to talking a little bit about how you really wanted people to try this out. You realized that there wasn't a great open source tool for this and you wanted to release it

• 34:17Daniel Whitenack?

  and kind of put your foot down somewhere in terms of your open source contributions. I'm wondering, like, being the, probably the greatest experts on using AI for GIFs, I was wondering, you know, for those that are interested in this subject, then maybe they'll look at the GitHub repo, which we'll link in the show notes. What are kind of the unique challenges of using GIF data in AI models as opposed to like videos or images, like single images? Does it carry over like the same techniques? Like you mentioned convolutional neural networks. Some of that I imagine carries through frame by frame or something like that.

• 34:54Speaker 03

  Exactly.

• 34:55Daniel Whitenack?

  How different is working with GIFs and AI models than working with videos and images?

• 35:00Speaker 03

  Yeah. It's like you said, you know, we do it frame by frame for the most part. You know, we use a ConvNet and just kind of iterate through the frames. The challenges though are just like, and I mean, in video is kind of the same way. And truth be told, we do transcode like all our GIFs into different video formats. And so we don't always work exclusively with GIFs. Sometimes it is easier just to, you know, if you're doing some stuff to just work with a video file or whatever. But you know, the challenges are as a, well, number one, the format's heavy. You know, it's kind of a hefty format. So the compression algorithm is, is it Limpel Ziv welcher or whatever? It's, uh, it's older, right? Um, and it's like kind of dictionary based from what I remember.

• 35:43Speaker 03

  So it compresses, but it's still a big format. You know, it still suffers from like, well, it hasn't been able to take advantage of like all the things that happen in video. So it's really just like a bunch of images that are squashed together into this file. And the player just kind of goes through each of the frames. And there's metadata at the beginning and end of the file to kind of say, here's the speed and here's the colors and those kinds of things. GIFs are also limited in the color palette. So if you have a high quality video or a high quality image and then you transfer it over to GIF, um, you know, each frame is only limited to like 256 colors, I believe. So you're going to lose some coloration there, which could very well affect how your machine learning system,

• 36:24Speaker 03

  you know, how your deep learning system like interprets the image properties of it in terms of just the, uh, determines the values. So ideally, you know, you would, if you can, get to work with this high quality, as high quality of images as you can in order to so your model can be the most accurate. Yep. So this is kind of the main thing. And that's just slowness too, like, you know, if you're moving with big files and it takes longer to train and it takes longer. If you're moving stuff over the wire, that takes slower. So those are kind of the biggest challenges, I would say, that we had to deal with. And maybe this happens with video too, but um, if you have, so you know, if a GIF has 90 frames or 120 frames, and it may become the situation where like evaluating every frame is not your best option just because of the overhead involved or the, or the time involved. So you gotta like do some sampling across frames and then figuring out which frames to sample.

• 37:10Speaker 03

  And then, you know, how you want to aggregate. And how you want to, if the end result is a single prediction of a celebrity, getting to that single prediction by looking at a single GIF, which could count 100 frames, maybe multiple faces, those kinds of things. Since there are so many frames or and more than one frame, one more than one, the frames are different, so more than one face could appear in a frame or not. So you do have to play a lot with, like I was saying earlier, with the clustering and how you want to group things in order to end up with that single prediction. So there was a lot of time spent in that component.

• 37:43Chris Benson?

  So I wanted to ask a quick question. I noticed in the, in the blog post, you talked about testing the model for different types of bias. And that kind of piqued my curiosity. I was wondering if you could give us a little bit of detail on that.

• 37:54Speaker 03

  Yeah, for sure. We want to do another bias. I mean, we want to do another blog post around that specifically. But, um, I mean, it's been a, it's been in the news, obviously, that, um, you know, there are biases built in to, or like certain models that have been released, you know, tend to have different biases around race or gender, these kinds of things. And, um, we were kind of curious to see about that. We took all the classes and I think our methodology was to, we wanted to see the representation of cultures, you know, just in terms of our data set, to make sure that it wasn't something that was leaning in a one specific way. Like, we wanted to make sure that we had a big, accurate, like a wide swath of people who were kind of being identified.

• 38:36Chris Benson?

  Sure. It's a big challenge for a lot of folks.

• 38:38Speaker 03

  For sure. And then making sure that, you know, those identifications are proper. So, yeah, we, uh, we looked at the different names and we checked out their, I think we used Wikipedia, you know, to get their ethnicities. And then we kind of did a breakdown and kind of see which ways we'd skewed. And there wasn't any kind of like major red flags, you know, we had a, I mean, Giphy itself, you know, we're a younger company and most people who work here, you know, are millennials. So like, we're very sensitive to these kinds of things. And our user base, you know, tends to be kind of proper representation of, you know, genders and diversity, those kinds of things. So, um, yeah, we found the classes that we had was like, you know, a good representation and there wasn't any kind of like overweight in terms of any single ethnicity or like gender or whatever.

• 39:21Speaker 03

  So we put up the, um, this is probably one of my favorite things about the whole project is that we, there's a link in the blog post to the vector space of all the faces that we have.

• 39:30Daniel Whitenack?

  Yeah, that was super cool. I checked that out.

• 39:32Speaker 03

  Yeah. And you can really see, um, it's interesting to see how a computer or when you, when you take an image and you crunch it down into numbers and then you group those numbers together and then you re-equate those numbers with faces. It's cool to see how the computer actually groups faces together based on their properties, you know, skin tone and hair color and all these kinds of things. So it was like, it was really funny to see celebrities to look alike or who had shared similar characteristics in their face that you had never, never seen. In fact, I mean, to be honest, like our, maybe our very favorite thing about this whole thing was, um, within the company itself, everyone would upload their own face and then they would get their celebrity prediction. And there was like a huge email thread that the company shared out where everyone

• 40:13Speaker 03

  was like, oh, look, you know, my match is Brad Pitt or, oh, you know, I'm Jennifer Lawrence or Idris Elba or whatever, you know, everyone's got their ego stroked a little bit.

• 40:22Daniel Whitenack?

  Yeah, that's awesome. I for one am, uh, I, I was just thinking while you're doing, or while you're talking about open sourcing things and having that online, it's something I need to, uh, I need to try out and share the results with my wife. I'm sure she would be, be interested to see that. But, uh, yeah, I really appreciate you taking time to go through the project with us. Just great work on everything. Really appreciate your, your practical perspective on things and also appreciate you, uh, you know, diving into some of the finer points of things like bias and other things. We'll, we'll look forward to that, that follow-up blog post. But yeah, I think all of this, uh, will be linked in the show notes. So Nick mentioned the 3D face embedding explorer, the, the GitHub repo, the blog post, we'll link all of that in the show

• 41:06Daniel Whitenack?

  notes. So check it out. But thank you so much for, for joining us, Nick. This has been really fun and, uh, I've definitely learned a few things about GIF along the way. And I learned a few things about pop culture.

• 41:17Speaker 03

  Hey, my job here is done then.

• 41:19Daniel Whitenack?

  What, uh, right as we finish out, did you have a favorite GIF?

• 41:24Speaker 03

  I do. And so within the Reddit release, when I released, when we put it on Reddit, someone asked for the favorite gif and I put it there. It's Vince McMahon. Do you know Vince McMahon from World Wrestling? He is like this crazy showman with insane faces. And there's a meme situation where he has this progressively more audacious reaction to images. So there's this one where there's this bodybuilder dude and then Vince McMahon looks at him and keeps getting more and more excited until he falls out of a chair. And it's just, it just he kept, I don't know, it's just really, really funny. And that whole Vince McMahon meme format for me, I don't know. Maybe this is the Georgia in me talking about wrestling. But yeah, that's maybe my all time favorite.

• 42:02Daniel Whitenack?

  Awesome. Yeah, I think wrestling is uniquely suitable there to that gif format, you know, with the expressions and the exaggeration in it.

• 42:09Speaker 03

  It's incredible drama. Yeah, we've got some, we've got some great, some great wrestling, just totally ridiculous wrestling gifs that I could just watch over and over and just kind of ponder what the hell, what's all happening within this thing.

• 42:21Daniel Whitenack?

  Yeah. Awesome. Well, we'll make sure and add that link to the show notes as well. But thank you so much, Nick. It's been great. Thanks for joining us.

• 42:29Speaker 03

  Of course. My pleasure.

• 42:32Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before our users do here at changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers at linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email keeping you up to date with the news

• 43:17Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 43:42Speaker 00

  All right. Because you've stuck in here to the end of the show, got a surprise for you. Here's another preview of our upcoming show called Brain Science. This podcast is for the curious. We explore the inner workings of the human brain to understand behavior change, habit formation, mental health, and the complexities of the human condition. It's hosted by myself, Adam Stokoviac, and my good friend, Muriel Reese, a doctor in clinical psychology. It's about brain science applied, not just how the brain works, but how we apply what we know about the brain to better our lives. There we go.

• 44:14Speaker 04

  That applied brain science really stood out to me because I want, I don't want it to just be data. I want you to go, how can this fit? What can I take away? Now, how am I going to change? And that that sort of is where you come in more and even some of the questions like, so like, I want to ask you, what are some of the most challenging things working in the tech world when it comes to relationships?

• 44:34Speaker 00

  Probably the most important one is isolation. More and more, the world and companies are being, for good reasons, they're being okay with what they call distributed teams.

• 44:44Speaker 04

  Yeah.

• 44:45Speaker 00

  And that means that you and I, we work for the same company, but you work from your home office. I work from my home office. I might go into the office a couple of times a week if I live local, but even if I live in San Francisco, I'm still probably a remote worker, even though I can hop in an Uber or hop on, you know, the train or whatever and go into the office and be there in a half hour, but why waste the time?

• 45:06Speaker 04

  You know, and this is where I would revisit what I, what I want to talk about with resonance. And that whenever we're, we're learning, no matter what thing, it's really helpful when we get feedback that's both immediate and specific. And so when you're by yourself and you don't have any in an interaction with other people, how can you get any feedback? I mean, you're losing most of the nonverbal communication and you also don't have all of the voice inflections or facial expression. Have you ever tried to, you know, be sad, feel sad and smile at the same time? Try it.

• 45:45Speaker 00

  It's it's pretty hard.

• 45:47Speaker 04

  Right, because facial expression is exactly what's involved when it comes to empathy, which is relationships. I was reading a research article recently and it talked about, you know, how couples who are together a really long time end up sort of looking like each other. Ever heard that's? Yeah. And so what they've looked at is when we actually empathize with other people, facial expression is really key within that. And so when you empathize with the partner you're with over and over and over again, your face begins to make the same creases and facial expression as it relates to where somebody else is emotionally.

• 46:29Speaker 00

  Wow.

• 46:30Speaker 04

  Right?

• 46:31Speaker 00

  Say it again. So that's that's creepy.

• 46:34Speaker 04

  Well, they've, again, this is sort of the hotbed when it comes to neuroscience these days is mirror neurons. And these mirror neurons are what are involved with empathy. And so mirroring, meaning I I get another person's emotional world. And so one of the research studies looked at Botox. And what they found is that Botox, because it it actually assists in paralyzing facial muscles.

• 47:04Speaker 00

  Right.

• 47:05Speaker 04

  But then then you can't contort your face. So you don't get wrinkles, but actually levels of empathy go down.

• 47:13Speaker 04

  Right.

• 47:14Speaker 00

  Because your physical appearance can't reflect your inner appearance.

• 47:17Speaker 04

  You're, yeah, you got it. And so when you're working in these remote locations, it it might facilitate better work or more focus, and it allows people to be distributed and to capitalize on the talents across the country, right?

• 47:32Speaker 00

  Yeah. Wow. So that's, see, that's like a treasure trove, in my opinion, talking about in a scientific way, you know, not just like, hey, this is my opinion.

• 47:41Speaker 04

  Yeah.

• 47:42Speaker 00

  About all the cons of that. Because I think what we can do is still have remote work, but do it in more healthy ways. Because I'm I'm fully, I mean, I've been self-employed remote worker since 2006. Now, I'm a unique animal. And I know, I know that. My wife knows that.

• 47:58Speaker 04

  Right.

• 47:58Speaker 00

  And I'm fine with it. I'm a good human being, but I've got some flaws. And I'm willing to accept and share those to some degree. And I think the problem is, is we just, we just lack. More, maybe a more purposeful or intentional feedback loop.

• 48:13Speaker 04

  Yeah.

• 48:13Speaker 00

  Which I think is super important to being able to operate in this world in just good ways. I don't know, healthy ways is probably the best way to use in this show context is healthy ways. That's a preview of brain science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us at [email protected]. In the subject line, put in all caps, brain science with a couple bangs if you're really excited. You can also subscribe to our master feed to get all of our shows in one single feed. Head to change law.com/master or search in your podcast app for change all master, you'll find it. Subscribe, get all of our

• 48:58Speaker 00

  shows and even those that only hit the master feed. Again, change law.com/master.