AI-driven studies of the ancient world and good GANs

Chris and Daniel take the opportunity to catch up on some recent AI news. Among other things, they discuss the increasing impact of AI on studies of the ancient world and “good” uses of GANs. They also provide some more learning resources to help you level up your AI and machine learning game. Sponsors: DigitalOcean https://do.co/changelog – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. DataEngPodcast https://www.dataengineeringpodcast.com?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – A podcast about data engineering and modern data infrastructure. Brain Science https://changelog.com/brainscience – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: AI is impacting studies of the ancient world: Machine learning has been used to automatically translate long-lost languages: https://www.technologyreview.com/s/613899/machine-learning-has-been-used-to-automatically-translate-long-lost-languages/amp/?__twitter_impression=trueMachine learning is about to revolutionize the study of ancient games: https://www.technologyreview.com/s/613780/machine-learning-is-about-to-revolutionize-the-study-of-ancient-games/Others studies: Deep Learning for Classical Japanese Literature: https://twitter.com/hardmaru/status/1071479761556238336In Codice Ratio: Machine Transcription in the Vatican Secret Archive: https://youtu.be/8khPUtwaVawReading Medieval Manuscripts with Deep Learning Technology: https://picampus-school.com/medieval-manuscripts-and-deep-learning-technology/The AI technique that could imbue machines with the ability to reason https://www.technologyreview.com/s/613954/the-next-ai-revolution-will-come-from-machine-learnings-most-underrated-form/ A new way to use the AI behind deepfakes could improve cancer diagnosis: https://www.technologyreview.com/f/613924/ai-deepfakes-gans-medical-cancer-diagnosis AIs are playing more games: DeepMind AI is secretly lurking on the public StarCraft II 1v1 ladder https://arstechnica.com/gadgets/2019/07/deepmind-ai-takes-on-the-public-in-starcraft-ii-multiplayer/An unbeatable poker bot offers glimpses of video game AI’s future https://www.polygon.com/2019/7/14/20693903/unbeatable-poker-bot-ai-pluribus-facebook-carnegie-mellon-journal-scienceRelevant learning resources: NumPy implementations of a bunch of models (from scratch): https://github.com/ddbourgin/numpy-mlGreat new-ish NLP courses: Advanced NLP with spacy: https://course.spacy.io/New fast.ai course! A Code-First Introduction to NLP: https://www.fast.ai/2019/07/08/fastai-nlp/Google Cloud - Deep Learning Containers https://cloud.google.com/ai-platform/deep-learning-containersBooks “The Pragmatic Programmer” by David Thomas and Andrew Hunt https://www.amazon.com/dp/0135957052“Data Science from Scratch” by Joel Grus https://www.amazon.com/dp/1492041130Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co slash change log. Again, do.co slash change log.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:46Chris Benson?

  Welcome to another fully connected episode where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I am a chief AI strategist at Lockheed Martin, focusing on artificial intelligence, high performance computing and AI ethics. And with me is my co-host, Daniel Whitenack, who is a data scientist working with various NGOs and nonprofits. How's it going today, Daniel?

• 2:15Daniel Whitenack?

  It's going great. Great to talk again after the July 4th holiday and get back into all AI goodness.

• 2:24Chris Benson?

  Lots of good family stuff, right?

• 2:26Daniel Whitenack?

  Yeah, lots of, lots of good family stuff. Lots of, you know, family stuff is always good and great and awkward and all of those things at the same time. But it was great to see family and have a couple of days off. So what about you?

• 2:41Chris Benson?

  It's pretty much the same. As you know, I have all these dogs. So for us, fireworks, you know, twice a year on the July 4th holiday and at New Year's Eve. We try to have a little bit of fun early in the day and then hunker down with terrified dogs. So.

• 2:55Daniel Whitenack?

  Yeah, that's rough.

• 2:56Chris Benson?

  Yeah, it was our usual. It wasn't bad. And I'm excited. As we are recording, I'm about to head to Boston to participate in the developer workshop for AlphaPilot, which is a venture that Lockheed Martin is partnering up with Nvidia and MIT and the Drone Racing League, where we have teams from major universities all over that are coming together and they're going to be starting the process of creating various neural network models that are going to drive some really serious drones around a race course at high speed. So I'm going to get a chance to meet a whole bunch of smart people this week. Really looking forward to that process. It should be a lot of fun.

• 3:39Daniel Whitenack?

  That sounds really exciting. Is there, is there a place people can follow up with that? Or do you know if there'll be like videos or recordings of this stuff going on?

• 3:48Chris Benson?

  As far as this one, this is kind of the developer workshop and we'll see about what can come out. There's going to be a whole series of AlphaPilot drone races going on through the end of the year. And I'll put a link in the show notes where people can kind of connect in if they're interested. But you know, every time I think about this, I'm sure the, I'm sure the people doing it would, will roll their eyes. But you know, in that, in Star Wars Episode One, when, when the young Anakin's flying around through the desert, that's the thing that always teases my brain. I think it's a little bit more down to earth than that. But I'm looking, looking forward to having fun with my version of Star Wars this going week.

• 4:25Daniel Whitenack?

  Yeah, yeah, no, that sounds that sounds like a lot of fun. It sounds very exciting. And it's always cool to see AI. You know, a lot of times we're just focused on like I'm staring at my monitor on my computer and doing quote unquote AI, but it's cool to see like AI and machinery of some type and interact, whether that be robots or drones or little cars or whatever it is. So that's that's always fun and kind of connects it to the real world, I think.

• 4:53Chris Benson?

  It does. I mean, it's going to be interesting to see this as AI that most definitely has a tangible, real world impact, you know, that people can enjoy as a sport and as televised. So looking forward to to the start of that process this week. And, you know, you and I had not done a show where we're really focusing on current events in a while. And as we were talking, we decided let's do that this time around because there's been so much news out that we have kind of gone right by in the weeks. And so we're hoping listeners will sit down, enjoy us as we kind of go through this process of talking about some of the more interesting things we found in AI news lately.

• 5:32Daniel Whitenack?

  There's been a ton going on. I feel like we've done the topical shows, which have been really good. And I'm glad that we've kind of, you know, done that and gotten deep in some of the subjects. But there's just been so much accumulating in the news. I think it'd be great to to cover some of those things now for our listeners. There's been actually so much going on in the AI community that it's really hard to narrow down into a number of topics that we can discuss in this format only in 45 minutes or whatever we have. So we've done our best to highlight a few things. But if you find anything interesting that you're you're interested in or involved with in the AI community, please share that with us on our Slack channel.

• 6:15Daniel Whitenack?

  You can find that at changelog.com/community and or on our LinkedIn page. And we'd happy to we're happy to hear about what you're doing. And also maybe feature some of that on the show in the future. We've done that in the past and hope to do it more in the future.

• 6:31Chris Benson?

  All right. Well, you want to kick us off with the first article that you had brought to our attention?

• 6:35Daniel Whitenack?

  Yeah, so this is actually like a series of articles that I've seen recently. I tried to think about some trends that I've been seeing in the AI community. And one of the things kind of this spring, moving into like very recently, a trend that I've seen starting to develop is various AI techniques that are impacting studies of the ancient world. So I'm thinking of like things like studying ancient languages or archaeology or preservation of artifacts or making discoveries about the ancient world. There's been various things that are happening where the AI community is kind of intersecting

• 7:20Daniel Whitenack?

  and worlds are colliding with archaeology and linguistics and other things. So I saw a couple of these things recently. One that seems to have gotten a lot of attention was a study. So I read about this in the MIT Technology Review, but there's an article on the archive as well. And the title of the article is "Machine Learning Has Been Used to Automatically Translate Long-Lost Languages", which seems pretty cool. Does it seem cool to you, Chris?

• 7:50Chris Benson?

  It seems very cool to me. There's so much happening. We keep calling out how much is happening in NLP these days. And that is natural language processing, for those not familiar. It's just been huge over the last year.

• 8:03Daniel Whitenack?

  Yeah, and this in particular, it's like all of those recent discoveries kind of mixed with Indiana Jones a little bit. So it makes it even more awesome, I think. But there was this study. So apparently there are these ancient kind of lost languages. And I guess there's a number of them. But archaeologists have discovered kind of tablets and other things with these languages that date back to various times. In fact, there's this kind of language called Linear A, apparently, which dates from like 1800 to 1400 B.C.

• 8:41Daniel Whitenack?

  And then there's another one called Linear B, which dates along the same time period. And Linear A actually hasn't, I guess, hasn't been deciphered, but Linear B has been deciphered. It was deciphered only like, I forget what the article said, not that long ago. But the article basically shows how these researchers, which are from, let me see here, from MIT and Google. So Google Brain and MIT. What they did was they basically showed how they could utilize similar words or similarities between various languages

• 9:22Daniel Whitenack?

  and how they were historically derived to kind of automatically decipher this Linear B language, which is actually a lost language. And they showed that they could basically do it in a way that didn't rely on like parallel data. So it's like a real deciphering of this language, which is pretty cool.

• 9:44Chris Benson?

  That sounds really neat. I mean, so they're not basically looking for for common words, for lack of a better description. They're almost common approach to how language would be formed at the time?

• 9:56Daniel Whitenack?

  Yeah, so there's this idea of cognates, which I guess, and I'm not a linguist, so I apologize to all linguistic people out there where I'm butchering things. But these are kind of words that look similar in multiple languages. So you've probably familiar with this, definitely in English and other languages. But they utilize these kind of similar looking words, I guess, or these corresponding cognates. And then they do a sort of character matching thing. And they have a pretty cool diagram in their paper where kind of lost language characters come in and they use like LSTMs and attention and softmax and kind of, you know, known characters come out.

• 10:40Daniel Whitenack?

  And so they do this kind of translation between Linear B, which is this lost language and Greek, based on these, also utilizing these corresponding cognates and this cognate mapping sort of stuff. So they don't really use, you know, they don't need parallel data between Linear B and Greek. They're really treating it as a lost language, but they're utilizing these similar looking things.

• 11:08Chris Benson?

  That seems really cool. It's, I love the fact that there are so many applications that we're able to apply neural network technology of today, too. It's impacting just about everything you can imagine. I can now, going back to your Indiana Jones, I can imagine Indiana just sighing wistfully, putting down his whip and sitting down at his laptop to get the real work done.

• 11:29Daniel Whitenack?

  Yeah, and there's this other article, which I think very much fits within that, that framework. There's this other article, which I actually also read on MIT Tech Review, which I guess I get a lot of things from there, it seems like. But this has to do not with language, but with games. So in a lot of, so I guess there's medieval manuscripts that include these representations of games, whether that's like games played on horseback, or games like played, like table games, or dice games, or these other games. And up till now, there wasn't really, I guess there wasn't really a field of like, study around

• 12:14Daniel Whitenack?

  ancient games. And so there's this researcher in the Netherlands who's kind of proposing that there's this new field of study of ancient games, which I'm probably gonna mispronounce this, archaeology. And anyway, he has these pictures of these ancient games that are represented in manuscripts, and he's proposing and has shown some evidence that using machine learning and AI, we could use things like computer vision, but also evolutionary algorithms and other things to actually learn about how these games evolved, about their rules,

• 12:53Daniel Whitenack?

  and even like, take the whole of games that have been developed through time, and use their kind of basic elements to generate new games. So there's also this idea of using all of these basic elements of games through history, and then using AI to take those components and generate new games that we might, I guess that we might want to play. I guess that's the goal.

• 13:17Chris Benson?

  Yeah, I flipped up the The webpage that you're referring to and like they have a picture of kind of an image of of an old game with, you know, that looks like a plus sign sort of with various squares on it and and images and then they have what I assume is a modern version of the same game. So it seems, let me ask you this, I'm kind of curious. There seems to be a little bit of commonality potentially between these two. I mean, in one, in one case, we're, we're talking about looking at old games to try to derive, you know, how they came about and what their rules are. And yet we just talked about, you know, lost languages as well and going back. Would you consider that to be somewhat similar approach as NLP being used in both areas?

• 13:58Daniel Whitenack?

  Um, I think it, I, I think maybe the commonality is maybe even a little bit higher. Like, I know one thing that you're going to talk about later in our chat today is unsupervised learning. And I think the, the connection that's happening in, in a lot of different areas of AI is really the kind of transition to these like semi supervised methods and methods that kind of start from some data, but not really like labeled data or annotated data and like try to learn something. So in the lost language case, they're really starting with, you know, text or characters from these languages. And, you know, learning and deciphering these lost languages in the games case, they're starting with these images and maybe rules.

• 14:43Daniel Whitenack?

  And they're kind of doing some sort of evolutionary thing to learn about how those, those evolved or to generate new things. So I think we're definitely seeing a lot of that all over the, the AI community.

• 14:56Chris Benson?

  So I'm kind of curious, do you think that there's any particular work that is interesting to you at this point that this might be applied to in other areas? Like if, if you were, you know, you're looking, I know, and I know you're deeply involved in, in doing work with languages in, in your day job. What other areas do you think may this be, you know, very applicable to in terms of, you know, going and rediscovering things from the past? Or, or taking something we're already doing and thinking about what's next?

• 15:22Daniel Whitenack?

  Yeah, I mean, I was really excited when I saw the Lost Languages article. I think that I only have a certain perspective, so I'm sure there's, there's other things that are happening, but you know, in the, in the language space, I mean, there's a ton of languages where we don't have good machine translation technology, but there's also a ton of languages where, you know, these languages exist and they represent a certain view of the world and they're endangered and you know, UNESCO has said like, we need to preserve these languages and, and other organizations have said that, but a lot of these languages don't, they may not even have written languages or written scripts or anything like that. They're just

• 16:04Daniel Whitenack?

  spoken languages. And so the fact that we could maybe use some of these techniques to document languages, to learn about languages, or even sign languages using computer vision and other things, all of those are, are pretty exciting to me. That sounds pretty cool.

• 16:29Speaker 03

  The Data Engineering Podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques, and difficulties of data engineering so you can learn and keep up with the knowledge to make you and your business successful.

• 16:45Speaker 04

  Can you give a bit of an outline about the motivation for choosing Jupiter Notebooks in particular as the core interface for your data teams?

• 16:53Speaker 05

  Yeah. And actually, when I first joined Netflix, it was a sort of toss at me and I was definitely like, are we crazy? And the answer was like, we might be a little crazy.

• 17:01Speaker 03

  Go to dataengineeringpodcast.com to listen, subscribe, and share it with your friends and colleagues.

• 17:20Chris Benson?

  So we were talking a few minutes ago about kind of NLP techniques being applied in particular to deriving the rules from, you know, kind of ancient games and such. And I'm going to take that as a rough segue into some of the things that are currently happening in games. There were two things in particular that I noticed. One has to do with DeepMind. And there is an article that Ars Technica has called "DeepMind AI is secretly lurking on the public StarCraft 2 1v1 ladder." And, you know, over the last few years, talking about DeepMind and StarCraft and that link in terms of, you know,

• 18:02Chris Benson?

  them having StarCraft being played by DeepMind a few years ago, and now they're kind of doing a second version of it that's really interesting. In that article, they're talking about the fact that they're having AlphaStar playing it, play the game in an anonymous fashion. And they've done a bunch of limitations on it to make it more like the way the human would play it. Some of the things that they had noted in it was the fact that the last time around, DeepMind was able to utilize the entire game all at the same time. So it kind of had an unfair advantage over the humans in that it could do anything and see everything at the same time. And one of the things

• 18:45Chris Benson?

  that they're talking about now is they're talking about limiting it on what it can do from an action standpoint, you know, in terms of acting, they're limiting it so that it essentially cannot outact humans. And they're also focusing really on the camera view for the game instead of having that kind of holistic view of the game. So I thought that was really interesting in that there is they implement these characters, and you may or may not, with you have a group of people in a game, you may be playing with obviously humans, but you also may be playing with non-humans now and have no idea. What do you think of that, Daniel?

• 19:20Daniel Whitenack?

  I think it's really interesting. I wonder, in the game, so other than maybe like the screen name or something, I wonder if there's any way, like if they're making it known that this is a bot or anything, do you know?

• 19:36Chris Benson?

  What they said in the article was explicitly no. They were deliberately doing it anonymously. And I don't know. I mean, if you were to say, you know, if you were to communicate and say, you know, are you a human or are you a bot? If they're just gonna lie if it's a bot and says, of course I'm human, you know, what a stupid question, that kind of thing. I don't know how they're handling that.

• 19:55Daniel Whitenack?

  Yeah, I wonder what the perception, I don't know, I'd be interested to like read forums and different things around the perception of this because I definitely think there could be that, you know, I have brother in laws who are really into gaming. I'm not quite as much, but I know, you know, there's that, like when you're playing especially like ranked games and that sort of thing, like, you know, if you're playing against someone that has an unfair advantage, like you were saying, you know, I know that they've tried to limit that, but that could not come off so well. Like especially if this player is moving up in the rankings or something like that. I don't know that that would be accepted very, very well.

• 20:38Chris Benson?

  Yeah, it explicitly was noting, you know, the fact that they're really putting all those restrictions on and that they refer to the previous time around a couple of years back as an unfair unrestricted view of the game. And so, you know, as they do that, I find that fascinating, first of all, just because, you know, gaming is a great, obviously a great way to do that. But there are so many things I could see moving beyond the gaming world, you know, that are out there in terms of general use cases. I mean, we keep hearing that this, we are at this moment where we're gonna start interacting with different types of AI models as just a standard course. And, you know, the first thing that comes to mind is medicine, you know, and, you know, I know we've reported in the past about, you know, that you'll have

• 21:23Chris Benson?

  bots as your primary care physician in, within the next few years, like less than five years from now. And so if they're able to insert these alpha star driven, you know, bot players and they're indistinguishable from the regular player in terms of how you're interacting and what they're capable of doing. Once again, it really brings us into that, that world of, of that collaboration, you know, between the human and the AI that we, that we see going forward for, for some period to come at least.

• 21:54Daniel Whitenack?

  Yeah, the one interesting thing about a lot of these video games that they're getting into, and I know this was a topic of discussion, I forget with which, with which video game they were talking about this, but the fact that these video games, there's like multiple, like really interesting problems to solve from the AI perspective. So if we think about a game that has like character selection, that's like one thing that needs to be solved that like a human would normally do. There's also like the gameplay elements, but then there's even other things like if, if you really want this agent to behave like a human in a game, there's also like the chat and interaction element, right? Like especially

• 22:34Daniel Whitenack?

  in a team play game, if this, if this, if an agent is playing as a member of a team, there's communication between the team often as well. So this intersects with a lot of different areas and I, I would actually be pretty surprised, given the state of conversational AI and other things like if, you know, really an AI covering all of those elements at the same time is really in our near term, you know, future, but, but maybe it is, I might be too, too skeptical.

• 23:09Chris Benson?

  Yep. I, I think, I gotta say, I would love to, to have somebody from, from DeepMind who is directly working with, with this AlphaStar implementation onto the show and just talk through some of the details of this. But I just, it feels like the implications of how this could be used. This is a great test case, but I, I just, it really feels like we're turning a corner at this point. And, and I guess while we're mentioning turning a corner, I, there was another article that I ran across that has to do, the article is on polygon.com. It was entitled, "An unbeatable poker bot offers glimpses of video game AI's future." And really-

• 23:48Daniel Whitenack?

  I saw a lot about this on Twitter.

• 23:50Chris Benson?

  Oh, good.

• 23:51Daniel Whitenack?

  A lot of people were talking about this a lot.

• 23:53Chris Benson?

  What, what did you hear? Before, before I leap into my part, I'm kind of curious what the social media side said.

• 23:58Daniel Whitenack?

  Uh, so I don't remember exactly what they said. Um, and I didn't look a ton into this, so you'll have to brief me, but I just remember seeing, as I scrolled down on my Twitter feed, a bunch of GIFs with this like flowchart of how this thing worked and like poker imagery and all of this stuff. So it seemed like something that really caught people's attention.

• 24:17Chris Benson?

  Yeah, the thing that the article really raises is that they kind of took a different approach with this bot. And so instead of the model kind of having the long-term strategic game view, they limit it to thinking only two or three moves ahead. So once again, it's in a sense, it's almost coming down to some human limitations that we have. But the thing that really jumped out about this was they trained this bot to be able to bluff effectively. And what I didn't know, and I'm not, I'm not generally

• 24:49Daniel Whitenack?

  What does that mean in an electronic sense? Like it has no face, right?

• 24:54Chris Benson?

  Yeah, I'm actually, it's a good question. I'm not sure. I think what they were kind of talking about in it, though, was that I think it's just based on the action, you know, so that even if you're not sitting around a table seeing faces, you know, whatever someone out there is saying, whether they're holding or doing whatever. And I'm not a poker player, so I don't want to humiliate myself by trying to to jump into that. But basically, it's indicating that by only looking two or three moves ahead, it changes how bluffing would occur. And there was, there was something in the article saying that if it had taken a longer view of the game, the way it would bluff would be different, and it would be distinguishable. It's different from human action, because it's taken a different view than what humans would do, which are typically looking just a few, a few moves ahead. And

• 25:38Chris Benson?

  so this could accommodate, it could bluff, it could, it could accommodate what humans that are playing against it are doing, whether they're bluffing or not. And so I think that was what really made it stand out as a different type of model from some of these that we've seen before and reported on in some cases. But yeah, so we're teaching AI to bluff its way through, to lie.

• 26:02Daniel Whitenack?

  Interesting. Is it, is this like, active in some type of, you know, some type of online poker play site? Or is it just kind of an experiment at this point?

• 26:15Chris Benson?

  I didn't see it. I'm looking back through the article right now. I didn't see it being deployed. So I'm assuming it's in a very controlled environment. But, but who knows? You know, sometimes, sometimes I, I go out to Vegas for various conferences and such, and you never know, you may walk into a casino at some point, and be playing against the bot, or at least one of the, that'd be one of the players up there. That's the new, the new gimmick.

• 26:40Daniel Whitenack?

  Yeah, I'm also struck in the article I was kind of reading through as you were talking. And they say that there was also like an advantage they saw. So they they trained the model, I guess, on a 64-core server with less than 512 gigabytes of RAM, saying that the cloud servers to train up the program would cost only $150. So I think that that's a really interesting point and consistent with some other things. So I've seen, of course, fast.ai and Rachel Thomas and other people talking. I think it was her I saw recently with a slide about, you know, how to do innovative AI things.

• 27:25Daniel Whitenack?

  It doesn't, you know, requiring a lot of compute isn't always necessary. And we shouldn't have that really as a precursor in our mind. And I really like that, both because, you know, there's a lot of people in the world that just don't have access to train a model on a TPU pod and spend like $7,000 on compute costs to train one model, right? So like, I think that that's encouraging, that there's some cool stuff like this that's happening with very little cost as far as compute goes. But also, of course, like, you know, these huge, large-scale NLP models and other things happening these days, like the compute cost is pretty ridiculous.

• 28:09Daniel Whitenack?

  And also, like, the environmental impact of that is massive, which is kind of depressing that like- Yeah, I saw that as well. Along with Bitcoin, we're just like, you know, destroying, you know, racking up all of these energy costs because of these models that we're training, which I don't know that that's like, I forget if there's like a study that shows the total percentage of our of our energy usage or something like that. But it's definitely, you know, something that's contributing a lot. And in some cases, more than it really has to be, because we don't always have to throw, you know, a TPU pod or a V100 or something at a model.

• 28:51Daniel Whitenack?

  I think there's a lot of interesting things that we can do with limited architecture, which is something that I'm really interested in exploring, for sure.

• 28:59Chris Benson?

  Totally, and I'm going, you know, I know we're sharing articles, so I'm going off script slightly. But I've seen the same kind of thing about the amount of compute being just, you know, kind of crazy levels, you know, in terms of environmental damage relative to the benefit we're getting. I was talking to, and I don't have any of my notes in front of me, but I was talking to a professor, I'm in the Atlanta area at Georgia State, who had been working on basically taking complex models and finding the key important nodes in them. He has a technique where it goes through that and it can kind of compress a model way down. So you can, you can take a really sophisticated model and, and once it's trained,

• 29:43Chris Benson?

  at least do inference much more cheaply than you would have been able to before. And I know that there's that, that he's doing, and there's a lot of other similar initiatives to try to bring the cost of compute down. And, you know, at some point, it might even be worth doing a show on that, is pulling some of these different techniques together, or talking to some of the experts on this. But I do think that, that is a giant area of research that, that we're, that the world at large is focusing on, is, is being able to do effective inference at least, and maybe, maybe training without such a catastrophic cost.

• 30:17Daniel Whitenack?

  Yeah, you make a great point that there's that cost, but also things are moving towards us running AI on more like in the browser and on edge devices, on mobile devices. And at least for the foreseeable future, you know, those devices are, are limited hardware wise. So I think that, you know, privacy wise, it makes sense to run those things in a more optimized way on edge devices. You know, in a lot of cases, but also, you know, we should, we should be careful that we're not using all of this compute as a, as a crutch in some ways. I was just listening to our partner podcast, The Change Log.

• 30:59Daniel Whitenack?

  They had the Pragmatic Programmers on the, on the show, which there's this Pragmatic Programming book, which is, is quite famous. And they were having an interesting discussion about hardware limitations and some things like that, that, you know, just. In some ways are good and in some ways are are bad. And I've talked to, you know, a lot of more experienced programmers that are really glad that they were forced to consider some of those things around like memory constraints and other things like that. Whereas now we kind of take it for granted that if we want more, if we want more power, it's just there in the cloud. And maybe that's not always a great thing.

• 31:42Chris Benson?

  No, it's, we do tend to get very tunnel visioned in terms of, you know, we're, I know that we have this bias where we're so convinced that, you know, this is world changing stuff and amazing things can come of it and all. And we tend to forget that, boy, we're using a lot of electricity there in the background. So, yeah, we need to have a better, more holistic view of the world in that way.

• 32:11Speaker 00

  Well, hello there, listeners. How are you? This is Adam Stokoviac. If you haven't heard yet, we're launching a new show called Brain Science. It's a podcast for the curious. Are you curious? Because if so, we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and what it means to be human. It's brain science applied, not just how does the brain work, but how do we apply what we know about the brain that can transform our lives? Learn more about the show and subscribe at changelog.com/brain science. Until then, here's a preview of episode number two. We're talking about how we're all designed for relationships.

• 32:52Speaker 06

  I think about it like scaffolding, that as our kids grow, and, and it doesn't matter, like, I just always want people to have this sense of hope and optimism around, like, look, it's not over if you didn't get it in childhood or it didn't fully grow. Like, neuroplasticity is one of the most amazing and hopeful things because we can continue to build this and grow. All throughout our lives. And so having another person participate in the development of our own mind, it's sort of helping build neural networks that say, hey, I totally understand that you're upset as a three-nager because you did not get ice cream and you think your world is now ending. But to actually, you can still empathize, but that doesn't

• 33:35Speaker 06

  mean you necessarily give them that desire, right? Because I don't want them to be conditioned, i.e. I don't want them to have the perpetual feedback that when they're upset, that they just get to have the ice cream that they want.

• 33:49Speaker 07

  Right. Let's also say we're using children as an example here because for the audience to empathize with us, that's, that's our breeding ground for research, basically. And I can give an example where my son, you know, he, just some, I can't recall the exact scenario, but there was a moment where I was like, to my wife, I said, hey, it's not that he's misbehaving, because we were both sort of like in this crazy mood with him. And he wasn't behaving. And I was like, you know what, it's not that he's misbehaving, it's just that he, he can't right now. He's just too far gone. He's too tired, he's too exhausted, he's overstimulated, and his brain is just not developed enough to really get that we're asking him to behave.

• 34:29Speaker 06

  Right.

• 34:29Speaker 07

  And desiring and expecting them to, but he's just not capable. So that moment, we both sort of just curled into ourselves and just cuddled to him and just was just, you know, loving to him rather than like, why can't you get this? Come on, three-nager, do this, you know what I mean? Like, you know, so our, our breeding ground in research is our children.

• 34:49Speaker 06

  Right, exactly. And, you know, I, in my line of work, I mean, I will see the people where this sense of attachment and connection and feedback loop didn't go so well. And so they've learned, I always say, it's sort of like they jerry-rig things. Like, they learned how to best function in their lives as well as they could, but we, we know this, whenever we jerry-rig something and don't actually fix it the way it was supposed to be, what happens?

• 35:16Speaker 07

  Hmm.

• 35:17Speaker 06

  It breaks down.

• 35:18Speaker 07

  Well, if you like what you hear, you should go to changelog.com/brainscience. The show is not out yet, so don't get too excited, but you can subscribe and be notified as soon as the show launches. Once again, changelog.com/brainscience.

• 35:50Daniel Whitenack?

  I wanted to follow up on one thing that we talked about in a recent episode and a fully connected episode, actually, which was deep fakes. And I think in that episode, and of course, in general around this topic, there's pretty much a negative view of this technology. You know, GANs are bad and like they're destroying everything that's real in the world, sort of viewpoint. And in that show, we talked about like, oh, well, what are some of the potentially beneficial things that this technology could do? And I remember talking through a couple of things. We weren't like totally sure on what those were.

• 36:31Daniel Whitenack?

  But I saw an article, a recent article that took a really good perspective on this. And I remember it, I forget what the article was titled, but I remember seeing it mention like good GANs. And for listeners that, you know, want more info on GANs or generative adversarial networks, you can listen to our episode on that. But this article is basically saying that a research group is using GANs and kind of the technology behind deep fakes to actually improve cancer detection. So the detection of tumors and abnormalities in x-rays and CT scans and MRIs.

• 37:16Daniel Whitenack?

  So the basic idea is that you would take kind of existing imagery of cancerous tissue and you only have so much of that, right? There's only so much data that exists in a nice form. And so your models are kind of limited to that data. And similar to what we were talking about at the beginning of the show, people are much more considering kind of semi-supervised and unsupervised methods, intersecting various things, in this case, computer vision. So What this group did was to use GANs to actually generate new cancerous tissue imagery

• 37:56Daniel Whitenack?

  in the same sort of style to where it like looked pretty much like what a different cancerous imagery would look like. And so they're kind of using this to create this new data set of simulated data that can help them in training models for what potentially cancerous things will look like, which I think is really cool.

• 38:18Chris Benson?

  I do too. And you know, I'm glad you mentioned that because I think going into that episode where we talked about deep fakes in particular, I know for me, I had just finished watching the congressional hearing on deep fakes and the danger from a national security and all that stuff. And I think I was in, I was definitely in a dark place mentally in terms of going, whoa, and then we sat down and recorded. So apologize to listeners if that was a very ominous show. GANs in particular, I mean, not only, you know, you're talking about the medical application there with cancer. And I think I actually remember running across that article. But you know, in the past, we've talked about lots of different use cases, including, you know, the various types of creativity that can arise from it. You know, we certainly talked about that Christie's had auctioned.

• 39:02Daniel Whitenack?

  You can get rich with new artwork.

• 39:04Chris Benson?

  Yeah, absolutely. Well, you know, I don't know. I don't know why we're not focusing on that considering that.

• 39:08Daniel Whitenack?

  Yeah. What are we doing, Chris?

• 39:09Chris Benson?

  What are we doing, man? We could be making a ton of money with GAN based artwork. So.

• 39:13Daniel Whitenack?

  I guess maybe it's more meaningful to cure cancer, although I guess, you know, there's no cure for cancer, but to help treat cancer.

• 39:20Chris Benson?

  That I, you know what? I, okay, I'm going to give up my greedy, my greedy fascination with AI based artwork.

• 39:26Daniel Whitenack?

  Which one is better? Yeah.

• 39:29Chris Benson?

  So, but it's interesting as we're seeing these, these leaps ahead. So I think the GANs are going to be a big area of growth over the next few years. And I think that, you know, while some deep fake use cases are certainly on the negative side, I think we'll also see deep fakes that are being used in pretty creative ways. There is a learning, that was pointed out to me after that episode. I was talking to somebody at the Atlanta Deep Learning meetup and they were pointing out that, you know, gaming and That there's lots of educational uses where you change up use cases to teach somebody, you know, kind of dynamically on the fly. And, and, you know, they're just, there are tons of good potential uses as well. So it may just be that, you know, we've, we've had our attention drawn to some of the early nefarious things where bad actors have grabbed

• 40:14Chris Benson?

  onto a new technology and done, done some naughty things with it. So I think we'll see some pretty good uses of the technology as well, aside from the kind of things that we covered in the, in the previous show.

• 40:24Daniel Whitenack?

  Yeah, I wasn't really thinking as much about like simulated data when we were talking about that, having that discussion. But now that like this, I read through this article and thought through it, that, that's kind of at the forefront of my mind around this topic because, you know, the idea of simulated data, similar to, you know, things we've even already talked about in this, in this podcast is really valuable as we kind of move into this phase of really focusing on semi-supervised and unsupervised methods. And I think one of the things that you also found this, this week had to do with, with unsupervised learning. Is that right?

• 41:03Chris Benson?

  Yeah. So let me flip over to that one real fast. There, also in MIT Technology Review. Wow, we've done a lot of stuff.

• 41:11Daniel Whitenack?

  Yeah, it's a MIT Technology Review Week. So plug for, plug for the great work that they're doing, which is apparently giving us all the information that, that we're looking at. Maybe not all of the information, but it's certainly, they've been on a roll.

• 41:26Chris Benson?

  They have. We, we keep referring to them here. So good job to that group there. One of their articles was called, "The AI Technique That Could Imbue Machines With The Ability To Reason." And in it, they had talked to Jan LeCun, who is Facebook's chief AI scientist. And he has for some time been talking about the fact that, you know, the field at large is going to find, you know, the future great, great evolutions of the field in unsupervised learning as opposed to supervised learning. And he, he mentions, I'm scrolling down to it, that, you know, if we are going to think about-

• 42:05Chris Benson?

  About, you know, kind of how the way humans learn and the fact that, you know, we start off being born with a very, you know, limited set of knowledge, obviously. Coming out of the womb and that, you know, babies are really, they're not, it's not, they're very little of what a human learns in their process is through supervised learning. And he also pointed out that, um, that very little is, is done through reinforcement learning in the scheme of things. And that it's really self, uh, kind of self-supervised, unsupervised learning, meaning it's self-regulated. Unsupervised learning is how the vast majority of human knowledge comes about. And I gotta say that I agree with it and I actually got interviewed myself a while back by Thomas Reuters.

• 42:50Chris Benson?

  And I was saying essentially the same thing is that some of the really tough things that we have in this field, such as having the data that we need to train with and accumulating it, you know, are much harder than the actual AI work itself that we're trying to do. Unsupervised learning presumably gives us a whole world to start exploring out there without these giant obstacles to overcome before we ever get started.

• 43:15Daniel Whitenack?

  Yeah, the, uh, I'm actually working with an intern this summer and we're working on a sort of, uh, language-related, uh, graph-based method. But anyway, the, um, we're kind of, what is this like halfway through the summer, I guess, at this point?

• 43:31Chris Benson?

  Roughly, yeah.

• 43:32Daniel Whitenack?

  And, you know, we're still on the, the data issues of, of the problem. And I'm sure everyone that's actually worked on these problems can sympathize. Those are really, the bulk of the issues is often the, the data side of things. Thinking about, uh, another conversation I had, uh, with someone just yesterday where we were talking about, you know, how we basically just all like lost any confidence in any data that we ever receive or confidence in having good data. Because like regardless of the process behind the data, like even if there's a good process in place, humans are generating data and they're like biased in how they do it.

• 44:13Daniel Whitenack?

  And so almost any data set that we get, even if it's like nicely formatted, I kind of just lost confidence in, in all of that. Which is kind of, it's kind of depressing, but it's also kind of encouraging that people are really saying at this point, well, maybe, maybe we should focus on that less and focus more on this sort of semi-supervised or unsupervised approach.

• 44:34Chris Benson?

  Yeah, before turning to that, I'll agree with, I've been spending a lot of time focused on AI ethics lately at work. And I think it is a fair statement to say that an enormous amount of data out there is inherently biased in various ways. And, and so we're certainly going to need to address with tooling how we approach that. We can talk about more of that later. I know we're gonna have that as a topic in the future.

• 44:57Daniel Whitenack?

  Yeah, biased or even just like bad, like in some cases, data sets I've got like it, and it looks good. Like there's a field labeled this and it like, it's a good name for the field. And it's like, you know, language identifier or something like that. I'm like, oh, that's what this field means. And that's how everybody used it. But then I find out like later on, oh, that didn't mean the same thing to everybody. And they input all sorts of crap into that field, right? So it's like, it's biased, it's like bad data, it's like, everything's messy. So if everything's already messy, like, you know, unsupervised methods and, and these other things, you know, maybe that's, that's where we need to put more focus.

• 45:38Chris Benson?

  Yeah. And, and as, as Lacoon points out, um, he, he notes, and I'll quote, he said, obviously we're missing something. A baby can develop an understanding of an elephant after seeing two photos. While deep learning algorithms need to see thousands, if not millions. And a teen can learn to drive safely by practicing for 20 hours and manage to avoid crashes without first experiencing one. While reinforcement learning algorithms must go through 10s of millions of trials, uh, including many egregious failures. And, and he has a great point there. We, we have, uh, we have these amazing biological models that have inspired us. But we're actually not very close to them currently in the vast majority of work we're doing. And maybe some of the work that comes out of that will, uh, will lead to some pretty big breakthroughs. I'm rather hopeful there.

• 46:24Daniel Whitenack?

  Yeah, it's good to be hopeful. I'm a great point to, to end on. As we do in these fully connected episodes, um, we like to go through some of this news stuff and updates and all of that, that good stuff. But also part of what we want to do in these episodes is point you to some good learning resources that we run across. So as you're Trying to learn about these methods that we're talking about. I'm always looking for new things, new ways to learn things because there's so much information and so many methods to learn. So we'll spend the last couple of minutes here just going through some good learning resources that we've found recently. Chris, have you have you found anything recently that's been interesting?

• 47:07Daniel Whitenack?

  Either that you've gone through or that you're interested in exploring a little bit more?

• 47:11Chris Benson?

  Yeah, so one that I saw was Google Cloud on their blog, had a blog that is deep learning containers. And then the subtitle says, "Build your deep learning project quickly on Google Cloud," obviously. And what they're really focusing on there is if you're in their ecosystem, and obviously there's the pull that each of the major providers has, that they have these, a number of containers that are Docker containers that give you complete environments. And the purpose there is to be able to have a standardized environment. And I know the other providers are doing that as well. And I really wanted to call that out is that standardization on the same thing that the software development world is on Docker containers

• 47:53Chris Benson?

  is kind of becoming the de facto atom of work, if you will, about how people are getting things done.

• 48:00Daniel Whitenack?

  Everything cloud native.

• 48:01Chris Benson?

  Yeah, really. And so, and you can pull them onto your laptop and have the same environment. And in this case, Google is just talking about the fact that they have a bunch of deep learning containers of various configurations with different frameworks and different configurations that they found that their data scientists and deep learning engineers are needing on a regular basis. And so, I really, the reason I wanted to talk about this today and draw it out as a learning resource is the fact that this is really how things are getting done. Most of the organizations I've been in have been container based over the last couple of years, last two, three years. And they're really focused on how do we want to do it? And they develop a standard.

• 48:42Chris Benson?

  And so if you're not doing that, if it's something that you need to ramp up on, maybe you're stronger on the data science side and containers are still feeling kind of newish to you, this is a good area to jump into. Read this article, there's similar ones out there from AWS and Microsoft. And understand how you can accelerate what you're doing by using standardized containers that give you a full environment.

• 49:04Daniel Whitenack?

  Well, when looking and thinking about learning resources, what stuck in my mind was a conversation we had with Joel Grus, which I think is the previous episode to this episode and how they'll be released. But he wrote this book, Data Science From Scratch, which we've referenced actually quite a few times on the show, I believe. So that's a great learning resource. But along with that, I was kind of looking at other things online that were from scratch. So kind of this idea of learning things by coding things from scratch. And I ran across this NumPy ML repo on GitHub, which seems to have drawn quite a bit of attention, but I really hadn't seen since, you know, until I ran across it just

• 49:47Daniel Whitenack?

  in the last couple of days. And there is a ton of stuff in there. So it's basically a bunch of things that are implemented from scratch with NumPy. So if you're working in Python, there's this library called NumPy, and you can do like matrix-y things and other sort of operations with it, mathematical operations. And there's just a ton of stuff implemented there. So hidden, yeah, it's crazy. I pulled it up. I'm just, whoa. Okay. Yeah, it kind of took me off guard, for sure. There's like, you know, LDA, topic modeling, hidden Markov models, neural networks of all types, preprocessing methods, reinforcement

• 50:30Daniel Whitenack?

  learning, there's various utilities. Now, I'm sure like some of this, surely there's multiple people contributing to this. It looks like there's six contributors. So I'm not sure, I haven't gone through and looked at like the quality of all these things, but there's definitely a lot of attention that this has received. There's like 5,000 stars. So it seems like a reasonable place to go to just check out some things from scratch. So I'm definitely going to be checking this out and probably, you know, trying to learn from it and pull some of this into workshops and other things that I give.

• 51:05Chris Benson?

  Yeah, I am bookmarking it even as we speak right now.

• 51:09Daniel Whitenack?

  Good stuff. I was convinced by Joel that doing things from scratch is maybe not where you always want to do things, but it's it's a good exercise when when trying to learn something. So I think this is a great resource. The other things I wanted to mention were a couple of new ish. So one new ish, one new NLP related things. So as our listeners probably know by this point, I or I'm trying to learn more about NLP and and work in that space a little bit. But there's a really great course that's free on advanced NLP with Spacy from one of the the developers there.

• 51:48Daniel Whitenack?

  And it looks like a really, really great course and a really nice interface that's all integrated with nice notebooks and all of those things. Also, fast.ai, which has just a ton of other great resources.

• 52:02Chris Benson?

  Yeah.

• 52:02Daniel Whitenack?

  Have introduced an NLP course. So a code first introduction to NLP, which covers a lot of different things, but including things like transfer learning, issues of bias, text generation, attention and the transformer. So all all sorts of great stuff there. So check that out if you're if you're interested in those topics as well.

• 52:24Chris Benson?

  Great finds, Daniel.

• 52:25Daniel Whitenack?

  Yeah, it's it's great that people are releasing such amazing quality material for for free as well. And in a really, in really great interfaces, there's a lot you can do without having to, you know, pay for a college course or pay for expensive books and other things. There's a lot of, a lot of good stuff out there.

• 52:47Chris Benson?

  Yeah, we had, and I wanted to call out that we had an episode, just a few episodes back, where we talked about how to get into the field and we had a long list of educational resources, courses, books, videos, a lot of which were free. There were some really top tier things in there that cost nothing other than your time and attention. And so we can add a link to that in the show notes as well. But yeah, great finds here. There's never been a better time to jump into this field right now because you can. You know, even if you're tied down, you're working full time, you have a family. Anytime you can get on your, on your laptop to, to focus on these, you can, you know, log into a course that doesn't cost anything, keep working through it and stuff. So it's a good time to do it.

• 53:28Daniel Whitenack?

  Well, I don't know a better way to end than with that pitch. So thank you for, uh, thanks so much for walking through some recent news with me, Chris. It was great to talk again as always, and I wish you luck with your drones.

• 53:44Chris Benson?

  Um, we're gonna go have some fun flying them. I'll see you, I'll see you next week.

• 53:49Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do here at changelog because of Robar. Check them out at robar.com/changelog. And we're hosted on Lino Cloud servers. Head to lino.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop

• 54:31Speaker 00

  in your email address, get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.