2019's AI top 5

Wow, 2019 was an amazing year for AI! In this fully connected episode, Chris and Daniel discuss their list of top 5 notable AI things from 2019. They also discuss the “state of AI” at the end of 2019, and they make some predictions for 2020. Sponsors: DigitalOcean https://do.co/changelog – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. Brain Science https://changelog.com/brainscience – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Google search and BERT https://www.blog.google/products/search/search-language-understanding-bert/Better Language Models and Their Implications https://openai.com/blog/better-language-models/Write with Transformer https://transformer.huggingface.co/Hugging Face Transformers https://github.com/huggingface/transformersHugging Face receives 15M in funding https://techcrunch.com/2019/12/17/hugging-face-raises-15-million-to-build-the-definitive-natural-language-processing-library/Solving Rubik’s Cube with a Robot Hand https://openai.com/blog/solving-rubiks-cube/Training a single AI model can emit as much carbon as five cars in their lifetimes https://www.technologyreview.com/s/613630/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/Other relevant Practical AI episodes: Robot hands and Rubik’s cubes https://changelog.com/practicalai/64BERT: one NLP model to rule them all https://changelog.com/practicalai/22OpenAI’s new “dangerous” GPT-2 language model https://changelog.com/practicalai/32Social AI with Hugging Face https://changelog.com/practicalai/35Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:18Speaker 01

  This episode is brought to you by Digital Ocean. Digital Ocean provides worry-free database hosting with their managed databases. If you need to get data in and out of Postgres, MySQL, or Redis, call on the world-class support teams at Digital Ocean and stop wasting time on setup, backup, and maintenance. Get simple, predictable pricing. Get detailed documentation. Get up and running in minutes so you can get on with your business. What are you waiting for? Head to do.co/changelog. Again, that's do.co/changelog.

• 1:03Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slake with us around various topics of the show at changelog.com/community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:28Chris Benson?

  Welcome to another Fully Connected episode where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I am Principal AI Strategist at Lockheed Martin. And with me as always is Daniel Whitenack, who is a data scientist at S I L International. Hey, how's it going today, Daniel? It's going pretty good. It's 2020. Crazy, man. Happy New Year, man. Woo. So we have, we have just put to bed our first full calendar year of podcasts as we started mid 2018. It's I think if I do the math.

• 2:10Daniel Whitenack?

  That's right. This is episode 71, unless we unless we switch anything up. But yeah, so 70 plus. It's pretty, pretty exciting. I don't know what we'll do when we hit 100, but we'll

• 2:24Chris Benson?

  We need to think about that.

• 2:25Daniel Whitenack?

  Have something exciting for listeners when we when we hit 100 for sure.

• 2:29Chris Benson?

  So and if any of the listeners out there have any suggestions for that, let us know. You know, join us in our Slack channel where we are on all the all the time, every day, talking to people. Or you can reach us on LinkedIn or Twitter. We are definitely out there having conversations with you guys.

• 2:47Daniel Whitenack?

  Yeah, definitely. And pretty soon, just as a as a final reminder for people, I think might have missed mentioned this on other episodes, but both of us will be at the Project Voice Conference, which I think as far as when this episode airs will be the following week. So January, like 13 through the 16. So if you're around at Project Voice, come find us. We'll be recording some some episodes in the S I L International booth and as well giving a keynote together, both Chris and I. So it'll be fun to be there and think a little bit about speech and voice and AI and what's going on in that world.

• 3:27Chris Benson?

  Absolutely. That'll be in Chattanooga, Tennessee. I think on the the week of the I think it's Monday, the 13th.

• 3:34Daniel Whitenack?

  Yep, exactly.

• 3:35Chris Benson?

  If I'm recalling correctly. So you know, today we're going to do the same thing we did about this time last year as we got into 2019. We really wanted to kind of look back on a couple of notable points in the AI world in 2019. Talk about kind of why we think they were notable and also kind of assess the current state of AI where we are right now. And then look ahead to 2020. And and of course, it would not be a start of your show if we didn't try to make a few predictions. Each of us on where things are going over the next year.

• 4:08Daniel Whitenack?

  It is practical AI. So I feel like we have to make predicting has to be a part of it.

• 4:13Chris Benson?

  There you go. Of course. Absolutely.

• 4:15Daniel Whitenack?

  And the predictions will be likely wrong, but maybe if after we do this for like 20 years, we can we'll have a proper test set of predictions that we can really determine what our what our accuracy was.

• 4:29Chris Benson?

  Oh, I'm not looking forward to that result. I'm not sure that's good. And I'm not sure if we should call them predictions or inferences, considering the field that we're in here.

• 4:36Daniel Whitenack?

  Maybe that would be better.

• 4:38Chris Benson?

  One of the things that we were talking about before we started recording is it's been an amazing ride so far, and that is entirely due to our listeners and our guests. And as you pointed out, you know, we just would not be where we're at with the, with the, the show being as popular and so many people expressing how helpful it's been for them to get into this field and understand the detail.

• 5:01Daniel Whitenack?

  Yeah. Thank you to our listeners and our guests. Um, I mean, the, the guests for sure, of course, are a lot of the great content comes directly from them. Chris and I are mostly, uh, you know, I feel like a lot of times we're just kind of facilitators and there to listen to the great content that is there. So thank you to our guests. It has been great to get, you know, uh, feedback on our Slack channel, talk to people on Twitter, talk to people at conferences who are aware of the podcast and, and are getting value out of it. And a lot of that is because we do get feedback. We hear it would be awesome if you did a show on this, or I'd love to hear about this. And, and we definitely try to integrate those things in. So thank you to being part of the community. I hope you

• 5:45Daniel Whitenack?

  feel welcome and are excited for 2020 like we are.

• 5:49Chris Benson?

  Absolutely. I think a huge part of the show is the community aspect of it, even more so than the, the technical. It gives people an ability to connect. So thank you all for constantly talking to us over this past year and a half and making sure that we stay on track on how best to, to meet your needs. So I guess with that said, I know that on a couple of things, looking back, we definitely.

• 6:10Daniel Whitenack?

  Yeah. Top things of 2019.

• 6:13Chris Benson?

  Absolutely. We definitely are in agreement on quite a few of those. You want to kind of, one of those big topics is transformers. And you want to kind of, kind of jump into that and set that up?

• 6:23Daniel Whitenack?

  So when we were thinking about, you know, the top AI milestones or notable things of, of 2019, we both kind of drew up our own list of things that, that we were interested in or, or thought were notable. There was a little bit of overlap, but the big piece that was overlap was transformers. And, um, we have an episode that talks about this in a lot more detail. Specifically related to BERT and we referenced GPT-2 a couple of times, but if you aren't aware of those episodes or haven't listened to them, 2019 has kind of been the year of the large language models, the year of the transformers. So this kind of got kicked off with BERT and GPT-2 and other models

• 7:08Daniel Whitenack?

  that were really large scale language models that in essence were able to learn a lot about language in general by being trained on many, many documents. So lots of data scraped from the web or other places. And we're able to transfer to a lot of different NLP tasks. So whether that's machine translation or reading comprehension or named entity recognition, all sorts of things, text classification. These models have allowed us to sort of have a zoo of really large pre-trained models

• 7:50Daniel Whitenack?

  that know a lot about language and transfer those easily to these various tasks. And so we can kind of stand on the, on the shoulders of giants in a sense of OpenAI and Google and others who have trained these large models, have a lot of data. And then allow us to kind of just level up our own NLP game by utilizing these, these pre-trained models. And that's, that's been a huge boost to NLP this year in particular.

• 8:19Chris Benson?

  I think the thing that really struck me about it is you and I actually come at this from different perspectives. You are a true NLP expert. Anyone who has listened to our episodes very much, when we talk about this will know that. It's what you do all the time. I observe it, but it's not my specialty. So I'm kind of coming from an outsider's perspective on that. And the, the thing that really struck me is these, these new large language models just impacted the entire world of deep learning and industry at large. Whether or not you were neck deep in it the way you are, or whether you're really kind of watching this from outside and just being a user of these externally, the way I am. And so it was like. Like the big hits just kept on coming through 2019 as we did this.

• 9:02Chris Benson?

  I know I was absolutely, as was probably most people, blown away when OpenAI did their first blog post on GPT-2. Early in the year, I think it was February, if I'm recalling correctly. And even as they, as they introduced it, they noted, you know, a couple of things that it is a, as we have specified, it's a transformer language model. And it's used as a generative model of language where you can essentially give it a sentence to start with and it will generate a great deal of text, which in many cases is indistinguishable to the casual observer, you know, on whether or not it came from a computer or from a human. And it was pretty amazing when we saw that and they did that initial release, which was a scaled down version.

• 9:48Chris Benson?

  Just to let the world try it and they had to, they were kind of recognizing there could be security implications on that. They were slow to release and released in stages. But ultimately, if I'm recalling, the larger model they released later on in the year was trained on, on webtext, which contains over eight million documents for a total of 40 gigabytes of text, which from, you know, if that was images, it wouldn't be so much, but for text, that's enormous. And they, they pulled that from URLs on the internet in an unsupervised approach that was from Reddit submissions, in which case they had at least three upvotes. So they had a huge, huge corpus of text to pull from.

• 10:27Chris Benson?

  And I just remember seeing those, or those early examples of what was possible and thinking, okay, we're, we're in a new place on the NLP front at this point.

• 10:35Daniel Whitenack?

  Yeah, definitely. I've talked to many colleagues who have expressed specifically with that blog post that you mentioned that prior to the blog post, if you were to ask them, hey, you know, what's the best that an AI model could do in generating text? Regardless of, of architecture, kind of everything that's been done in the past, what's the best we could do? They would have guessed, you know, a much lower quality than what was published in that blog post.

• 11:03Chris Benson?

  Yeah.

• 11:04Daniel Whitenack?

  And just kind of being blown away by that. And then, of course, That fueled all sorts of things throughout the year. So I think these years kind of 2018, 2019 have been referred to as NLP's ImageNet moment. So if you remember further back when ImageNet came out, which is a was a challenge around object recognition and computer vision. There was, there was a huge boost in computer vision and AI. And this, I think, is kind of a parallel in, in what's gone on. And so there's just been an explosion in all sorts of things that build on this technology.

• 11:46Daniel Whitenack?

  So the technology itself, you know, these large language models, again, they're not, they're kind of building blocks in, in a way. We, we talked in the blog post about BERT, about how these are structured often into sort of encoding layers and decoding layers. And how you can utilize BERT or these other models to create these sort of word embeddings or representations of text that can be used for a variety of tasks. And so that spurred not only innovation in sort of text generation, but innovation in all sorts of NLP tasks. Like I mentioned in, in translation and all sorts of things. And I just saw one of the big indications of this, I think, is I saw that

• 12:30Daniel Whitenack?

  Google Search, which is arguably Google's kind of bread and butter, right? They just switched over to actually integrating BERT, which is one of these transformer models, large language models, directly into Google Search in production, you know, live now. So I don't think Google would be taking that risk if they weren't convinced that this was a transformative technology. So that's pretty cool.

• 12:56Chris Benson?

  Is almost a meta issue around this. There was quite a bit of controversy in how GPT-2 was released. And, you know, we already talked about the, the staged release that they did. In that original blog post, they note under release strategy, they say, we're not releasing the data set, the training code, or the GPT-2 model weights. And they specify that we expect Safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety policy and standards research. And that was really the first time that a major AI research organization had done, you know, everybody up until that moment was just publishing as fast as they could

• 13:38Chris Benson?

  as new stuff came out. And that was the moment where they suddenly said, we have a greater concern. And there was quite a lot of debate about, you know, whether or not that was the right approach or not. I know we talked about it on the show a little bit, but it was just interesting to see how that policy debate shaped out over time.

• 13:55Daniel Whitenack?

  Yeah, I would specifically, you know, like to note, and longtime listeners of the show will know that I, I like to mention this group quite a bit because I really think that they're a big part of what's, what's happening. I specifically don't think that the momentum that's built up this year around transformers would have been quite as much without Hugging Face's contribution. Absolutely. So we had Clem from Hugging Face on a while back. We'll reference that episode. That was actually before a lot of the stuff I'm about to talk about really built up. But after that episode, Hugging Face, they came out with a few things. One of those was this application called Write with Transformers,

• 14:40Daniel Whitenack?

  which is a really, you know, for non-technical people, you can just go to this app and choose any of these language models you want and just try to generate some text with it. And it's kind of like a Word document where you can kind of integrate these models. And I think that was just like a huge eye opener for people because not, a bunch of non-technical people could go in there and do this. It also forced Hugging Face to really deal with this, well, how, how do we productionize these models? How do we integrate them practically? Which led them to release the Transformers library, which is one of the widest used NLP AI libraries.

• 15:21Daniel Whitenack?

  I think that's been mentioned a lot this year at top conferences, even all the research conferences, but industry conferences, even. TensorFlow Dev Summit, even though Hugging Face has traditionally worked with PyTorch, I think. So this is really transformative. I even, in my car, often I listen to NPR. I do too. I was listening to NPR and there was someone on there, I forget the exact topic, I don't remember the context, but they were talking about AI and they were like, "Well, I can use an AI model to generate some new NPR show titles for this show." And they used RightWithTransformers, the app from Hugging Face, to do that sort of on the show, which was pretty cool.

• 16:06Chris Benson?

  So I'm looking over what they do and they've done such a good job of integrating their transformers in with the existing tooling. As TensorFlow 2 was out this year and PyTorch, and those two still probably, I know there'll be some people disagreeing with me, but probably the dominant two frameworks and the tight integration. They've really made NLP not only powerful, but incredibly accessible to people. In your view, I know that you follow them very closely, even beyond us having them on the episode. What do you think Hugging Face has done so well and so right? They've kind of become, to some degree, it seems, the darlings of the NLP world this year. That's at least how, that's my own feeling of it.

• 16:51Daniel Whitenack?

  Yeah, for sure. I mean, I think that a couple of things that maybe can be extracted from that and we can learn for our own work is that they have focused on making things, sort of giving people an immediate satisfaction with using these tools. So the RightWithTransformers thing, you don't have to even go to GitHub or download any models or anything. You just go and you try it out, right? And so that, I think, is one thing is kind of, that we could extract from this is, you know, making AI consumable to all sorts of audiences is something that is incredibly valuable. But then also for developers, prior to the transformers library, it was still rather difficult to integrate these large-scale models.

• 17:38Daniel Whitenack?

  Models into a normal workflow. And transformers really gave a standardized API that people could use to pull in these models, utilize them for various tasks, or just utilize them for generating embeddings. And so I think that sort of standardization is something that we also saw with Spacy. So Spacy, which we had on the show recently.

• 18:02Chris Benson?

  That's true.

• 18:03Daniel Whitenack?

  It has been and it still is extremely popular in the space, in NLP space. And I think those are also characteristics that we've seen with Spacy where they value good design, they value a good user experience, they have a nice way to standardize the sort of workflow around NLP to these sorts of pipelines. So I think those are really key ideas that we could take away. And just to give Hugging Face kind of a final congrats, they ended the year with an announcement of 15 million in funding to continue development of transformers and what they're doing. So I think it's worth taking time to mention them and always happy to.

• 18:45Chris Benson?

  Absolutely. They've had such a profound impact on the industry this year. It's they've just been very, very impressed with them. It was a great episode. So if anyone out there hasn't listened to that episode, you definitely should.

• 19:11Speaker 01

  If you like this show, I bet you'd enjoy listening to Brain Science. Join clinical psychologist Muriel Rees and Adam Sokoviac as they explore the inner workings of the human brain to understand behavior change, habit formation, mental health, and being human. Here's a quick taste of what you can expect. It's from episode four about coping skills and strategies. Take a listen.

• 19:32Speaker 04

  I often use this acronym with people when they're trying to cope because and it's halt. H A L T halt. Because if we are hungry, angry, lonely, or tired, your coping will invariably look different. I don't care if you're three, thirty-three, seventy-three. Right. If you are hungry, or hangry, angry, lonely, or tired, you just have less to be able to navigate it.

• 20:03Speaker 01

  Brain Science is a great podcast. Check it out at changelog.com/brainscience or just search Brain Science in Apple Podcasts, Spotify, or your favorite podcast directory, you'll find it. While you're at it, upgrade to our master feed at changelog.com/master and let your podcast app download all the shows we produce. Then you can pick and choose the ones you're interested in the most and skip the rest. What have you got to lose? All right, back to the show.

• 20:39Chris Benson?

  So there were a couple of things this past year. I don't know that they were the the most important things necessarily, but they were certainly events that that really captured my imagination. And we did have actually episodes on on both the things I'm about to mention. The first one, people may recall a few months back, OpenAI did some work with robotic dexterity, using a robotic hand that where the hand was trying to solve a Rubik's Cube. And just to specify, and we had a whole episode talking about this, it wasn't the algorithm of the Rubik's Cube that the AI portions were solving because those are known solutions out there. So they just implemented one of those. But what they were doing was using reinforcement learning to really get the dexterity and sensitivity

• 21:24Chris Benson?

  of the robotic hand to really a whole new level that had not been doing. And they shared some videos out there showing the robot manipulating the single robot hand manipulating the cube in all sorts of ways. And it really inspired me seeing that, to see the delicateness of it, the

• 21:42Chris Benson?

  Ability of being able to to do very minute turns on the cube with with digits and it was interesting that it would make you hold your breath as you watch the video and at moments the Rubik's Cube would roll right up onto the fingertips of the robot and it would stop, balance there and then spin. And it just made me realize that we were at the dawn of a new age for robotics in terms of what AI could do to really supercharge where robotics are right now, not only in in more traditional movements and such, but but also in these tiny little dexterity things with with sensors that are able to capture delicate things. And you could, after watching that video, you could easily think of robots, you

• 22:23Chris Benson?

  know, as we've talked about medicine and things, you know, doing incredibly dynamic and precise forms of surgery, you know, on humans in that way, that if you had all of the right sensors and everything, that you could take AI and robotics in medicine to a whole new level. And that really had a fairly profound, for just one story, it had a fairly profound impact on my perception of the state of the art. How about yourself?

• 22:49Daniel Whitenack?

  Yeah, it was interesting. And this is a space, you're much more familiar with the space, but I think the the things that stood out to me with that was their focus on making the models robust against perturbations and sort of new scenarios. So they developed these techniques around domain randomization and increasing the randomization during during training, such that the the hand was able to deal with all sorts of unexpected scenarios. And I don't know if it's accurate, maybe you can tell me, but it seems like maybe one of the things that's that's held back AI and robotics a bit is is this fact of generalizing

• 23:34Daniel Whitenack?

  to all sorts of different scenarios. Like if you're saying with medicine and surgery, you know, people are come in all sorts of different shapes and sizes and ages. And so having, you know, a hand that would perform certain procedures would need to deal with

• 23:52Daniel Whitenack?

  Scenarios and you can't have all of those in your training set. So how do you make sure that your your system is able to extend and generalize to different scenarios? I think that that focus in the project was was really interesting to me. If that focus continues, that maybe there's a way to to kind of push the boundary there.

• 24:13Chris Benson?

  Yeah, it's really created a revolution in robotics in terms of, you know, we've had robotics for decades, you know, deployed in various industries, particularly industrial uses. And for a long time, everything about, you know, for instance, an assembly line had to be very precisely measured. And there could not be variability, substantial variability in those workflows. And so we've really seen over the past two or three years. At the moment, culminating in this robotic dexterity demonstration that we saw is the ability to accommodate some variability and have the ability to make change based on that variability dynamically in real time. And so when we're looking forward, and I know we're going to talk about kind of the world ahead,

• 24:58Chris Benson?

  you know, time ahead later on in this episode. But it really starts creating new possibilities in terms of using these in scenarios that it just wasn't practical and realistic before. And so it was a neat demo. I don't think people should get too hung up on the Rubik's Cube aspect itself. I just think that was a tool to show what they were getting to. But it was a it was a pretty cool moment. The other thing that had a very profound effect on me this last year, and we had an episode on it kind of mid-year, I think it was in June, was on deep fakes. And more specifically, the very realistic types of deep fake videos where you're using a generative adversarial network again to generate those deep fake videos.

• 25:39Chris Benson?

  And I think the thing that became obvious, not just to us in this field, but to the public at large, you know, we had congressional hearings on it, was the fact that you were now entering a moment with this tool, which could be used for both wonderful and nefarious purposes. It's not all bad, but you're really blurring the lines of what is real and what is not with this capability. And there can be fantastic. Wonderful things you go to an amusement park where they're able to implement deep fakes you know in rides and it can be a lot of fun you know where it personalizes the riding ways and you could do some pretty cool stuff or obviously you could have you know things as bad as national security concerns you know about elections the US elections in 2020 where

• 26:20Chris Benson?

  we already have had the American FBI and the intelligence community warn us about that it is highly likely that we'll have adversaries and strategic competitors trying to interfere in the elections and then as I've heard some other people talk about you know what happens right now as we look at this technology and we have a little bit of time to assess it in some cases what happens when we get to a situation where there is no time to figure out what is real and what is not if if you had a deep fake that showed the President of the United States describing that he had just launched nuclear weapons you know and you're you're somebody out there who may be the target of that you know and that's not a real video how do you assess in a responsible appropriate but expeditious manner to do that so we're in

• 27:03Chris Benson?

  a world that has changed in terms of our ability to know what's real and know it in essentially real time any thoughts on that

• 27:12Daniel Whitenack?

  yeah I mean I think you're definitely right and along with that we've seen an increase I think in research into detecting fakes which is encouraging and I hope that continues and then also I know in a few of the episodes after we talked about those sorts of things it's come up that there definitely are good uses of this technology we've talked about you know generating medical imagery of tumors and that sort of thing which is is very expensive to annotate and generate manually but you know we can kind of simulate that data and create simulated data using these methods that can improve you know tumor detection algorithms and that sort of thing

• 27:54Daniel Whitenack?

  so with any technology I think there's good and bad sides that you could draw from it I think this one the deep fake things and the videos that came out emphasized the bad ones first and so it'll be interesting to see as as GANs are become more and more And integrated into different systems, what the positives are and how we deal with those those other negatives.

• 28:19Chris Benson?

  Yeah, there was one website I came across and it used the deep fakes that were GAN powered to to animate the Mona Lisa. So it took the, you know, what is the most famous painting in the world and the Mona Lisa was was busy gesturing and talking and stuff like that. So it was a cute thing. And there's, I think we're going to see many good uses.

• 28:38Daniel Whitenack?

  Yeah, some of the uses are just kind of interesting in that sense. There's nothing, I don't know who uses that sort of animated Mona Lisa for any sort of practical purpose, but it is still fun and it's pushing the boundaries.

• 28:53Chris Benson?

  One thing slight downer, not national security level downer, but I think I've heard, I've read some stuff that telecalls, you know, that you get from marketers and things like that, that's supposed to be the next big wave is people scraping social media sites to get images of you and people you know, and then trying to mimic voice or whatever on on those. So beware as we go forward over the next year or two, that that kind of thing could be at a very personal level. It doesn't always have to be these giant end of the world scenarios. It can be something that is very immediate and known to you.

• 29:24Daniel Whitenack?

  For sure.

• 29:24Chris Benson?

  What about you? What are, what are some of the things that that you noticed in 2019 that were awesome?

• 29:29Daniel Whitenack?

  I think one thing that we definitely have to note is TensorFlow 2.0. So I think the the final official release of TensorFlow 2.0 was November 9th, if I if I searched that right. I mean, I use Google search. So if that's the wrong date, then I guess they can blame themselves.

• 29:47Chris Benson?

  I was gonna say, if anyone should know the date, it should be them.

• 29:49Daniel Whitenack?

  But yeah, TensorFlow 2.0. So for those that aren't aware, with the release of TensorFlow 2.0, TensorFlow made quite a few significant changes, especially to the, yeah, yeah, I mean, especially to the sort of default API to TensorFlow, which is now Keras, and also to the way in which computations happen instead of always generating this static graph that has to be executed later. And so I think TensorFlow 2.0 was just an amazing. Demonstration that the TensorFlow team, this is coming, I guess, I should say, this is coming from a PyTorch guy.

• 30:30Daniel Whitenack?

  I've, I've used PyTorch way more than I've used TensorFlow and I really enjoy PyTorch and still really enjoy PyTorch and use it a lot. But for me, you know, I think it's a great demonstration that the TensorFlow team saw that, oh, we have this really powerful technology. But we've gathered feedback that we need to kind of shift some focus in some areas and make it more usable. Make it more approachable and make it more practical. And so I think the sort of usability and practicality of TensorFlow 2.0 is just amazing. And I think that they should be given congratulations for, for an amazing release and I can't, can't wait to see more.

• 31:09Chris Benson?

  Yeah, as, as someone who used both the version one and version two now, I much prefer version two and you can use it. You can use the Keras interface for the vast majority of use cases that most people are likely to see. Much more user friendly and it was funny, this past year, a couple of conferences. I tend to, to keep my skills up. I'll go to TensorFlow classes and stuff. And I remember it was several months after the TensorFlow 2 beta had come out. We weren't to final, to be full disclosure, you know, and I, so I can understand. But I remember going to a class, it was a TensorFlow class, the entire class on the beginning of that first day was immensely disappointed that we weren't using the TensorFlow 2 beta.

• 31:48Chris Benson?

  In the class instead of TensorFlow 1, I felt sorry for the instructor. I'm going to keep all the identities out of it, but it made that big of a difference in that community. And so kudos to the TensorFlow team for listening to user feedback and turning out a great product that made great strides on the first one.

• 32:06Daniel Whitenack?

  Yeah, I just tried this to see how easy it was to find and I just searched for TensorFlow 2 code lab notebooks. Cause that's probably where I would start if I was trying to find something or you could probably also search TensorFlow 2 quick start. Cause the, the first two results are TensorFlow 2 quick start for experts, TensorFlow 2 quick start for beginners. And if you go in there, it walks you through the code itself, but also they have nice. CoLab notebooks that you can open and try things out without even running anything locally. So it's super easy to get into and would recommend people to take a look. Probably the last thing that I want to mention, I mean, there was so much in 2019.

• 32:46Daniel Whitenack?

  I'm sure we're so sorry to all of you out there who are leaving, leaving out your favorite thing from 2019. But the other thing that I wanted to mention in 2019 is something that I detected throughout the year. And that was a sort of realization, which hadn't been there in 2018, that at least the way I felt it this year, was that training AI models is super compute intensive. And this year I felt a little bit of pause from the community in saying, hey, how much energy are we expending to train these AI models?

• 33:24Daniel Whitenack?

  And what can we do to make that more efficient and more responsible in terms of the environmental impact and all of that? So an article was released in 2019, I think, which caught a lot of people's attention, that, you know, training a single AI model, one of these larger language models, for example, can admit, just training it once, a single model can emit as much carbon as five cars during their whole lifetime of use. And which is pretty staggering. And I personally felt like, you know, not everybody took this seriously this year in the community necessarily. And it's not like training large models has stopped.

• 34:06Daniel Whitenack?

  But I do think there is beginning to be a sense that we need to really pursue technologies that make AI more efficient and responsible in that sense.

• 34:18Chris Benson?

  Yeah, I remember us talking, I don't remember which episode it was, but I remember when it came out, we talked a little bit about that. And I know that both of us are very environmentally focused in terms of being responsible. And so I was very happy to see people taking it seriously as well. And I heard a lot of conversations through the year about the topic. So I think it's a problem still to be solved. I think, you know, when you have very large scale. Model training you have to do, there are currently not enough, we don't have enough solutions out there yet in terms of having the compute capability and yet still be able to to be responsible because you know, this technology is is here to stay. We're going to be computing more and more. And so we need to be thinking about those those responsible

• 35:00Chris Benson?

  solutions, just as we have in other aspects of AI that have come to pass that we'll be talking about in a few minutes.

• 35:05Daniel Whitenack?

  Yeah, there's multiple facets to this. So I mean, there's the side of things, which is of course, making data centers more efficient and also running those off of sustainable energy sources. And I think that's been going on prior to this year. And there's been a good amount of effort put into that. But also, I think the the pieces that I've seen develop this year, much more emphasis in sort of distilling and optimizing models to make them more efficient, make them run faster, which is partly driven by just practicality, right? If you're using a model in production, and it's smaller, or you're wanting to port it to a mobile device or something like that, it needs to be smaller. So some of those

• 35:49Daniel Whitenack?

  things factor in as well. But also, I've seen some efforts in sort of envisioning new, more efficient architectures for for modeling. So not always relying on, let's say, the next larger transformer model. But are there other architectures, maybe just regular RNNs, that can do this task just as well as using, or almost as well as using, you know, the full large size BERT model, and are much smaller and can be trained in much less time. So I think, yeah, we need to approach this from various angles. But I think it's something that people started hopefully taking seriously in 2019.

• 36:34Chris Benson?

  Yeah, I think we have the benefit of the fact that it doesn't require only a mindset in terms of responsibility toward the environment, but also just just sheer performance. You know, if you're able to find these other approaches that are allowing us to actually get there sooner, it's better for all concerned. So, you know, one of the things before we turn to what the future looks like is kind of, let's take a moment and kind of assess where we are right now. You know, we've just gotten to the end of 2019. We're at the beginning of 2020, not only the beginning of a new year, but the beginning of a whole new decade. And so what are your thoughts toward where we are now as we hit this point? Anything? Daniel?

• 37:14Daniel Whitenack?

  Yeah, sure. I think one super positive point of where I think we are and will continue to be in 2020 is really an amazing place in terms of the practical side of AI, which is what we're concerned about a lot here on the Practical AI Podcast. And I say that because you have these things, like we already talked about, like transformers, but other libraries as well and other toolkits or just code on GitHub, whatever it is, infrastructure pieces, tooling. I feel like as compared to where we were at the end of 2018, there's just a lot more ways to be sort of robust and build system, AI systems with, that have

• 37:59Daniel Whitenack?

  a lot of integrity in a much shorter period of time than we used to be. It kind of used to be very much the Wild West, and maybe we still are a little bit in the Wild West. But I think that a lot of the principles from software engineering have kind of come into the AI world and we're a lot more focused on versioning things, tracking things, monitoring things, whether that be with tools like TensorBoard or other things, or it's like infrastructure pieces like Pachyderm and things like that, CubeFlow. We're just thinking a lot more about the AI systems that we're building rather than just AI models. And I

• 38:42Daniel Whitenack?

  think that's really encouraging and it helps people that are actually trying to build products and be practical and integrate AI. I think there's so much opportunity there and there's so many choices available in that regard.

• 38:55Chris Benson?

  Agreed. Just seeing, you called out something a moment ago that really struck me, and that is- When we talked about this a year ago now, you know, going back to that episode, so much has changed. You know, we used to, when we first started this podcast, we were always searching around for good tutorials and examples. And sometimes we would struggle a little bit to find them in just that, that amount of time. And especially in the last year, there's so much available out there. The open source tools have really matured, great communities, the tutorials enabling people to do that. And, and we're finally seeing some of the surrounding infrastructure and tooling improving.

• 39:35Chris Benson?

  I think, I think there's still a struggle there as people really try to productize what, how they get models, not only trained, but deployed in the rest of their environment. But I think that's, that's definitely something that's, that's working hard now, you know? And another thing that I've really noticed, I know at my job at Lucky Martin, I'm very, very involved in our own AI ethics and responsibility initiatives. And so I spend a lot of time focusing on that. And, you know, over the past year, we've seen pretty much all the major players out there, whether they be Google or Microsoft or, and many others, releasing ethical frameworks and, and their principles and such. And I think it's really gotten called out.

• 40:18Chris Benson?

  The, the difference between now and, and last year at this time, where people were starting to talk about ethical AI, but the, the conversation has matured a great deal. And the recognition that even with some of the limitations of where we are right now in terms of what deep learning can accomplish, that the dangers of abuse are very real. And we're seeing lots of the significant luminaries in our field, kind of calling that out and expressing a need for standardization as we go forward on that. So that has been a fairly significant change in the last 12 months.

• 40:52Daniel Whitenack?

  I think things like, for example, China's use of facial recognition, which we've talked about on the show before, and you know, Russia's use of behavioral modeling and that sort of thing to influence, for example, elections, those have hit everybody, right? And have Been just kind of widespread or have been acknowledged in a sort of larger sense that AI isn't sort of something that is really cool and for sci fi, but there's like real uses of it that are going on, but not only real uses, but potentially really bad uses as well. So.

• 41:33Chris Benson?

  Yeah, you know, the, and I know we've also talked about it in previous episodes, but the, you know, as an example of something that it depends on, on where you are in the world and your values, but I know based on generalized Western values, China has their social credit system. And as we have been looking at that and talking about that for some time now, you know, they're using AI to not only surveil, but analyze and monitor their citizens and, and either reward or punish them accordingly. And so, you know, that's such a profound effect upon that particular country and the society that it's given us a lot to think about in terms of what do we want? If you live in a democracy where you have a say so and how things are

• 42:16Chris Benson?

  implemented and your one voice of many that can contribute to that voice, I certainly hope people are thinking about what is right for you and the community that you live in, no matter where you are and where does that make sense? And I think so that that has become, it's gone from being a fringe conversation to becoming a mainstream conversation in this past year, I'd say.

• 42:34Daniel Whitenack?

  Sure, sure. So one, one thing that, that I'll bring up in terms of where we currently are in terms of the state of AI going into 2020 is I think that as we move forward, it's going to be more and more crucial that if we're really serious about using AI to tackle large scale problems like climate change and the death of languages around the world, access to good healthcare around the world, we're going to have to better involve researchers and developers from all over the world. So we've had some really encouraging things this past year and,

• 43:14Daniel Whitenack?

  and things going into next year around that, like, you know, various workshops being held around the world. In Southeast Asia and Africa, there have been conferences that have been placed in those areas. There's like the deep learning in Daba in Africa that's going on and offices of Google and and others that are opening in those areas. But we're definitely not where we need to be. For example, you know, NERIP still this year, there was a huge problem with researchers from around the world getting to NERIPs and having their their visas denied.

• 43:53Daniel Whitenack?

  You know, if you just look at publishing, we're still pretty driven by the US, by Europe in certain areas. And so if there's been one thing that's been clear to me as I've worked more with the NGO I'm a part of and also other NGOs, is that if we really want to make an impact on these sorts of problems, we need to have representation from these local communities. You know, we can't just take, for example, if we want to extend translation, like Google Translate to all sorts of languages. We can't not involve these communities because there's, you know, we can't just publish research papers that say

• 44:34Daniel Whitenack?

  we're studying low resource languages and we just undersample English as our low resource language because that leaves out so much. It leaves out unique scripts, problems and unique domain issues and cultural things. And so I think there's a lot of shifting that needs to happen in this area. And I certainly hope that that continues to happen as we move into 2020.

• 44:57Chris Benson?

  I think that's a really great point that you make there. Before we move on to predictions, the last thing I wanted to mention, just about state of where we are right now, is I also think there's a general consensus developing in the industry. We're seeing a lot of top luminaries. I know the VP of AI at Facebook recently said that we are very, very far from human intelligence. And there were, that was in an article, I believe that Wired had, and I think there was another article ironically that Wired had where there were some comments about the fact that with us hitting Kind of some limitations on the types of problems that deep learning is likely to be able to solve.

• 45:37Chris Benson?

  And given the fact that it is a technique that is very narrow in terms of you get highly specialized results in a narrow scope. That one of the things that at NERUP that was talked about was the fact that we really need to get to biological roots of natural intelligence to understand what our next steps are going to be in the AI space. So what I think is that you may end up having people trying to reassess as they enter this new year about where they want to focus their research on and trying to do that. And I guess, any thoughts on that before we move into predictions?

• 46:10Daniel Whitenack?

  No, I think I think it's a great point and I I've definitely seen. I think we'll put some links into the show notes about various luminaries statements on this sort of stuff. I've seen those as well, and I think that we can get into pattern that is is kind of natural, but can be limiting in that, like, for example, we we're all about transformer models and we just do transformer models over and over and over. And it breeds this sort of like NLP is transformers. But actually, you know, there's a lot of things that have happened historically in AI that we could pull from. And there's new things that we could pull from, maybe like you say, that are rooted in other sorts of

• 46:52Daniel Whitenack?

  in other sorts of ideas related to biology or evolutionary algorithms or whatever it is. So I think we need to keep keep our flexibility intact, I think, maybe is a is a good way to put it.

• 47:06Chris Benson?

  I would agree, and I think the industry at large is it would agree with those sentiments based on the the sentiment we saw at NERUP and and that I think has been building over this past year in general. Let's look ahead to 2020.

• 47:17Daniel Whitenack?

  All right, inference time.

• 47:19Chris Benson?

  Inference time now, figure out what we think, what we think might happen. I will start us off with a couple of them and and then and then turn it over to you. I think, you know, as we talked about kind of AI ethics and responsibilities. I think we're now at an inflection point where we've had many organizations putting out their principles on what they think. Should be, but we don't have a very good way to execute on that. So not everybody is going to be an ethicist, especially in the engineering field. And so I think that we're seeing a consensus that the next step now is to turn toward the creation of supporting tools

• 47:58Chris Benson?

  or retrofitting existing tooling that enables non-ethicists to appropriately implement the various aspects of ethical AI, everything from eliminating bias, from data sets, to being able to think about where different types of AI should be applied to different types of solutioning. And so I think we're going to see, I'm predicting that we're going to see a surge over the next year and beyond in tooling to support ethical AI. Any comments on that? I hope so. You hope so? Yeah. I'll be looking for it. That sounds good. I think another thing that I think is, that is happening already, it's starting to, I see a lot of conversation.

• 48:39Chris Benson?

  I've been a part of a lot of conversations about this, is the fact that we're getting to a point where instead of being a separate, you know, neural, deep learning and neural network development, being a separate little shiny object with dedicated people that only do the modeling, you get to the problem of how do you implement this in real life? And you can build a great model, but then people and organizations really struggle to get it deployed into production and getting a kind of a dev ops and feedback loop associated with what they're doing in those activities. And so I think you're going to see a lot of effort into moving neural network development into existing software development lifecycle and workflows that organizations already have in place and that they'll make adjustments to those workflows to accommodate these new technologies.

• 49:24Chris Benson?

  And I think that's really important for them to see a good return on investment about for their efforts in this space.

• 49:31Daniel Whitenack?

  Yeah, I'm thinking of, we've talked about that a little bit, maybe Joel Gruse's episode on on responsible AI development practices would come into play here. I'll link that in the show notes.

• 49:44Chris Benson?

  Sounds good. Another thing I'm seeing, and we actually already talked a little bit about it, you know, I think TensorFlow 2 is an example of this, is we're seeing, I think we're going to continue to see simplification of neural network tooling and trying to make that learning curve more manageable. And I think you'll, you'll see different, different users and developers within this technology being able to buy in to toolsets that are suitable for them and their own backgrounds. So I think that you'll see more tooling that is specific to, you know, that may cater to certain types of data scientists versus certain types of software developers, and you'll be able to kind of customize that tooling to match your level of knowledge,

• 50:28Chris Benson?

  expertise and your background as well, so that you can be productive quicker. And then, I guess my final prediction is that I think that, kind of given what we talked about, this acknowledgement at large that's, that's developing within the deep learning field, that it's not well-suited for certain problems, that it is taking a lot more data to learn than maybe a human might use to learn something, and is less flexible and stuff, and overly focused on a particular solution and not able to move, like by way of example, from one game that you might have a deep learning algorithm that has learned, be able to move to another game and be able to, to leverage what it learned from the first one. So we've seen many examples of that,

• 51:11Chris Benson?

  and I think as, as that message really permeates through the field, that we will see people reassessing, and maybe, and this is really the first time I have said this ever in our podcast, we will start looking at AI in the future as moving into a post-deep learning world. I tend to, when I'm talking about the present, I tend to, to tell people, I think of AI personally as equivalent to deep learning right now, as we are at the beginning of 2020. But I think we also, we may get to the end of this year, and that may not be a true statement anymore, and I may, I may have a different answer. So I think that is where we're going. What about you, Daniel? What are some of yours?

• 51:53Daniel Whitenack?

  Well, I decided to go the safe route and I'm gonna say that uh I'm gonna my prediction is that uh at least one of the following three things are going to be a huge a huge player and a huge emphasis in 2020 and will really pick up steam and maybe all of them or maybe just two of them or one of them, I'm not sure. Um, so I'm kind of covering my bases there that way. Um, you know, my test scores are better when we look at things after a time. But um, uh, the three things that that I was thinking of were first multimodal

• 52:34Daniel Whitenack?

  learning, then mobile AI or AI on mobile devices, and then federated learning. So multimodal learning is where, for example, you you make inferences off of multiple modalities of of input data. Maybe you have an image and text that are input to a model and then you you make some inference. This was, I think, already emphasized recently by our guest from Etsy in their sort of search technology where they have titles for their products and descriptions, but there's also more information in the uploaded pictures of the products,

• 53:13Daniel Whitenack?

  right? And so you could take both of those input signals and do much more than you could with just the text or just the imagery alone, right? And I think that this is gonna be really revolutionary and pick up steam in terms of a lot of things, whether it be chatbots or recommendation like in that in that Etsy case or or whatever it is. I think we're gonna see a lot more of that. In fact, we saw that also with OpenAI's robot-hand Rubik's Cube thing where they were taking signals off of the hand itself, but also using the imagery from cameras and and all of that.

• 53:51Chris Benson?

  I think you're right. I know that at my own company, multimodal learning is is a big deal. Um, one kind of globally impacting use case that we're seeing it is in humanity. 助災救援和災害緩解 這就是你有 你說你正在努力製作數據集 對於一個特定災害情景 可能是一場野火 如果你可以從很多不同的影像 各種電台的呼叫 正在發生 然後你可以製作一個模型 比較堅實和準確 能夠接受更多狀況 所以我想你完全沒錯 multimodal learning 我想那將會很大 將來那是一個好建議

• 54:34Daniel Whitenack?

  好的 希望至少 至少那一個是真的

• 54:39Chris Benson?

  我對你有信心

• 54:40Daniel Whitenack?

  好的 好的 其他的 我想是 真的根據我的感覺 privacy當然 它已經很重要 但正在逐漸重要 還有就是 你知道 AI的規模擴展到全球各地 所以我想我們看到很多 我想我們將在2020年 看到很多手機裝備的 deployment 和更多工具 可能與 像浏览器的 deployment 和那些東西 我們在 running running 模型 在用戶設備上 和精準調整它們在用戶設備上 與之同時 federated learning 我認為

• 55:23Daniel Whitenack?

  就是我們不是真的中心化數據 從各種用戶 然後進行中心化訓練 然後載入模型回頭 但是有這種 federated distributed 變練習 正在發生 很多數據從用戶設備 不需要離開用戶設備 所以有 這個當然有優點 因為 因為 privacy 但也是數據轉送和所有那些 我看到這個在過去的年頭 但沒真正看到它 你知道 真正來到一個 widespread 的方式 和 你知道 可能這是 這是這個年 我不知道

• 56:06Chris Benson?

  我想你沒錯 我想那是 我想那是 你有跟 safe bet 他們都是很好的 bet 但我想他們的 bet 在所有三個中 其實 所以 是的 我想你做到了

• 56:16Daniel Whitenack?

  Well, I probably then have to learn a little bit of, I don't know, I need to learn a little bit of mobile development or something. Maybe, maybe we'll have, have an episode where we have some learning, learning resources on that. But, um, yeah, I've enjoyed this look back and look ahead, Chris. It'll be interesting to look back at this episode at the end of 2020 and see, you know, see what, what, uh, what came true and what didn't.

• 56:45Chris Benson?

  Yeah, so much has changed in the past year, um, as we've, as we've called out, and I suspect we'll have even more so this coming year. So, uh, it was a good conversation. Happy New Year again, and looking forward to, uh, to seeing you in Chattanooga, uh, at Project Voice and, uh, and doing all sorts of cool stuff in the year ahead.

• 57:03Daniel Whitenack?

  Awesome. Happy New Year.

• 57:08Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go on your podcast app and favorite it if you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers. Head to linode.com/changelog, check them out, support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email, keeping you up to date with the news and

• 57:53Speaker 00

  podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.