BERT: one NLP model to rule them all

Fully Connected – a series where Chris and Daniel keep you up to date with everything that’s happening in the AI community. This week we discuss BERT, a new method of pre-training language representations from Google for natural language processing (NLP) tasks. Then we tackle Facebook’s Horizon, the first open source reinforcement learning platform for large-scale products and services. We also address synthetic data, and suggest a few learning resources. Sponsors: Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We catch our errors before our users do because of Rollbar. Resolve errors in minutes, and deploy your code with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogAlgolia https://www.algolia.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our search partner. Algolia’s full suite search APIs enable teams to develop unique search and discovery experiences across all platforms and devices. We’re using Algolia to power our site search here at Changelog.com. Get started for free and learn more at algolia.com https://www.algolia.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: News/Discussion: Is artificial intelligence set to become art’s next medium? https://www.christies.com/features/A-collaboration-between-two-artists-one-human-one-a-machine-9332-1.aspxBERT (a new method for obtaining rich contextual language representations during pre-training): Google research article https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.htmlTensorFlow BERT https://github.com/google-research/bertBERT paper https://arxiv.org/abs/1810.04805PyTorch BERT https://github.com/huggingface/pytorch-pretrained-BERTNY Times article https://www.nytimes.com/2018/11/18/technology/artificial-intelligence-language.htmlExample Colab notebook https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynbBERT explained article https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270Transformer paper https://arxiv.org/pdf/1706.03762.pdfGoogle Open Sources BERT to Train Natural Language Models Without Breaking the Bank https://towardsdatascience.com/google-open-sources-bert-to-train-natural-language-models-without-breaking-the-bank-813ef38018fcHorizon: The first open source reinforcement learning platform for large-scale products and services https://code.fb.com/ml-applications/horizonDoes Synthetic Data Hold The Secret To Artificial Intelligence? https://www.7wdata.be/business-leadership/does-synthetic-data-hold-the-secret-to-artificial-intelligenceAI Experts: Moving forward with AI likely a series of small steps, not giant leaps https://blogs.thomsonreuters.com/answerson/moving-forward-with-ai-likely-a-series-of-small-steps-not-giant-leapsLearning resources: The Backpropagation Algorithm Demystified https://medium.com/@nathaliejeans7/the-backpropagation-algorithm-demystified-41b705229727Books “Grokking Deep Learning” by Andrew Trask https://github.com/iamtrask/Grokking-Deep-LearningUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Bait with for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log.

• 0:20Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 0:46Daniel Whitenack?

  Well, welcome to a fully connected episode of Practical AI where Chris, my co-host and I will keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and then we'll dig into some learning resources to to help you and us level up our machine learning game. So Chris, how's it going?

• 1:11Chris Benson?

  Going great. How you doing, Daniel?

• 1:12Daniel Whitenack?

  Doing well. Getting ready for American Thanksgiving.

• 1:16Chris Benson?

  Yep. It's as we're recording this, it is the day before Thanksgiving for us. And so looking forward to to over stuffing myself tomorrow and then and then worrying about how I'm going to lose the weight thereafter.

• 1:28Daniel Whitenack?

  Yeah, looking forward to some tofurkey for sure.

• 1:30Chris Benson?

  Exactly.

• 1:32Daniel Whitenack?

  What have you been seeing in terms of AI news recently, Chris?

• 1:36Chris Benson?

  Well, actually, the first thing that caught my eye was something that you had had shared with us the last time we were talking about this stuff. And that is, we were, we were talking about artwork in the world of AI. And there was a, a particular piece of art that you discussed, and that was where a generative adversarial network had been used to generate this piece of kind of period art. I'm not an art person, so I'm not gonna do that.

• 2:03Daniel Whitenack?

  Looked like a portrait of a guy.

• 2:05Chris Benson?

  Yeah, it was called Portrait of Edmund Bellamy, if I'm saying that correctly. It's a fictional person from a fictional family. When you were telling us about it last, it hadn't gone to auction yet, but Christie's was going to. And they were expecting it to raise somewhere between $7,000 and $10,000 US. A few days after our episode went live, it actually went to auction and it ended up selling for $432,500 US.

• 2:32Daniel Whitenack?

  Boom. Boom. That's crazy.

• 2:34Chris Benson?

  Crazy. So, I mean, and it really.

• 2:36Daniel Whitenack?

  Wait, wasn't it? Yeah. And so they were saying that it was going to sell for like seven to 10 grand, right?

• 2:42Chris Benson?

  Yeah.

• 2:42Daniel Whitenack?

  Something like that?

• 2:43Chris Benson?

  Yeah, it was like, it was 45 time multiple on what they thought. And, you know, and so we, we had been looking at it with interest, you know, in the AI community, but I think the entire art world got rocked by that one because suddenly, suddenly, yeah, you're a major contender from, from the AI world in terms of, you know, high value art going. So I think it's something that a lot of people that are not traditionally thinking about AI are having to digest and realize that the world is changing.

• 3:11Daniel Whitenack?

  Yeah, I think people are gonna have to start, or are gonna stop going to Chicago Institute of Art and start going to MIT or something to, to go into art.

• 3:21Chris Benson?

  Yeah, it was kind of funny. A few days after that, I host a meetup in Atlanta called the Atlanta Deep Learning Meetup. And I know I've mentioned it before, but we actually had a generative adversarial network tutorial last month, where we had an expert named Riza Katabi come and show us. And it was funny. He came in, he goes, you know, I had this, I had this little project where we got all coded as we go along the way. But I've changed my mind. We're gonna try to build some artwork in this session and we'll split the proceeds if it makes enough money. And so it was, it was just funny because, you know.

• 3:52Daniel Whitenack?

  Sounds, sounds good. Yeah. Yeah, that's great.

• 3:56Chris Benson?

  And there's been some other, some other big news, not necessarily art related. What have you seen lately?

• 4:02Daniel Whitenack?

  Yeah, so I don't know about you. I'm on Twitter. That's where I hear about a lot of things. And it seems like to me, and I don't know if you've seen this thing, same thing, let me know if you have, but it seems like every other AI-related tweet that I'm seeing, at least in the people that I follow, is about natural language processing.

• 4:21Chris Benson?

  Oh, yeah.

• 4:22Daniel Whitenack?

  So like, over the past, I would say, like, I don't know, like, three weeks to a month, it seems like there's just been a steady rise in all things like neural nets and natural language. Have you been seeing the same thing?

• 4:35Chris Benson?

  I sure have. And I think I know where you're going with that, because there was a particular thing announced that I'll let you lead into, that has really caused a lot of interest in the last few weeks.

• 4:46Daniel Whitenack?

  Yeah, you guessed where I'm going. So there's this new model out, so a pre-trained model called BERT from Google. So that's a new approach to pre-trained natural language processing, which we can talk about here in a second. But there's actually been, like, I've seen even yesterday, I think it was yesterday, this HTML model from Hugging Face, which is pretty incredible. Take a look at that if you haven't seen it. And by HTML, it's not meaning the HTML of the web, but like a multitask learning model. And, I'm sorry, now I'm saying, I'm getting confused even with the acronym. So it's HMTL, right?

• 5:31Daniel Whitenack?

  So hierarchical multitask learning, not HTML. So there's a confusion there.

• 5:36Chris Benson?

  You just disappointed an entire world of front-end developers who thought they just now had their way into machine learning.

• 5:42Daniel Whitenack?

  Their way into, yeah. No, so HMTL, so I saw that yesterday. There's also, like, I've seen ELMO, which I think came from the Allen Institute. And also, there was this challenge, so one of the challenges at the now rebranded NERIPs conference, which was a much-needed rebranding. But now, they had a competition that's actually, it's kind of in the schedule for the presentations phase, I think now, around chatbots and dialogue systems. And so it seems like, at least from my perspective, all things with neural nets these days are like with natural language.

• 6:20Chris Benson?

  Yeah, it's kind of funny. We've, we go through these waves, you know, for a long time, everything seemed to be about computer vision and all the different convolutional variants that came out and capsule nets. And you're right, there hasn't been as much in the news lately, but with BERT being released, the NLP world is just on fire right now.

• 6:38Daniel Whitenack?

  Yeah, for sure. So let's, actually, I'd love to, if you're willing to kind of dig in a little bit, I'd love to dig into exactly, you know, what BERT is. I'm still learning about it. So I'll confess, you know, as we get into this conversation, please connect with us on our Slack team and our LinkedIn page of Practical AI. You can go to changelog.com-practicalai and join our Slack team. But I would love to hear if I say anything that's not right. I'm kind of learning about these things as I, as I go. So we'd love to hear your perspective on these things as well. So keep us informed in that way. But my understanding of what

• 7:16Daniel Whitenack?

  BERT is, is the goal is to create this kind of pre-trained NLP model or pre-trained language model. So some of this terminology is new to me, like I mentioned, but in my understanding, what they're trying to do here is create an encoder that will be pre-trained that you could utilize for various natural language tasks. And so, for example, like sentiment analysis, or question answering, or named entity recognition, these are all kind of natural language processing tasks. And so their goal is to create this pre-trained

• 7:54Daniel Whitenack?

  encoder that will essentially kind of be a, act as a language model, or a model that understands the kind of structuring of words and the context of those that can be utilized as the first bit of other models for these other sorts of tasks.

• 8:13Chris Benson?

  Yeah.

• 8:13Daniel Whitenack?

  Is that kind of, Chris, correct me if I'm wrong. Am I, am I on the complete wrong track here?

• 8:18Chris Benson?

  No, no, I think you're right. I think they have some specific terminology they use. I think they call that a transformer. And the transformer is learning contextual relation between words and text. And then it has two separate pieces to it. One is an encoder that's reading the text input, and then one's a decoder that is producing the prediction, you know, for whatever task you're applying it to. So I think the, when you combine the encoder decoder, they're calling that a transformer.

• 8:42Daniel Whitenack?

  Yeah.

• 8:43Chris Benson?

  But you, I think everything you said was accurate.

• 8:45Daniel Whitenack?

  Yeah, and I think what they're saying, because this isn't the transformer model, I guess, has been around for a while. And we'll link to the info about that in our notes. But this, so the BERT model, which again has come out of Google. So it stands for bidirectional encoder representations from transformers. So essentially, this is like, you just mentioned, Chris, this is based on the transformer model. And it's kind of like, like you mentioned, in the transformer model, there's an encoder and a decoder level, because they're trying to do a specific, one or more specific tasks. In this case, they're kind of basing this BERT model on the encoder piece of that transformer.

• 9:27Chris Benson?

  Thanks for the clarification there.

• 9:28Daniel Whitenack?

  Yeah, so it took me a second to get there because it is, it is confusing with the.

• 9:34Chris Benson?

  It's a lot to digest. There was a lot of information that's been pouring out. And I know, I know both of us have been going through some of the different articles and stuff that kind of break it down. So definitely a, a learning task and process for us.

• 9:47Daniel Whitenack?

  Yeah, yeah, for sure. And my understanding is that it's based on this transformer encoder, which is kind of unique amongst encoders. Because when you think about trying to understand the language, like the context of a word in a sentence, you can think about it directionally, like moving forward in the sentence. And based on the kind of forward direction, getting the context of a word. But actually, the transformer in this case is, well, they call it bidirectional. But in my understanding, it's really like non-directional, because it considers all of the text surrounding a word at this, you know, as it's determining the context of a word.

• 10:29Daniel Whitenack?

  So it's not directional in the sense of like going forward through a sentence.

• 10:33Chris Benson?

  Yeah, that's my understanding is, is I've seen it described in different people blogging about it as, as either non-directional, as you put it, I've seen it as directional in either way or bidirectional. I think you have a choice in how you're doing it. And, and the masking of, of the word that you're building the context around is, is pretty key.

• 10:50Daniel Whitenack?

  Yeah, yeah, definitely. You mentioned the task. I think that one of the also the key features here is that it's the bidirectional encoder representation. So they're creating this this kind of context for the for language. But in order to do that, they have to kind of decide about what tasks will help them or what tasks or predictions will help them get the best understanding of language or create the best kind of encoder or language model, like they're calling it. And in this case, they're actually using multiple tasks to do that. They're using one task, like you mentioned, which is kind of like a masking

• 11:31Daniel Whitenack?

  of words, like in a sentence, they'll kind of remove certain words and have them train the model with the encoder to actually kind of fill in those words. The other task that they're doing is next sentence prediction, which is like, given two sentences, can you tell if one of the sentences is actually the next sentence that comes after the other sentence?

• 11:53Chris Benson?

  Yep, I agree. One of the things that it occurred to me, we probably should do is is kind of talk about, you know, what encoding and decoding is. Encoding is where you're actually taking your input and putting it into the sequencing. And by the way, I found this on Quora. For the audience, Daniel and I are Googling this stuff, just like you are. You know, we're all learning as we go. And encoding and decoding is obviously a common task in a lot of neural network architectures. But putting it into sequence and then decoding is where you're actually getting the output that you're going to use on that. Also wanted to note, we've kind of not mentioned that there's really two stages to BERT, and that's important because of there for different purposes. There's there's a pre-training

• 12:34Chris Benson?

  stage, and then there's a fine tuning stage. And the pre-training stage is very expensive. It takes a lot of resource. I think in what I'm looking at here, they talk about it takes four days on a four to 16 cloud TPU system, just to get through, yeah, a lot of

• 12:50Daniel Whitenack?

  That's some crazy stuff.

• 12:51Chris Benson?

  Lot of processing.

• 12:52Daniel Whitenack?

  I have the thing pulled up right now with GCP's cost, and that turns out to be around like, well, With the number I'm seeing around 7K US dollars, so 7,000 US dollars in TPU costs, which of course they're Google and I guess they don't spend that because it's their own cloud. But significant effort.

• 13:13Chris Benson?

  My, my wife would not like it if I did that for a weekend project. I would get in trouble for that, spending that kind of money. But fortunately, Google has put out a whole bunch of pre-trained models, you know, recognizing the expense of that, they have, they have helped us, all those of us who are going to be trying to apply this technology. They have a good starting point and really when you're deploying it into your own application, the fine tuning, which is an inexpensive thing, doesn't require nearly as much processing, is really where you're going to be focusing. So you'll be able to go and find a pre-trained model, hopefully, maybe, or maybe not even need to make tweaks to it. They mentioned that there was very little adjustment needing to be made for different use cases and then do the fine tuning for your own specific, which is inexpensive and something that I probably could do on the weekend without getting in trouble.

• 13:57Daniel Whitenack?

  Nice. Yeah. So correct me if I'm wrong, Chris. So I, I'm trying to think about like, because I actually have potentially a couple use cases that I, I have in the back of my mind for this and correct me if I'm wrong when I'm thinking about like how one would go about this. But in my understanding, a best use case for me to use BERT is if I have some natural language processing task, let's say I'm trying to identify certain entities in text, like named entity recognition. What I could do is take a pre-trained BERT. I don't know if that's the proper way to say that, but that's how I'm going to say it. Pre-trained BERT, sorry to any of you out there that are named BERT. And this is confusing for, but I would take a pre-trained BERT, which Google has spent

• 14:43Daniel Whitenack?

  much, much time training and many update steps and lots and lots of data. And so they've developed this BERT and what BERT's going to do is allow me to put in sequences of words and BERT will then output sequences of vector representations of those words, also give kind of a context within the language model of BERT. And then I could kind of bolt on to that. At encoder layer, some classification task or some other sort of task. In my case, maybe it would be named entity recognition. And because BERT is so good at understanding the context of language,

• 15:27Daniel Whitenack?

  actually, the update for me to actually do one of these tasks like named entity recognition or a question answering or something is, like you said, fairly inexpensive. So I'm utilizing all of the expertise that has been built into Google's model and just adding on the little piece that makes it particular to my use case. And so the first thing that is pre-training, the second thing is fine tuning. Is that right?

• 15:51Chris Benson?

  I think that was a great explanation and that is consistent with my understanding of it. It's, you know, BERT is really, the way I'm reading it is BERT is really to be embedded into a larger architecture toward you get this incredible capability for, maybe not for free, but at low cost relative to having to, to figure out how to do it yourself or use a lesser technology. So from my standpoint, I think this is another great step where Google, in this case, is providing what would otherwise be a very challenging, specific task in a larger architecture, and they're helping us do that almost like a software component in a larger software system.

• 16:27Daniel Whitenack?

  I think that there's kind of two threads that I see running through this that are also kind of, you know, hugely impactful, I think, in the industry in general. One, one of those is transfer learning. What here we're calling maybe the, the fine tuning part where in transfer learning, you're taking something that was trained for a certain task and then updating it or fine tuning it to another type of task. And, and as we've mentioned on the podcast before, I think that's hugely impactful and a huge benefit for actually people that are doing applied AI. The other thing is this multi-task learning framework. I see that this is done in BERT. I also see it being done, like I mentioned, in the HMTL model and other cases

• 17:11Daniel Whitenack?

  where this encoder layer is being trained based on being able to do multiple tasks, not just, not just one task. And I would highly Recommend looking at that HMTL model as well. This is pretty impressive in that respect.

• 17:25Chris Benson?

  Sounds good. Anything else?

• 17:28Daniel Whitenack?

  Well, I was just going to mention, as you can tell, I've kind of been sucked down the rabbit hole of BERT. But I did want to mention to people, again, this is open source. You can read the article from Google, but also you can go to their GitHub and they have the pre-trained models that you can go ahead and use. But there's also actually already been an implementation in PyTorch by Hugging Face. And it's not maintained by the Google team, but by someone else. And I just thought it was pretty cool and useful to already see that implementation in PyTorch so soon after seeing the stuff come out of Google. So keep that in mind, whether you're working on PyTorch or TensorFlow, not that those are the only two, but I think that covers a lot of people. You'll be able to utilize this tech, so.

• 18:11Chris Benson?

  That's true. And I think we end up talking about TensorFlow and PyTorch right now because there's so much coming out in terms of advancements being made where people are really centering around those two platforms. But as you said, there are tons of great tools out there. We're not trying to exclude anyone on those and we would love to hear back. If we are not talking about your favorite tool as much as you'd like to hear, join us in the Slack community and tell us what you're doing in it because we really go out and see what people are writing and talking about, and then we end up talking about that on the show. So we definitely would love feedback and whether it's that or other areas, steer us in directions you want to hear from.

• 18:48Daniel Whitenack?

  Yeah, definitely.

• 18:50Chris Benson?

  So I noticed that there was another release, and this time it was from Facebook. They open sourced their applied reinforcement learning platform. It's called Horizon. And with that, I noticed that it's pretty cool. I think if you're not familiar with reinforcement learning, that is an aspect of machine learning where you are using a software agent to take actions that are in the environment that you're operating in. So if you have a model that you're developing and actions are being taken through those, you are trying to reward when things are going the right way and you're trying to learn. So as your model is converging in the right Right direction, you reward it and you don't reward it when it doesn't.

• 19:35Chris Benson?

  And you see that in a lot of different applications, everything from different AI learning on how to play games. You see it a lot in robotics. And so it's really great to see Facebook open sourcing how they're approaching that because they're doing a lot of work on this. Had you seen that one, Daniel?

• 19:52Daniel Whitenack?

  Yeah, this is definitely interesting to me. And I'll note as well, in a previous episode, so episode 14, Wochek Zaremba talked with us for a whole episode about reinforcement learning. It's an area that I definitely want to get up to speed on. So I did run across this. It was also one of the things that, you know, kind of crossed my path multiple times over the past couple of weeks. One of the interesting things that I thought was interesting about this framework that they open source, or really it's more of a platform, right? So this reinforcement learning platform is that it's not kind of, it's not just like a specific library for PyTorch or something.

• 20:31Daniel Whitenack?

  It is actually like a platform that utilizes multiple open source projects to do, help you do the task of reinforcement learning. So I see that Spark is involved here along with PyTorch, along with SciPy, along with OpenAI Gem and the Onyx framework, which I'm a big fan of and excited about. So you've got the kind of large scale data processing element that's kind of coming from Spark. You've got the scientific computing and numerical machine learning pieces coming from SciPy and PyTorch. And then there's other things as well, including model serialization and interoperability that's coming with Onyx.

• 21:11Daniel Whitenack?

  So it was really cool to see that this kind of convergence of multiple different projects to enable this, you know, what seems like a really great platform for actually enabling reinforcement learning in production.

• 21:25Chris Benson?

  Yeah, I noticed, I was looking across their GitHub page, thinking of it as a platform other than just a library for another platform. You build it in Python using PyTorch for the modeling and the training, and then you can serve models with Caffe 2. So it does have dependencies with other platforms, specifically PyTorch and Caffe 2, but it's a whole system unto itself.

• 21:49Daniel Whitenack?

  Yeah, yeah. And I don't know, this was actually pretty surprising to me. And maybe this is just my lack of following a lot of reinforcement learning things. But it was kind of a shock to me for them to describe how they are using, how Facebook is using reinforcement learning in production. So they mentioned on Messenger, on 360 video and more. And that was a shock to me. If someone was to ask me before I read this article, you know, who was using reinforcement learning in production? I would probably just kind of give them a blurb about how it's mostly a research thing right now. And OpenAI and DeepMind and other people are using it for robots and other things.

• 22:32Daniel Whitenack?

  But it's not really, it hasn't really filtered into production usage and industry. And clearly I'm wrong about that because they're using this, you know, practical platform for reinforcement learning in production, at least on, at least on a few things. They say, you know, Horizon has allowed us to improve the image quality of 360 degree video, optimizing bitrate parameters in real time and other things. So this is actually like, you know, real usage of reinforcement learning, rather than just kind of like funny videos of robot arms and stuff.

• 23:08Chris Benson?

  Yeah.

• 23:08Daniel Whitenack?

  This was a pretty big shock to me.

• 23:10Chris Benson?

  I have seen it used in industry, but it was strictly in robotics. When I was with a previous employer and we had several teams doing some fairly advanced robotics tasks. My team was not, we were very much focused on the computer vision side of things with Maskar CNN and other convolutional technologies. But yeah, I know another team that we were working with was doing reinforcement learning and deep reinforcement learning, where you're combining reinforcement learning with, with a deep architecture as well to do that on the robotics side. And that's used a lot on kind of strategy for robotics movement and things. So, but it had been, my own personal experience had been very specific to that use case.

• 23:51Daniel Whitenack?

  Yeah, and I mean, even so, I'm looking at their GitHub page for Horizon right now, and it says a platform for applied reinforcement learning, or Applied RL. And of course, that fits right in with what we're passionate about on this show, which is practicality. And this has definitely changed my perception of reinforcement learning outside of kind of the domain of robots, like you were talking about, which I have never worked in robots. And so to me, reinforcement learning like, didn't really come across as something that maybe I would apply directly, at least in the near future. But maybe I need to reevaluate my perceptions there. And actually, I'd love to just kind of go through and see, I haven't been through

• 24:35Daniel Whitenack?

  the, all of the docs of Horizon, but it looks like that you can install it with Docker. So it would be fun to just kind of spin up Horizon and say, at least say I've done some reinforcement learning. I feel like I could, you know, check that box off of my bucket list, at least.

• 24:50Chris Benson?

  Absolutely. And I want to try to find a use case for both BERT and Horizon, from a learning standpoint, to dive into them. Because, you know, it's kind of funny, as we, as we talk about these different things in the, that are, that are happening in the AI community on these fully connected episodes, it is, you have to really pick and choose what you want to do. But we're seeing so much advancement right now in these areas. So I'm trying to find ways of, since you don't get to do everything in whatever job you're doing in the AI world, I'm trying to find small projects where we can apply those. So if anyone has ideas, I hope you'll share them in the Slack community or on LinkedIn, in the LinkedIn group, because that would be very welcome. Things that are scaled, that are affordable for people to dive in and have fun with.

• 25:34Daniel Whitenack?

  Yeah, and we'll also, just so you guys know, we always try to include a bunch of links to what we're talking about in our show notes. So there's actually, I have a list here, right now, of all of the things about BERT. Like I said, there's been a lot. There's been a Google article, TensorFlow, GitHub, there's been a paper on the archive, the PyTorch repo, New York Times article, there's even like a Colab notebook, it's like Jupiter notebook, but kind of Google Docs style. So there's one of those for you to try it out. Of course, like I mentioned, Horizon has the Docker install and all of that. So barrier to spinning up a lot of this stuff is a lot lower than it used to be, which like you mentioned, Chris, in some ways it's,

• 26:18Daniel Whitenack?

  I mean, in a lot of ways, it's it's super exciting, but in other ways, it's like there's too much to try. So there is, I probably need to focus my attention a little bit. But yeah, so I think that was pretty much the what we had to say about Horizon. I'm excited to dig in more. Have you seen anything else in the in the news recently, Chris, that you want to highlight?

• 26:39Chris Benson?

  Yeah, I ran across a blog article that's called, Does Synthetic Data Hold the Secret to Artificial Intelligence? And it caught my eye, kind of dives into just in general about synthetic data and how it's used and in terms of generating enough data to operate on. The reason it really caught my eye is I had some personal experience from my own work, having to do with synthetic data. And I also was interviewed a short while back by Thomson Reuters on a series of AI articles that they were posting on that. And I've tweeted, if anyone has an interest, I've tweeted about it and stuff, and you can find the article. But really talking about using synthetic data going forward to generate larger data sets, how it fits into

• 27:22Chris Benson?

  unsupervised learning for the future. And in my own experience, I found a lot of people tend to say, yeah, we'll just synthesize the data, you know, and there's a variety of techniques for that. We found it very hard to do that. And I'm hoping that on our on my own learning curve, that me and the people on the teams that I've worked with can can figure out better. But that can be really challenging. So the article caught my eye because of the the hope forward. And as well as everybody does, I would love to be able to say, if I want to hit a particular use case and don't have sufficient data, we can go synthesize the data and train it. When we were doing that manually, in terms of trying to generate through a number of automated things at a company I used to work at, we found that the the data set, we had a small data set

• 28:07Chris Benson?

  that represented the real life problem that we were tackling. And I'm not allowed to disclose what that was, but we also, we didn't have nearly enough to address it. And so we went and tried to synthesize it through a bunch of different techniques and we found that the, we really had a struggle with getting enough diversity into the data. We could generate the volume, but it was very hard to synthesize the diversity that we needed to where our goal had been, if you take a synthetic data set and compare it side by side with the real, much smaller data set that we already had, that it would be indistinguishable or close to that. So I would love to hear back from listeners and I would love to hear, Daniel, if you have any thoughts on that about how people are approaching synthetic data and some

• 28:52Chris Benson?

  of the different techniques and successes or failures that they've had.

• 28:55Daniel Whitenack?

  Yeah, maybe just to kind of pause a little bit, because I actually, I don't have a lot of experience, this whole idea of synthetic data. But you know, what I'm thinking when I hear you talk about this is like, you know, hey, Chris, like, what exactly do you mean by synthetic data? Because isn't data just data? I think you kind of got into that, but maybe you could describe, like, maybe a little bit more about why there is a need for synthetic data.

• 29:19Chris Benson?

  That's a great point. So I'm kind of referencing in my brain, my own project, but because of non-disclosure issues, I can't address it directly. So I will try. It is oftentimes the case in an industry, in the real world, when you're trying to tackle a business problem, in the case that we were in, it was to enhance an existing product. And you will say, okay, this is what we need to go solve that problem for training purposes. And, but when you look at the amount of data that you have, you realize that you might need hundreds of thousands of records or millions of records to train against. And you might have, you know, 2,000 or less, maybe a few hundred. And that's, and that might not be nearly enough to get a high quality model

• 30:03Chris Benson?

  trained for your purposes. So one of the things that people will do is say, are there ways that we can generate our own sense of reality that looks very much like the real thing? So you're generating more data that looks a whole lot like the data that you already have, but you need more volume. And there's a number of different. Software packages that can help you do that. And we tried some different techniques in this project that we were working on. The challenge that we had there was was simply having enough variability, enough diversity in the synthesized data. So that if you were to hold those two data sets up, the synthesized versus the real life one, the real life one was messy. It had all of the little tweaks and diversity.

• 30:44Chris Benson?

  As you change things in real life and you get noisy, messy data to train off of that represents the real world that you're trying to to get a model to represent. That's what it is. It's very hard to do, at least in the stuff that we had done. It was very hard to generate synthesized data that didn't look synthesized, that had so much diversity that you would never realize it was generated. The number of different options for various inputs, that kind of thing. As I go forward, and I I'm sure this will come up in the not too distant future, where we have to take a synthetic data approach. I'm looking forward to having other people out there say, hey, this is what worked for me or what didn't work for me.

• 31:18Daniel Whitenack?

  Yeah, I was just looking as you were talking at, you know, models like we're talking about here, which I'm assuming like the models that you were talking about in your use case, but other cases like robotics or natural language, like the BERT model says it has, you know, like, you know, hundreds of millions of parameters, right? So to train that many parameters, to fit that many parameters, takes an enormous amount of data usually, which sometimes you just don't have, don't have access to. I'm glad you brought up this point. It's something that I definitely feel like I need to learn a little bit more about. And I would be interested to hear from any of our listeners if they have good resources or pointers on that front.

• 32:00Chris Benson?

  Yeah, I would note the use case that we were generating from, I'll say it was not a convolutional. I've also done it on the convolutional side with more success, because you can take the images that you're using in your convolutional neural network and make adjustments, you can change angles, change sizing.

• 32:14Daniel Whitenack?

  Or like flip them or some sort of stuff.

• 32:17Chris Benson?

  Yeah, there's a lot of image manipulation things you can do to generate more data there. So had great success there. Unfortunately, the use case that I was kind of describing around was not that. So I just wanted to distinguish between the two. I think it's I think it's easier with certain types of architectures than other.

• 32:32Daniel Whitenack?

  Yeah, yeah. Well, on that note, noting that we all have a lot to learn about multiple things, let's go ahead and move into the part of Fully Connected where we highlight a couple of learning resources that have been useful for us or look interesting. The first one that I'm going to point out, which is something that I want to look into a little bit more and maybe order the physical copy of, is this is a new or almost, I don't know if it's actually out yet, but it's called Grokking Deep Learning. And there's a physical and ebook from Manning called Grokking Deep Learning. But one of the things that I was looking at was that there's also a companion GitHub

• 33:15Daniel Whitenack?

  repository, which itself is kind of helpful, even maybe even without the book, because it goes through like from the beginning, kind of from scratch, how do we, how do we kind of understand and dig into deep learning? So it goes through, you know, forward propagation and introduction to neural networks, gradient descent, generalizing that back propagation, regularization, activation functions, and really kind of starts to pick apart like convolutional layers and word embeddings and other things, more from a scratch perspective and trying to get into those things. So I think that this would be a great thing to go through if you're wanting to really kind of understand deep learning and neural networks at a more granular level.

• 34:00Chris Benson?

  I have the book and have read it, and it is very good compared to a lot of books where they don't give you a sufficient understanding. The Grokking part of the title, I think, is accurate in that they, is that the author really tries to explain those. And so having, having the examples in the GitHub, which I had not looked at actually, is really nice to have in the book. So I know I've read that and enjoyed reading it and thought was one of the better explanations out there, so definitely concur with that.

• 34:27Daniel Whitenack?

  Well, I'm glad that I wasn't making wrong assumptions there.

• 34:32Chris Benson?

  I have one that's very specific. I've done other, I've talked about this in different articles, but there on Medium, I found a Medium post, I probably am going to butcher the name, it's Natalie Jeans, J E A N S, on Medium, it looks like it's her only article that I see here, but it's "The Backpropagation Algorithm Demystified", and it's another really good explanation. A lot of people, as we're getting into the field, you know, this is one of the very first things you learn, and actually, if you haven't been exposed to backpropagation, it can take a while to really understand it and get it. And so, I thought this was one of those articles, that if you're a newbie into the field, and you're trying to understand just how feedforward with backpropagation works,

• 35:15Chris Benson?

  this is another good place to start. She takes you kind of through the initial concepts about, you know, the inputs to a node, and what it means to have an activation function, and what those are, and kind of describes backpropagation at high level, and then she goes into gradient descent, and that's a group of different related algorithms, gradient descent, that allow you to minimize your error, and she has some good visuals and some great explanation on that. She talks about what those different variants are, and then kind of takes you through some examples using Sigmoid, which is not often used in real life these days, but is a good training tool that people will use.

• 35:56Chris Benson?

  And then actually goes to what people do use in real life, which is backpropagating rectified linear units, or you might hear it as ReLU. And so, did a good job of kind of giving you a good stab at understanding that is. So I hope people will go see it, we'll put the link in the show notes. And that's it for me this week. You have anything else, Daniel?

• 36:15Daniel Whitenack?

  Nope, that's it. I think those are great, and I think it's great that you brought up today, how, you know, we, just like everyone else, even though they don't always admit it, are always searching through Quora, always searching through Stack Overflow, and GitHub, and papers, and all of that. If you run across any good ones that we haven't highlighted, let us know on our Slack team, and yeah, it was a great discussion today. Thanks for being patient with me. Chris and helping me dig through some of these things.

• 36:45Chris Benson?

  Yeah, I had a good time. This was a slightly different type of show than anything we've done in terms of our, just you and me digging in ourselves and digging in not as experts, but as many of our listeners just trying to take it. So I hope it made sense to our listeners. And if we get good feedback, I'm looking forward to talking about specific technologies some more in the future.

• 37:05Daniel Whitenack?

  Awesome.

• 37:05Chris Benson?

  Okay. Well, I hope everybody has a great week and we will talk to you sometime soon. Talk to you later, Daniel.

• 37:11Daniel Whitenack?

  All right. See you, Chris.

• 37:12Chris Benson?

  Okay. Bye bye.

• 37:16Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Leno cloud servers. Head to leno.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com.

• 37:58Speaker 00

  When you go there, pop in your email address. Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 38:14Speaker 03

  I'm Tim Smith and my show Away From Keyboard explores the human side of creative work. You'll hear stories, sometimes deeply personal, about the triumphs and struggles of doing what you love.

• 38:25Speaker 04

  I ended up in hospital with burnout. I just kept ignoring the way that it was making me feel and just kept powering through it. And then eventually my body started to give me physical symptoms to say like, hey, you should stop and listen to me.

• 38:38Speaker 03

  New episodes premiere every other Wednesday. Find the show at changelog.com/afk or wherever you listen to podcasts.