Exploring deep reinforcement learning

In addition to being a Developer Advocate at Hugging Face, Thomas Simonini is building next-gen AI in games that can talk and have smart interactions with the player using Deep Reinforcement Learning (DRL) and Natural Language Processing (NLP). He also created a Deep Reinforcement Learning course that takes a DRL beginner to from zero to hero. Natalie and Chris explore what’s involved, and what the implications are, with a focus on the development path of the new AI data scientist. Sponsors: Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Featuring: Thomas Simonini – Website https://www.simoninithomas.com, GitHub https://github.com/simoninithomas, LinkedIn https://www.linkedin.com/in/simoninithomas, X https://x.com/ThomasSimoniniChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonNatalie Pistunovich – GitHub https://github.com/Pisush, X https://x.com/nataliepisShow Notes: Thomas Simonini | Hugging Face https://huggingface.co/ThomasSimoniniDeep Reinforcement Learning course https://simoninithomas.github.io/deep-rl-courseUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  For me, AI will change everything, but it's really important to consider that it will not solve everything. For instance, you know, we all talk about the question that AI will help a lot in healthcare, as I mentioned, but the problem is that if people can't afford, you know, to access these tools, the problem is not solved. So I think what's important to consider is that we need to improve AI, but AI must be, you know, really accessible to everyone to be a solution to human problems.

• 0:30Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/Changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 0:47Speaker 01

  Hey, Jared here. One of the things we can count on in the software industry is change. The state of the art changes so fast, in fact, that keeping up can feel like a whole other job on top of your actual job. That's why we created Changelog Weekly. It's our totally free newsletter that we drop in your inbox each and every Sunday. We link to the latest news, the best articles, and the most interesting projects that you should be aware of. We also add a little commentary from us saying why something's important, pointing you to other instances of a trend, or just making a dorky joke to keep it lively. So if you haven't yet, I recommend subscribing to Changelog Weekly and help us help you keep up with the latest.

• 1:30Speaker 01

  Head to changelog.com/weekly and sign up today. Again, it's totally free and we never spam you. Yuck. One last time, that's changelog.com/weekly.

• 1:53Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI.

• 2:19Chris Benson?

  Welcome to another episode of the Practical AI Podcast. We are the podcast that tries to make artificial intelligence practical, productive, and accessible to everyone. My name is Chris Benson. I am one of your co-hosts and Daniel is actually out this week. He has told me I'm allowed to tell you he's down with COVID and so we are hoping Daniel feels better much very fast. But for those of you who joined us last week, I have a treat for you. I have an excellent co-host, a proven co-host. You already know her. Her name is Natalie Pistinovich. Sorry Natalie. And thank you for agreeing to come back in while Daniel is struggling through COVID and co-hosting with me. Welcome back.

• 3:01Speaker 03

  Thanks, Chris. It's always fun to be here.

• 3:03Chris Benson?

  Fantastic. And we are to dive right into the show here. We have a guest who is from one of the leading organizations in the AI world. It's one that we always love to talk about, one of our favorite organizations. And this particular guest has been doing some pretty cool stuff, I know from the pre-show conversation. So I would like to introduce Thomas Simonini, if I got the last name correct, Thomas?

• 3:30Speaker 00

  That's correct.

• 3:31Chris Benson?

  Welcome. You are a developer advocate at Hugging Face, which is the organization I was talking about, working on deep reinforcement learning. And I know you're also the founder of a deep reinforcement learning course. But I guess if you, other than that little tiny tidbit that I've offered the audience already, if you would tell us a little bit about your background, how you arrived at where you're at doing deep reinforcement learning and how you got there. And a lot of people I know ask us out of the episodes, kind of, how did you get there? Because they want to learn from you. So if you can kind of share people, share with people how you got to where you're at?

• 4:05Speaker 00

  Sure. So first thing first, thanks for inviting me. It's a vast question. Initially, I have a bachelor degree of law and political sciences. And what's happened is that I was working in a startup incubator and I was helping people who want to make startup to choose, you know, the technology. Behind their website because I was a full stack web developer in addition to my studies. And what happened is that I met a lot of people who were working on machine learning. And I didn't know about that and I discovered this domain. And after my bachelor degree, I quit my studies. And I start to self study deep learning initially.

• 4:46Speaker 00

  And in two thousand and eighteen, I discovered deep reinforcement learning and I fall in love with this domain. And at the same time, I start to write articles about deep reinforcement learning. And it became a course. It was not supposed, it was just one article, but it works really great. And it becomes a free and open source course. And after that, so in two thousand and nineteen, I start to work at Dataiku for one year and a half. Where I work on implementing deep reinforcement learning agent for real world application. Dynamic pricing, for instance. Dynamic pricing is when you need to find a price. For instance, if you have an airline ticket. And the goal is that you find the best price to make the most benefits.

• 5:31Speaker 00

  And after that, I quit to work in the video game industry. And finally, I'm working at Hugging Face to work on implementing deep reinforcement learning in the ecosystem of Hugging Face.

• 5:43Chris Benson?

  So I'm wondering, just because you mentioned it several times at the front, and I know I mentioned it at the beginning as well. There are a few people out there that may have heard of deep reinforcement learning, but they don't have the background, they haven't had a chance to dive into it themselves. Could you give us a quick, kind of, you know, tell us a little bit about what it is and what's involved in it that, you know, especially if to differentiate it from some of the other forms of deep learning?

• 6:05Speaker 00

  So to say it simply, deep reinforcement learning is to learn from action. So the idea is that you have an agent that will learn by interacting with his environment. If we take a simple example, you can make an agent that will learn to play Super Mario Bros. So to play a video games. And how he does that, he just, you know, tries some action in the environment, so going left or right. And based on the rewards he gets, so for instance, if he die, it's negative. It will improve its behavior to win the game.

• 6:37Speaker 03

  And Hugging Face, where you're the developer advocate now. So this is an AI community, right? And it's all open source. And so what is your mission? What, how do you do things? Why?

• 6:51Speaker 00

  So in Hugging Face, what we want to do is to democratize AI and what we call good AI through, as you mentioned, open source models and open source library. The idea, if you want, it's the GitHub for the AI models. It means it's a place where you can access a lot of trained models, very powerful, and you can use it for your own problems you have. And you know, for instance, you can use a model to translate from English to French, et cetera. And our goal is really to democratize the access of AI by providing a place where you can share or upload models. But it's, you know, it's much than that. You know, there is the transformer library

• 7:34Speaker 00

  that allows you, you know, to create transformers. So transformer is models, natural language processing models. You have dataset library to create rapidly dataset or to share dataset and so much more. So I think, yeah.

• 7:48Chris Benson?

  So I have a follow up question to that. And that is in my day job, because believe it or not, running the podcast isn't the primary thing that I do. This is fun. This is passion. But in my day job, I was talking to my boss about an hour ago. And I told him I was interviewing you and you know, that you're with Hugging Face as developer advocate. And he, not for the first time either, said, I just love Hugging Face. He said, that's the library I want to use every time I'm doing NLP. And he went on for several minutes about you. And so about the company and stuff. So, but it begs the question, for those who aren't intimately familiar and haven't used it, what is it in your opinion, as a developer advocate, about Hugging Face that you're doing right? So many people really love using the libraries out of Hugging Face.

• 8:32Chris Benson?

  And it's kind of a darling in the industry. What is it that Hugging Face is doing right? That maybe others, I'm asking you to give away the secret sauce, I guess, that others would do well to emulate?

• 8:42Speaker 00

  Well, I think I will take my experience because I discovered Hugging Face, I think it was eight months. I think it's really fact that you can rapidly, even though you're not a specialist in NLP, you can rapidly grab a model and use it for your own needs. You know, in two lines of code, we have something called pipeline. It's the first thing I discovered is in two lines, you can, for instance, translate, you can having a model that generates conversation, etc. For me, it's the secret sauce is really the place where you have so much trained model and so much no powerful model that you can use in two lines of code. And I will also add that it's also community because we are very, a very, very strong community.

• 9:26Speaker 00

  It means that when you have question, you know, people are really willing to reply, even people outside of Hugging Face, you know, on the forum, on the Discord. And it really helped me when I started, you know, to train my first model, to have people who are very involved and that do the network, you know, at Hugging Face to help us.

• 9:44Speaker 03

  And would you say that the community of users of Hugging Face, is it more developers? Is it more people who are just general enthusiastic about AI or more data scientists? Who do you see active there?

• 9:57Speaker 00

  I think it's both of them because we have a lot of researcher, but we have also a lot of enthusiasts. And for instance, there is a lot of people, you know, who worked on computer vision models, on natural language processing models. And there is a lot of people who just started, you know, in deep learning and or in deep reinforcement learning, would directly, you know, try models. So it's very diverse, you know, but I think the majority of people who use Hugging Face are researcher and, you know, data scientist. But there is also a growing part of really, you know, beginners who just want to learn and to try some stuff.

• 10:36Chris Benson?

  So one of the things that, you know, and I know that Hugging Face covers a lot of different kind of approaches in the libraries. We're hearing the last year or two, especially a lot about kind of the rise of deep reinforcement learning. It's been around for a while, obviously, but it's really come into its own. We're seeing it in more and more places. And what is it about that particular approach compared to some of the other approaches that it has replaced over time? That is helping that become such a kind of one of the, you know, top tier, first things you want to go to approaches these days? What's caused the rise of deep reinforcement learning?

• 11:15Speaker 00

  Well, I think it's because deep reinforcement learning is able to learn, we call that a policy, so a behavior, it means choosing an action given a state, is able to learn, you know, hidden behavior that you couldn't, you know, train with classical models. So I think is the first thing. And the second thing is also because deep reinforcement learning is becoming more and more efficient. You know, it was a big problem for a long time in deep reinforcement learning. It takes a lot and a lot of time to train compared to, you know, other models. And we are seeing that nowadays we have new models that are more and more powerful and require less and less data to train.

• 11:58Chris Benson?

  So one of the things I was also wondering about is, are there limitations to deep reinforcement learning that you're seeing? You know, what are the limits? We've seen some pretty amazing capabilities from all of these different deep learning capabilities over the last few years, these different algorithms. But what are the limitations? I know that I've heard people make some kind of crazy suggestions in various areas and you're kind of like, we're not there yet, you know, or something. What are some of the limitations that you might caution people and say, maybe not, or maybe you're not looking at the right solution, or maybe we're just not there yet? Any thoughts around that?

• 12:33Speaker 00

  Yeah, there is, you know, for people who are doing deep reinforcement learning, they all know, you know, this article by Yerpan Alex called, "Deep Reinforcement Learning Does Not Work Yet." I think it was published in 2018. Nobody wants to talk about these articles, but it really presented every limitation of deep reinforcement learning. Limitations are, as I mentioned, sampling inefficient, but it's really changing. So sampling inefficient means that you need a lot and a lot and a lot of, you know, experience from your agent before, you know, it's becoming good. And the second one is the most important is the problem of transfer learning. What we simply means is that we are not able for now, when you train an agent, to transfer to another agent.

• 13:17Speaker 00

  So, and the problem with that is also the problem of generalization. For instance, last week I tried to train an agent to play So it's a famous game. And when I tried to change the background, the background color of the game, the agents become very bad. So the the big problem is really the transfer learning and generalization error. What we simply means is that for now, we can consider that a lot of deep reinforcement learning agents just overfit their environments. So they can't generalize, which is why I think it's a big problem in the deep reinforcement learning community.

• 14:13Speaker 01

  The change log is deep discussions in and around the world of software, and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

• 14:22Speaker 04

  At the time, drones were like predators and global hawks and military industrial. They were classified and super, you know, $10 billion things. And we had just built a drone with Lego pieces around the dining room table programmed by a nine year old. And it's like, OK, that should not be possible. You know, it's when when a nine year old can do something that is classified, that literally export controlled as ammunition with Lego with toy pieces. It was something important in this world has changed.

• 14:52Speaker 01

  Leaders like Devin Zougal from GitHub.

• 14:54Speaker 05

  In the like 10 to 15 year range or 20 year range, what I would really like is for if you have like three 12 year olds hanging out and one of them is like, I want to be a firefighter. Another one is like, I want to be a lawyer. I want one of them to say that I want to be an open source developer.

• 15:09Speaker 01

  And innovators like Amel Hussein.

• 15:11Speaker 06

  I've yet to kind of see applications at scale that don't use multiple languages, that don't you have just arcane stories behind why this weirdo thing exists. You know, like, all right, when you open this file, you're going to have to turn around three times and tap your nose once.

• 15:28Speaker 06

  Like it's just the most hilarious stories, you know, but applications are living, breathing, they have craft, that's normal. So I want to normalize weirdness because that's just how applications evolve over time.

• 15:44Speaker 01

  Welcome to the change log. Please listen to an episode from our catalog that interests you and subscribe today. We'd love to have you with us.

• 16:06Speaker 03

  So the deep reinforcement learning course that you've been developing, how long have you been working on that?

• 16:14Speaker 00

  It started in 2018 and the last update was during the first lockdown. So it was in 2020 and initially it was just supposed to be two articles and it become 10 articles and eight videos. I stopped updating it because I just updated the first articles because they were the oldest. But now there is so much more, you know, educational resources that theory is still good, but the implementation, you know, are not relevant. You can still use it, you know, there are still people who use it, but I really advise people to look at the theory and use over implementation. There is, for instance, Costa Young, who made very good implementation with what we call the clean RL

• 16:58Speaker 00

  library that you can use if you want to learn to implement from scratch, which is always a good idea to learn to implement from scratch, even though there is very good libraries. The best way to understand each architecture in deep reinforcement learning is really to implement by yourself.

• 17:14Chris Benson?

  Gotcha. So would the target audience be someone, it sounds like someone who probably already is comfortable writing some code and probably has a little bit of familiarity with some of the other architectures out there? Or do you think it's a good one for someone who's just getting into it? How would you position somebody that you're recommending for your course?

• 17:33Speaker 00

  The course is really from beginner to expert. You just need to know to program. In Python, but even though you don't know about PyTorch or TensorFlow, you can learn with the course. And I think that's why the course works great is that you don't need to be a mathematician to follow this course because in each chapter, what we do is that we really explain each part of the formulas, you know, because I think the big problem in deep reinforcement learning and in AI in general is that most of the people who start, you know, if they don't have a big, strong mathematical background, they can be very scared, you know, when they read the paper and there is big formulas. So the idea with this course is really to be very technical, but to go it step by step,

• 18:17Speaker 00

  not just to start, you know, with mathematics first.

• 18:20Chris Benson?

  Do you think that deep reinforcement learning is a good first step into the deep learning world? It came along after, you know, like Natalie and I had our first exposures and stuff to the larger frame framework of deep learning. But if somebody was just coming in today, maybe they're in university and they're studying. Is this a good place to start? Or would you say, no, no, go back to convolutional neural network or go back to other NLP things historically, start with transformers? How would you position deep reinforcement learning compared to that array of of possible starting points?

• 18:55Speaker 00

  Well, I think it's better first to start with deep learning because we in deep reinforcement learning, we use, you know, neural network and also to work a little bit on convolutional neural network before starting deep reinforcement learning because it's skills that you need to have before being able to to start to learn deep reinforcement learning.

• 19:14Speaker 03

  And you mentioned in your about you that you're building the next generation AI in games. So tell us a little bit about that. That sounds really interesting.

• 19:23Speaker 00

  Yeah, so before joining Hugging Face, I really wanted to work, you know, in the video game industry. And what I started to do was initially to train reinforcement learning model for small games, casual games. But what I discovered is that in this part, you know, game studio use what we call behavior tree, which is much easier, you know, it's AI made by hand where the developer define Each action to take for an agent. And then after that, I started to see some demos about people who have conversation with AI, especially using OpenAI GPT three. And I was fascinated, you know, with this part because I was like, you know, there is Anna Kipnis, which works a lot on the video game industry that explained that

• 20:08Speaker 00

  in video games, you had a lot of improvement in terms of graphism. But in terms of interaction, we still, you know, at the dawn of age. So the idea was how we can use natural language processing models to create better interaction with what we call non-playable characters. So non-playable characters are in the games, you know, the AI with you have conversation with. And so I start to make some demo on that. We made a demo where you directly speak with NPC and the reply using OpenAI GPT three and an AI called Replica, which is an AI that generates voices. And at Hugging Face, we are currently working on a project. We don't have the title yet.

• 20:52Speaker 00

  It's called Murder on the Lighthouse, where you need to, there is a murderer and you have three characters and you need to investigate who's the killer by, you know, speak with them and find contradiction in their testimony.

• 21:04Chris Benson?

  So that sounds good. So now that we're like into gaming, I gotta ask, as you talk about it, I want you to expand just a little bit on it. Just, I'm taking a slight detour here, but because I can. This is one of those moments, I will say, as a parent, when I have a nine-year-old daughter interested in this and she's gonna wanna hear this part. And so I'm kinda taking a detour to ask on behalf of my daughter. How should, as you're doing, you're using deep reinforcement learning in gaming, how do you kind of think about doing that? So there are a ton of kids, I know, because I'm listening to their calls. You know, we're in the pandemic, they're all doing online calls. So I will hear a whole bunch of young ladies that are deeply interested in gaming, talking about it.

• 21:47Chris Benson?

  How can they think about what they need to be doing going forward to get into things like deep reinforcement learning? They wanna do it in the context of gaming. Do you have some advice, since I have hijacked the conversation, so that those young ladies can start thinking about how they do this thing? That that's what they really wanna do? You're doing the thing that they're aspiring to.

• 22:07Speaker 00

  Well, if they want to use deep reinforcement learning in games, I think so the first advice is to start to learn deep reinforcement learning. But there is also a lot of library that can use, and I think the best is to use Unity. They have something called ML Agents that allows you, you know, to create your game and then to train your agent on the game you created. For now, in video games industry, what can be done with deep reinforcement learning is mostly testing the game. So it means, you know, trying, having an agent that will try every part of the environment. But I'm pretty sure that in the future, we'll have AI in games that will be made with deep reinforcement learning. For now, it's not the case. We don't have published games who use it.

• 22:52Speaker 00

  But I really think that, you know, in the next year, it will be the case. But there is still some, you know, drawbacks to to find solution. But yeah.

• 23:02Speaker 03

  So as a bit of a tease for those who were curious to try more of the course, for example, I saw that on the website, there is, of course, Doom, right? Run Doom on everything.

• 23:13Speaker 00

  Yeah.

• 23:14Speaker 03

  So what are you doing in that course, for example, with Doom and with the games that we know?

• 23:18Speaker 00

  So during the course, we use, yeah, a different environment. We use Sonic, we use Mario, we use Doom. And in Doom, what you do, there is two levels. One, it's a simple level where you are in poison place. So you you lose health every time you walk. And so you need to grab, you know, the health pack to survive. So there is health pack response randomly. And the second one is you are in a corridor and there is some monster and you need to shoot them before going at the end of the corridor, otherwise they will kill you. And after that, you have tons of other environment that you can use because always what we do in the chapter is that we always start with simple environment. And then we show you how you can iterate a more complex one.

• 24:02Speaker 00

  And in this Doom, there is, you know, death match, death match environment that you can use. And you can train it, but it takes much more time than this smaller environment.

• 24:12Chris Benson?

  So I guess at this point, you know, with a little bit of the background on kind of how to get into this. The gaming and how to do that. Where if someone goes through and they've done your course and they've started and they've kind of, they're learning their way into it, it's their first steps. When they get to the end of the course, what kinds of things do you expect that they'll be able to do and be comfortable with? And then also, in addition to that, kind of where would you think they might want to go next? So they've kind of come through your course, it's the first one. They've now not only dipped their toes into this field, but have gotten some good hands-on experience. What comes next? What can they do and where can they go from there?

• 24:50Speaker 00

  So after the course, they normally have the skill until the state of the art. So we go until Proximal Policy Optimization, which is one of the state of the art, architecture and deep reinforcement learning. So they have the skills in terms of theory and in terms of practice. The next step will depend on what we want to do. If they want to work in video games, I would advise to learn to use Unity ML agents. And in Unity ML agent, they have three courses on how to use this library. And on the other hand, if they want to work in research, I would advise to use Spinning Up Reinforcement Learning. So it's a course and a library made by OpenAI.

• 25:34Speaker 00

  Or to use Clean Reinforcement Learning, the one I talk about, to dive deeper on more complex implementation.

• 25:43Speaker 03

  So what are some surprising or unexpected uses that you see deep reinforcement learning was being used in the community?

• 25:53Speaker 00

  I think there was multiple. The first was the mine who use reinforcement learning for their color system in their server bay. The other thing, I think, we work, when I was at the tech, who we work on what we call dynamic pricing. And for video games, yeah, video games, there is a lot of funny things. For instance, you can train your agent on what we call Mudjoko. And you can train, you know, like two feet, but you train to learn to walk with only two feet. You train to make what they call the half shita to run, etc. So there is a lot of funny things around it.

• 26:33Chris Benson?

  So you say that the thing that was in my head as you were describing that was it seems like you really have to break things down to address things in a very precise way. So as we are moving forward, and this is in the deep reinforcement learning toolkit, for lack of a better word, is now part of creating games and all sorts of other things as well, obviously, is that really what you kind of have to do is break down all of those things and think about how do I train just the feet, just the walking? Is it, might there be a time that you foresee where you can combine those and it's able to do very complex things? Or are we still a long way from that point? Are you still gonna have to do everything broken down in its individual constituents for a while?

• 27:17Speaker 00

  No, you can, there is some environments in ML agent where you train a complete humanoid to walk, etc. But there is a technique that we call curriculum learning. The idea is that you start to train your agents on very simple environments and then you increase the complexity. So for instance, if you have an ensemble fight environment, you know, you start first to train your agent to learn to shoot snowball and then you add, you know, the enemy. This way, you know, the agent will learn step by step. So it helps, you know, the training to have that kind of techniques.

• 27:52Speaker 03

  You bring some interesting perspective into those different things that are, or the different usages for the deep reinforcement learning in your community and in general. Do you see that your original degree, which was not technical and was a bit of a different background, did it contribute to something? Did it help you in the way? Can you motivate people who have, who are positioned in a similar starting point as you were when you started? And how can this be a benefit?

• 28:20Speaker 00

  Yeah, I think it's very nice because I think the premise that when I started, I thought, you know, that it was, it was something like a negative thing, you know, to not have a technical background and to come from, you know, humanities background. But I think it's really important to consider that it's something positive when you start because you have, you know, You have other way of thinking of other way of walking than people who are more technical. So it's something to to keep in mind. And I think that it's something that is is feasible because nowadays you have access to so much, you know, free educational resources online. For instance, if you want to learn mathematics, you have MIT OpenCourseWare.

• 29:06Speaker 00

  If you want to learn deep learning, you have Udacity and you have a lot of MOOC, you know, online. So my advice, if for people who are come from, you know, humanities or less technical thing is just try, start. And you'll see that, you know, it's really funny to to to study. So. And the second thing is really, don't feel like you don't belong here because you know, you're not technical. You need to continue to work on and just don't think about the fact that you don't belong here because you belong here. If you do stuff, you know, you belong here, whatever your background.

• 29:40Chris Benson?

  I'm so glad Natalie asked me that because your answer really got me thinking about something here and that is, I'd like to expand that a little bit, extend that the the line of thought that you're on. We've been historically kind of in a place where there was quite a lot of barriers to entry for people moving into deep learning. And early on, you know, there the university systems around the world are changing now to accommodate this. Most of us in the early days were coming from someplace else. It was hard for some people more so than others. I was just a software developer and I didn't have a lot of the mathematics on that. And so I had a steep learning curve myself. And with, but we've seen this trend over the years of more tooling. Hugging face has been a huge part of that, of the tooling becoming better and better.

• 30:25Chris Benson?

  And it's kind of democratized the entire space. And it's allowed a lot more people to come into it without having to jump through quite as difficult of, you know, a set of hurdles as maybe the beginning days when there weren't wasn't so much in the way of of tooling and learning resources. So do you envision a day where you can be productive along these things? Where the things like doing deep reinforcement learning for gaming and stuff becomes a low enough bar to where somebody with some level of aptitude, some interest in that can kind of dive into it, even if it's not their primary thing. To where they, you know, where it's, do you envision that it becomes that level of democratized?

• 31:09Chris Benson?

  Or do you think there'll always be? A certain level that you've got to really get to that's meaning when I say that meaning a bit of a challenge to get to as an entry point into it. How do you see that changing over time?

• 31:22Speaker 00

  Well, I think it's clearly possible because, you know, at Hugging Face, we already have something called Auto NLP that allows you to train and to create models without coding. And I really believe that the next step in deep reinforcement learning is really that type of thing, you know, being able to just, you know, directly train the model without having, you know, to know everything about the model or how it works, et cetera. But I think, yeah, it's very still work before that. But in the future, it's clearly possible because, you know, AI will become something that we'll use as we use software, you know. So we will not need to know how it works. We will not need to know behind the hood, you know, it works.

• 32:03Speaker 00

  We just need, you know, to start the training and having some tremendous results.

• 32:08Speaker 03

  And in addition to the human part of this, what are the fields that you see in the future that will be revolutionized by deep learning?

• 32:19Speaker 00

  I think everything, you know, especially the industry, you know, with the robotics, we'll see more and more, you know, robotics. And I think on the one hand is a good thing because it will remove a lot of work that are very hard and that really, you know, underpaid. So it's a good thing, but also I think in health, you know, we already see, you know, application in health with, you know, cancer detection. We see also, you know, with DeepMind who works, you know, on the question of protein detection. So for me, AI will change everything, but it's really important to consider that it will not solve everything, you know. For instance, you know, we all talk about the question that AI will help a lot in healthcare,

• 33:01Speaker 00

  as I mentioned, but the problem is that if people can't afford, you know, to access these tools, the problem is not solved. So I think what's important to consider is that we need to improve AI, but AI must be, you know, really accessible to everyone to be a solution to, you know, human problems.

• 33:19Chris Benson?

  That's a great point you make there. And we, we find ourselves in different aspects talking about, you know, pieces of that all the time in terms of, of who has access to the technology. And, you know, some of us who are privileged to be in this field, maybe we get stuck in our bubble at times. And, you know, we're, you know, there's robots are starting to come around here and it's right around the corner. And we're thinking about this. But there's kind of huge, huge segment of the world out there who hasn't really had access to this yet. So given that, you know, even with the best work from people like you and Hugging Face and others out there, we definitely have challenges of equity

• 34:02Chris Benson?

  in terms of people having access to, to good solutions for their life. Do you have any thoughts on things that we can do? Because I know, I think it needs to be forefront of our mind, things that we can do that are in this field to try to steer these solutions toward those places and people that might need them worse than we do in a lot of ways that don't have all the benefits that some of us are enjoying right now. Any, any thoughts on how we find that path? It seems really hard to me.

• 34:31Speaker 00

  Well, I think there is a lot of positive things that's happening, especially because of hope and education. You know, the majority of people who look at MOOC are people from developing countries. So the new generation in especially in countries, you know, developing countries are more and more educated. So I think if we want that AI will be accessible for everyone, we need to make sure that our models are open source, that, you know, everyone can participate in the documentation, participate in the project. And the second thing is more, I think, a politician thing, because we will need, you know, laws that make, to force, you know, companies to make their AI accessible, to make their AI, you

• 35:15Speaker 00

  know, that we can analyze what's inside the AI to avoid bias and all that problems. So I think it's, it's two part for us. So for people who work in AI, we really need to work on the question of making education. Content free or at least very cheap and working on open source projects and try to share as maximum as possible. And on the other side, we need to push our politician to work on legislation, um legislation. To make sure that AI will be here for good and not for negative stuff.

• 35:51Speaker 03

  Yep, that makes a lot of sense. I absolutely agree. Oktoberfest is also time of the year where everybody who wants to start with open source. Uh, I always like recommending contributing to the docs. Like you say, this is a wonderful way to participate and it has a less of a threatening entry threshold. Oh, I could absolutely not agree more. So the future scientists who is listening to this episode today, what is the first step that they should take in this field?

• 36:19Speaker 00

  I think the first step will be to just start with an introduction to deep learning. And then, you know, to try different projects, just to see what you prefer. If you want to dive more on NLP, so natural language processing, on deep reinforcement learning or, you know, on computer vision, audio, etc. I think the idea is not really to overthink, you know, this part because most of people, you know, that I met want to start always get stuck, you know, because there is a wide range of possibilities. And the, the only thing is just, you know, type on Google what you want to, to start from and start to learn deep learning. Take the first link, see how it whales, how it goes, you know, and then iterate from that.

• 37:03Speaker 00

  But yeah, don't overthink, you know, because most of people overthink and just say, oh, this is maybe not the best resources. This is not maybe what I want to do, etc. Just try and you'll see.

• 37:13Chris Benson?

  So that future scientist has taken that first step. They are diving into this and, and doing exactly what you just said. Along the way, they're taking your course and they are learning these skills. And they are, they're in a world that is increasingly, you know, having this, this just amazing gamified experience that is now becoming available widely and not just, you know, in the context of Video games, but but across all of life. And simultaneously, deep reinforcement learning is driving robotics, which are becoming ever more common as these new scientists are growing up. I'm thinking, obviously, of my daughter in this context, as I'm sure you figured out.

• 37:54Chris Benson?

  So what is they grow up and they are adults now and they've come through this process. What do you think the world looks like? What do you think maybe some opportunities are challenges that they might face? And along the way, I would love your opinion about what you'd like to see as opposed to what you think will be there. You know, if you have any preferences about what you're aspiring for right now as a data scientist doing this kind of work.

• 38:18Speaker 00

  Well, I think, yeah, in the future, clearly, we will have much more robots than nowadays. They will do a lot of tasks, especially social tasks in terms of, you know, helping older people. We know with the question of autonomy, we will have autonomous cars, all of that question. What I really don't want to see especially is the question of using these technologies for, you know, military things, you know, because we see more and more, you know, the question of especially deep reinforcement learning. We see more and more people talking about using deep reinforcement learning for, you know, robots in during battles on that kind of stuff. And I think this is the worst thing to do.

• 39:01Speaker 00

  I think it's something that's unfortunately will going to happen because, you know, if someone does it, all the other country wants to do it, you know, to protect themselves. But it's really something bad because it's really something we don't want to do and we shouldn't do. But unfortunately, I think we don't have a lot of power on that. The thing is, if I want to be positive, is when you work in this industry, we always need to think about the consequences of what we do. You know, sometimes it's very small consequences, but when we work on some models, we can, you know, induce bias that can, you know, be negative for a group of people. And it can be, you know, much, much worse than that if we speak about, you know, robots in army.

• 39:46Speaker 00

  But it's always important when we work on something to think on the consequences. Because sometimes, you know, we are too focused on the project and we don't think about, you know, the outcomes of what we are doing.

• 39:57Chris Benson?

  Well, Thomas, thank you very, very much for coming on to the show today. And Natalie, thank you very much for co-hosting with me. This has been a lot of fun. It's been a great conversation and really appreciate you. I hope you will come back sometime and tell us kind of what you've done next in deep reinforcement, learning and other things. Sure, thanks.

• 40:16Speaker 07

  Thanks, Thomas.

• 40:20Speaker 01

  All right, that's Practical AI for this week. Thanks for listening. If this is your first time with us, subscribe now at practicalai.fm or simply search for Practical AI in your favorite podcast app. We're in there. And if you're a long time listener, do us a solid by recommending the show to a friend. Word of mouth is still the number one way people find new podcasts they love. Special thanks to our partners for supporting our work. Fastly, Launch Darkly, and Linode. We appreciate it. And to the mysterious Breakmaster Cylinder for cranking out new beats for us all the time. That's all for now. We'll talk to you again next week.