Robot Perception and Mask R-CNN

Chris DeBellis, a lead AI data scientist at Honeywell, helps us understand what Mask R-CNN is and why it’s useful for robot perception. We also explore how this method compares with other convolutional neural network approaches and how you can get started with Mask R-CNN. Sponsors: DigitalOcean https://do.co/changelog – Enjoy CPU optimized droplets with dedicated hyper-threads from best in class Intel CPUs for all your machine learning and batch processing needs. Easily spin up a one-click Machine Learning and AI application image and get immediate access to Python3, R, Jupyter Notebook, TensorFlow, SciKit, and PyTorch. Our listeners get $100 in credit! Hired https://hired.com/practicalai – Salary and benefits upfront? Yes please. Our listeners get a double hiring bonus of $600! Or, refer a friend and get a check for $1,337 when they accept a job. On Hired companies send you offers with salary, benefits, and even equity upfront. You are in full control of the process. Learn more at hired.com/practicalai https://hired.com/practicalai. Featuring: Chris DeBellis – Website https://www.linkedin.com/in/chrisdebellis/Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Matterport R-CNN https://github.com/matterport/Mask_RCNNMask R-CNN paper https://arxiv.org/abs/1703.06870COCO dataset http://cocodataset.org/#homeStanford CNN course http://cs231n.stanford.edu/Stanford Deep Learning course http://cs230.stanford.edu/Facebook’s Detectron https://github.com/facebookresearch/DetectronUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Baiting for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to lino.com slash change log. This episode is brought to you by digital ocean. They now have CPU optimized droplets with dedicated hyper threads from best in class Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application image. This gives you immediate access to Python 3, R, Jupiter notebook, TensorFlow, side kit and PyTorch. Use our special link to get $100

• 0:42Speaker 00

  credit for digital ocean and try it today for free. At the do.co slash change log. Once again, do.co slash change log.

• 1:03Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:29Chris Benson?

  Hey, welcome to another episode of the Practical AI Podcast. I'm Chris Benson, an AI and digital transformation strategist. And with me is my co-host Daniel Whitenack, who is a data scientist who likes to use AI to do good.

• 1:43Daniel Whitenack?

  Hey, Chris, how's it going?

• 1:44Chris Benson?

  Doing very well. We have a pretty good episode lined up here. I think we have Chris Tabelas, whom I'll introduce in just a minute. And he is a guy I've known for a while. And we're going to be talking about some computer vision stuff today that's really steady art.

• 2:01Daniel Whitenack?

  Yeah, I'm excited to kind of dig into a few of the nuts and bolts of some things that we've mentioned in previous episodes, but haven't really gotten into. The weeds with, so I'm excited about that.

• 2:12Chris Benson?

  I am too. So I'm going to introduce Chris De Bellis and tell you a little bit about him and then I'll turn it over to Chris for a second. Chris and I have actually worked together at Honeywell where we were both kind of plank owners of the very first dedicated AI team at Honeywell. Safety and productivity solutions. And I'm no longer with Honeywell, but Chris still is and is doing some really cool work. And we, I miss being able to work with him. So how are you doing today, Chris?

• 2:39Speaker 03

  Hey, Chris. I'm well. How are you?

• 2:40Chris Benson?

  I'm doing great. Thanks for joining us, man.

• 2:42Speaker 03

  Yeah.

• 2:43Chris Benson?

  This is gonna be confusing with two Chris's. That's true.

• 2:47Speaker 03

  Chris and I worked together for a while and we, we dealt with that. So hopefully that experience will come in handy here.

• 2:52Chris Benson?

  Awesome. So maybe I'll say Mr. De Bellis and make it sound all official.

• 2:56Speaker 03

  That's very formal.

• 2:58Chris Benson?

  Yeah, it doesn't feel right, though. So, okay. So we are, just for our audience, last night I was actually with Chris. Cause he was at the Atlanta Deep Learning Meetup doing a fantastic presentation on MASK R-CNN, which is a deep learning algorithm for computer vision. And we're gonna, we're gonna do a deep dive in this episode into what that is and the pros and cons and cool things about it. So I'm pretty excited about it. And Chris, thank you for doing such a fantastic job last night. And I'm really looking forward to our conversation today.

• 3:33Speaker 03

  Oh, my pleasure. Last night was a lot of fun. Had a good time doing it and looking forward to talking about it some more today.

• 3:40Chris Benson?

  Cool. Well, you know what? I am, I gotta start us off with the first question. And I guess I'd like you to just kind of tell us what robotic perception is, since we're talking about robotic perception for MASK R-CNN.

• 3:52Speaker 03

  Sure. So robotic perception is basically about scene understanding. Using sensors so that the robot can interpret the environment and understand its place within that environment. Typically, we do that through a combination of 2D sensors, 3D sensors, other types of sensors. But it's basically helping the computer that's part of the robot to understand that environment.

• 4:18Daniel Whitenack?

  So what kind of robots are we talking about? You know, like, what are, in the context of manufacturing? Or like, you know, Roombas? What are we, what are we talking about here?

• 4:32Speaker 03

  Yeah, so good question. Really, all robots, right? It's easy to think of it in terms of, say, a robotic arm in a manufacturing environment where it has to move and perform some task. But robot perception also applies to something like self-driving cars, where you have to understand the environment that you're in. So it's really all robots.

• 4:52Daniel Whitenack?

  Yeah, so a robot, I guess, doesn't have to mean like, you know, articulating arms and quasi-eyes or something, but really, any kind of, you know, machine that is trying to get some sense about its environment. Is that, is that accurate?

• 5:09Speaker 03

  Exactly. That's exactly right.

• 5:11Chris Benson?

  So I guess I'd like to get a sense, as we're tying some of this together in the beginning, you know, robotic perception's been around for a while with some other techniques, but in, in recent time, deep learning has really had a profound impact on it. And so I guess, can you describe the role of deep learning in robotic perception? And maybe put it in some context with, with some of the other methodologies that either are also currently being used, or maybe have been used in the past?

• 5:37Speaker 03

  Sure. So, you know, traditionally, we've used computer vision techniques that were not based on deep learning. So an example would be something like candy edge detection, Huff line transforms. These are, these are more traditional approaches to detecting curves and lines and edges of objects. And then, and there's still a lot of that type of approach being used within robotic perception. But around, I'll say, 2010, 11, 12, right? That's when we started to see deep learning being applied to computer vision tasks. So AlexNet came out in 2012, and it was one of the first algorithms, the deep learning algorithms, to try to classify objects.

• 6:22Speaker 03

  And then, things have just kind of built on top of that. And so later, algorithms came out. They get, the neural networks became deeper and deeper and more capable of detecting and classifying these objects. And so really, that's sort of been the trend over the last few years, is to move from that traditional approach of computer vision to the deep learning approach for computer vision and perception.

• 6:46Daniel Whitenack?

  And you mentioned kind of detecting and identifying. Is there really two parts of it? Is it about like knowing, you know, knowing there's an object in my environment? And then secondly, knowing what that object is? Are those typically separate things? Or do those go together? How is that handled?

• 7:07Speaker 03

  So in deep learning, typically we're doing a few things. So we're taking classification, which is typically thought of for single objects in an image. And you'd say, oh, that's a picture of a cat or a dog or a person. But the more advanced algorithms are able to detect multiple objects within that scene. So you might say, hey, I see two cats and two dogs and a person. And you would be able to identify where within the scene each of those objects actually is. And so then this mask R-CNN algorithm can actually go a step further and say, which pixels within that image that I'm seeing belong to which object?

• 7:49Speaker 03

  So these pixels go to this cat and these pixels belong to this dog.

• 7:53Daniel Whitenack?

  Interesting, yeah. And just for our audience members who are joining us in this episode. In a previous episode, in episode seven, we had a great discussion with Jared Lander about what deep learning is itself and how it fits into the landscape of AI. So definitely check that out if you're kind of first learning about these things. But you mentioned, you know, mask R-CNN. Maybe we could just kind of start moving that direction by talking about, you know, breaking down that acronym. So like the CNN part is not a news network, right? What are we talking about?

• 8:32Speaker 03

  So CNN is, in the deep learning world, known as convolutional neural network. So it's a neural network that's based on the idea of these convolutions. The R in R-CNN is region. So the way that the algorithm goes about figuring out what's in each part of the image is by generating these regions. They're called regions of interest. And then it looks at the regions of interest that it generates and tries to detect if there's an object in that region. And if it does detect an object, It tries to classify it. If it doesn't detect an object, it just says, oh, this area is just background.

• 9:12Chris Benson?

  So, quick question for you, Chris. As we start looking at Maskar CNN, could you actually give us a little bit of a an understanding of how that fits into the larger category of convolutional neural networks and give us a baseline of what CNNs are able to do? And then contrast that as we start working into Maskar CNN?

• 9:35Speaker 03

  Okay, so maybe we'll talk a little bit about CNN versus just a traditional feed-forward neural network. So maybe folks are familiar with like an LSTM or RNNs, things like that.

• 9:50Chris Benson?

  Yeah, if you could even give us, I think probably a lot of people in our audience are most familiar with just basic feed-forward networks. And if you could talk about what a convolutional neural network does on top of that, what it's adding to it, and then we can kind of go into Maskar CNN and go farther. And that way, if someone hasn't been exposed, all three of us have been exposed to CNNs for quite some time, but it gives somebody a path on evolving where they're going with this.

• 10:16Speaker 03

  Perfect. So, in a traditional feed-forward network, you have the data coming in at one end of the network, and then you have several, maybe many, hidden layers. And the input, sorry, the output from one layer becomes the input to the next, and that's how it's a feed-forward. And typically, as you move through the network, you have fewer and fewer nodes within each layer. So you're doing fewer and fewer computations as you move along the network.

• 10:47Chris Benson?

  That helps a lot. And what does, when you add convolutions in, what does that do to that architecture?

• 10:53Speaker 03

  Right, so the convolutions are important for images especially because the convolutions are, the best way to think of it is, say, a three-by-three grid. So you're looking at three pixels by three pixels at a time, and you're moving that grid across the image from left to right, and then you go to the next row and you do it again from left to right. So you're moving down the image and going left to right, looking at a set-defined number of pixels at a time. And those, we call them kernels, and that kernel varies in size. So you might start with a three by three and go across the entire image. And then you could do a, maybe a nine by nine kernel and look at the image.

• 11:36Speaker 03

  And then a bigger one, maybe a, you know, thirty-two by thirty-two. And the important thing to remember with CNNs is because those kernels are square, you're maintaining, maintaining that spatial relationship between the pixels, which for images is important, right? If you think about an image and you're looking for, say, a cat, you know that the eyes are going to be close together. So you want to maintain that spatial relationship because the eyes should be close together. You shouldn't be looking for an eye in the upper-left corner and an eye, you know, in the bottom-right corner. That, that would be unlikely.

• 12:10Chris Benson?

  So there's a relationship between the pixels that matters because we're talking about images?

• 12:14Speaker 03

  Yes, the relationship, the spatial relationship, that's right.

• 12:17Daniel Whitenack?

  Yeah, and, and so you mentioned like things like AlexNet in, uh, earlier in our discussion, which, you know, are various, uh, various models that have been developed over time for image-related detections. Do a lot of these, uh, image-based models, um, or, or models trying to do object detection in images, um, is it fair to say that, that most of them involve convolutions in one way or another?

• 12:44Speaker 03

  That's right, they all involve, involve convolutions in one way or another. Uh, the difference really is in the size of that kernel, uh, the combinations of sizes that they're using, the values that are within those kernels for each of those cells. And then, over time, uh, how many layers there are in that network. Because as, as the technology got better, as the GPUs got faster, they could do more and more training in a set amount of time. And so they were able to have deeper networks.

• 13:14Daniel Whitenack?

  Awesome. Yeah, so I mean, we've kind of gotten into the convolutional part. And you've mentioned that there's kind of various incarnations of this. Maybe we can step back and kind of go back to the robot perception use case. And maybe you could share with us some of the challenges that you face in that particular use case and maybe challenges that might motivate you to use something like mask, R, CNN, which which we'll eventually get to. But what are the kind of motivating challenges in in this field that make just kind of the normal convolutional neural network, quote unquote, normal? I don't know if any of them are normal, but what might not make it enough?

• 13:59Speaker 03

  Sure. So with the CNN networks, basically, they're used for the classification. So I see a cat or a dog or a person or whatever the object is. And then typically, what we're doing is we're taking that network and also at sort of at the end of it, applying more technology to say, OK, can we identify where the object is in the image? So that bounding box, if you're familiar with that term, where we try to draw the box around the object, that kind of comes towards the very end. But we're still using the CNN network to identify what the object is. And so we do that additional processing later.

• 14:41Speaker 03

  So with something like mask, in addition to that bounding box regression to determine the position, we're also then doing another set of steps or calculations to say where within that bounding box is the image exactly?

• 14:56Daniel Whitenack?

  That makes sense. I'm wondering, just kind of maybe off of one of my my previous questions. So when you say you're kind of adding to the end, are you meaning like you have kind of different models for the bounding box thing and the detection thing? Or that's all kind of rolled into one end to end architecture? Is that kind of how it works? Or is?

• 15:18Speaker 03

  Yeah, good question. It's added on at the end. It becomes part of the overall network, but really it's tacked on at the end. And so the base, what we call feature extraction, which is pulling out those little features, the straight lines, the curves, a lot of the relationships between the pixels, that can often, and actually is often based on an existing classification network. So for instance, in the case of mask R-CNN, it uses something called ResNet to do its feature extraction and classification. And then on top of that, the creators of mask R-CNN added the ability to define exactly where the pixels are within the object.

• 16:03Chris Benson?

  And you mentioned something that I think would be worth taking a moment. You talked about feature extraction. And as we, as you work toward doing classification, could you take just a second and talk about how you do feature extraction from the simplest lines, you know, up through the different things that eventually become the classification of an object? Can you, can you speak to that for a moment for those who may not be familiar with it?

• 16:26Speaker 03

  Sure, I'll give that a try. It's a little tough without visuals, but basically, you know, a CNN network is really good at extracting what we call features, which is the example I just gave. So we're looking for maybe curved lines to one direction and then curved lines to the other direction. We're looking for those edges where maybe we have light and dark colors coming together. Maybe we're looking for straight lines. So if you think about like detecting something like an airplane, right, you would need a combination of all of these features to be recognized, right? So you would need straight lines to detect the wings, but you would need curve line detections for like the front of the aircraft, all right? So, and then where they are, because we talked a little

• 17:10Speaker 03

  bit earlier about spatial relationships. Where those features are matter, right? You need to have, for it to be an airplane, you would have straight lines kind of out to the sides and you would have more roundness in the center, for instance.

• 17:26Chris Benson?

  So would it be fair to say that you're starting with some of the most basic or atomic things, such as a simple line or a gradient from dark to white, and you're building up almost like Legos, an object out of these very primitive detections up into something that's more comprehensive, leading to your object?

• 17:45Speaker 03

  That's exactly right. So the earlier layers of your network are detecting those simpler features like you described. And then, as you add more layers, remember the output of the earlier layers become inputs to the next layer. So the next layers are operating on those features that were detected. And so it's trying to build patterns from features. So the earlier feature detection is looking at like straight lines and curved lines and things like that. And then it's looking for maybe curves that are like an eye. And then you're looking for, oh, two eyes together. That maybe is the part of a face. And then you add more features that have the whole head. So you're building, as you said, Chris, you're building from the finer representation

• 18:29Speaker 03

  of the features to more complex.

• 18:31Daniel Whitenack?

  Yeah, so I mean, this all sounds great. So I mean, it sounds like you've got your network. It's got this, you know, portion that's detecting all of these features and determining, you know, let's say if you have a cat in your image. And then you've got this portion tacked on that might be detecting bounding boxes of where that cat's located within the image. What prompts someone to go further than that? So to go beyond CNN? Kind of going back to where we started this conversation, why the mask R part? What challenges are still present, even if we're using kind of this CNN-based network?

• 19:10Speaker 03

  Yeah, great question. So the example we used last night at the meetup from the Udacity Robotics Nano Degree Program, one of the assignments is to take a robot that has two arms and in front of it is a desk with various objects, things like a bar of soap. I believe it was an eraser. A tube of toothpaste, et cetera. And you have to perceive what's on the desk. And then you have to manipulate the robot arms to grab the item and then put it into a basket to the side. So if you think about grabbing that tube of toothpaste, well, if it's perfectly aligned

• 19:51Speaker 03

  with the table, then you just kind of reach forward and you grab it. But if it happens to be turned at a 45-degree angle, you also have to adjust the arm to match that rotation, and then you can grab it. So if you think about a bounding box, well, the bounding box just says, hey, somewhere in this box is this tube of toothpaste, but you don't know which way it's pointing or how it's oriented. The mask, since it fills in the pixels for you of where the object is, you can look at it and say, oh, it's not straight up and down. It's actually at a 45, and so I need to turn my arm.

• 20:26Chris Benson?

  Okay, so I'd like to ask a question before we dive fully into MaskRCNN about what the options are within the different CNN architectures that might be available for robotic perception, you know, such as YOLO or others. And at a very high level, if you could just give us a sentence or two on kind of of what different options there are and then why you might have chosen to go MaskRCNN for a given solution that you're looking for.

• 20:58Speaker 03

  Okay, so something like a YOLO, which is a great algorithm, it only gives you the bounding boxes. So a lot of times, though, that's all you need, right? So I'm trying to think of some good examples. So like if you're doing maybe a self-driving car, if you're able to detect in front of you is a pedestrian or another vehicle, and you have a bounding box around it, that's probably close enough, right, for being able to make a decision as to what you should do, right? If this thing is clearly in front of you, it doesn't really matter exactly where the mask outline of that object is. You're able to detect that there's something in front of me, and I should perhaps slow down or stop.

• 21:39Speaker 03

  MaskRCNN, because it gives you the masks, it's really good for when that orientation matters. So the example we just gave about a robot arm having to pick objects off of the table is a good example.

• 22:01Speaker 00

  This episode of Practical AI is brought to you by Hire. One thing people hate doing is searching for a new job. It's so painful to search through open positions on every job board under the sun. The process to find a new job is such a mess. If only there was an easier way. Well, I'm here to tell you there is. Our friends at Hire have made it so that companies send you offers with salary, benefits, and even equity upfront. All you have to do is answer a few questions to showcase who you are and what type of job you're looking for. They work with more than 6,000 companies from startups to large publicly traded companies in 14 major tech hubs in North America and Europe. You get to see all of your interview requests. You can accept, reject, or make changes to their offer even before you talk with anyone.

• 22:46Speaker 00

  And it's totally free. This isn't going to cost you anything. It's not like you have to go there and spend money to get this opportunity. And if you get a job through Hire, they're even going to give you a bonus. Normally it's $300, but because you're a listener of Practical AI, it's $600 instead. Even if you're not looking for a job, you can refer a friend and Hire will send you a check for $1,337 when they accept the job. As you can see, Hire makes it too easy. Get started at hired.com-practicalai.

• 23:25Daniel Whitenack?

  So in terms of, you know, some of the challenges with moving beyond this bounding box sort of idea and moving more towards the mask idea. It occurs to me that, you know, it's already a somewhat challenging problem to get good, you know, labeled training data for just like a bound, you know, the bounding boxes and labels of, of objects within images. It seems like that would be even more challenging if you're wanting to, you know, label the proper masks within an image for particular objects where you're getting even more detail, not just, you know, where the objects are within what region, but, you know, what the

• 24:09Daniel Whitenack?

  actual mask of the object is. Is that a problem or are there, are there, have there been techniques developed to deal with that?

• 24:17Speaker 03

  Yeah, it's a huge problem. So if you think about the simpler example of classifying an object. So is this a cat, a dog, a person? You could, if you were doing training on those images, you could do something simple like create a directory for each type of object. And for instance, you have a directory called dog and that directory name becomes the object name, the class name. And you put all of your pictures of dogs into that directory and you train. And that's, that's your labeling, right? But to do something like detecting the right location of the bounding box, you would have to take those images and draw the bounding box around the individual objects and then train.

• 25:02Speaker 03

  So extending that further to something like mask, since you want to get accurate masks, you can't just draw bounding boxes around each of the objects. You have to draw the actual outline. So you, you end up generating a polygon typically, some, some really odd shape, enclosed outline for each of the objects. So if you had an image, say of, you know, four cats and four dogs, that's eight objects you have to outline. And it becomes really tricky when they're occluded or, or one is in front of the other. So it's only partially showing and you have that common boundary between the two. You want to be really accurate when you do that. So yeah, labeling or annotating data for, for masks is cumbersome and tedious.

• 25:47Chris Benson?

  And one thing I'd like to, to clarify in case we have any listeners that aren't familiar with, with what masks are. Masks are specifically where you apply a bunch of pixels together to form that polygon that Chris was alluding to, to where if you were looking at it visually, it would almost be, maybe you're applying a color for those pixels and you'd almost have like a, a, a green overlay over a person's body that you're masking in a picture. And you might have many of those masks, but I, I just wanted to define that for, for everyone so they could follow along. With, I guess, as you, I, I like to talk a little bit about as you're, as you're getting into labeling the data and you're looking at the data sources that you're pulling in and, and how you do that.

• 26:31Chris Benson?

  What, what are the, the typical data sources that are used in the process? And how do they come together for the training?

• 26:37Speaker 03

  So if you're familiar with the COCO dataset, over the last few years, folks have been taking the COCO dataset and providing the masks. So they've been going in and annotating, providing that polygon around the individual elements or the individual objects within each of the images. So that allows the people that created the original mask R-CNN network to do transfer learning, which is you start with an existing set of weights. So they were able to use an existing set of images that were already annotated and create their algorithm. And then what we do now is we take those weights that they use to create the original

• 27:18Speaker 03

  mask R-CNN network from, and we use that as the starting point to train for images that we want to now detect. So let's say there's something else that we want to detect that's not part of the original COCO dataset. So we train with new images. So we have to go out and obtain those images, annotate those images, and then apply the training on those images with the COCO weights as our starting point. And that's actually called transfer learning.

• 27:48Daniel Whitenack?

  Awesome. Yeah. And when you're doing that, I mean, because if I'm thinking of, you know, I'm, you know, listening to this podcast, listening to you, you know, talk about all of these exciting things, I might want to, you know, I might have a use case that's really interesting for this, or I might want to try it on data that's maybe, you know, like you said, not already masked as part of, say, the COCO dataset. Is that just like when you're, when you're doing that in your context, is it a matter of you and your team going through and annotating those images? Or have you kind of found any efficient ways to, you know, crowdsource those within your organization or anything

• 28:32Daniel Whitenack?

  like that? Or have you heard of any, any ways to kind of speed up that process? Or is it still just kind of brute force getting through everything?

• 28:40Speaker 03

  Yeah. So that's a great question. Unfortunately, it was me and my team that had to annotate the first set of images, and that took Took quite a while, the images we were.

• 28:48Chris Benson?

  Like how long?

• 28:49Speaker 03

  So the images we were doing would have anywhere from, say, up to maybe forty or fifty objects in it. And it might take fifteen or twenty minutes to annotate one image. And so, you know, with deep learning, you want to have a lot of images. You want to have a lot of training data. So after, I think, a few hundred of these images, we kind of said, you know what? Let's just do a proof of concept with what we have because it's taking so long to annotate. And we got to that point and we created our model. And then we said, okay, you know, we proved out the concept and said, okay, if we really want to go forward with this, we need to do this at scale. And so, as you pointed out, yeah, you want to either engage with a company that does this. There are a number of them that do this for you.

• 29:34Speaker 03

  They hire folks really around the world that can go ahead and annotate your images for you. And that's really the way to go at scale.

• 29:43Chris Benson?

  Yeah, so, you know, bribing people with pizza and getting together, you know, one night to annotate data sets only gets you so far.

• 29:52Speaker 03

  It really does, especially if it's taking fifteen minutes per image. You know, you wouldn't get too many done even with a couple of pizzas.

• 29:58Chris Benson?

  So I guess I would draw us back to mask R-CNN. And I guess ask you to kind of, as we start talking about the algorithm itself, can you define what R-CNN is? And then define when you add mask over that? How would you do that with the intention here of taking us deeper into the specifics of the algorithm?

• 30:22Speaker 03

  Sure, so the CNN, as we said, that's the convolutional neural network. R is the is region proposal. So again, the way that this algorithm decides whether or not it sees any objects is it looks in different regions or different parts of the image. And it tries to classify what it sees in each of those parts as being either background. Or not background. And if it says, hey, this is not background, then it tries to figure out what it is exactly. It tries to classify it. So the regions are different sections of the overall image that it's looking at in different scale and different proportions, different sizes.

• 30:59Daniel Whitenack?

  And then the mask bit is just the idea. So instead of tacking on the end, the bounding box piece, you're kind of tacking on the piece to actually map out these masks. Is that right? Is it working the same way in that you would kind of bolt this on to the end? Or is that different?

• 31:17Speaker 03

  Exactly. Exactly. So towards the end of the network. And the reason it's at the end is because you're using those same features that you've extracted earlier in the network that you're using to classify it. You're also using those features to decide where the mask should go. So a point about the mask, probably the best way, at least the way that I think about it, is the mask gives me the X and the Y or the, if you want to think of it in terms of the image, the row and the height coordinates of each of the pixels that belongs to that object. And that's really important in something like a robotic application because everything, we said earlier, you have multiple sensors, right?

• 31:58Speaker 03

  So all of these sensors need to be triangulated and aligned so that you can make decisions from multiple sensors from the same point of view. So having that X and Y coordinate or that row height coordinate that exactly defines all of the points that make up this object is really important.

• 32:16Chris Benson?

  So when you're considering mask R-CNN as an architecture for your own use case, I guess when you're comparing it against alternative architectures, YOLO or others, is it really the use case that's dictating going there because your use case needs the benefits of the mask versus a bounding box? Is that how you would think about it?

• 32:37Speaker 03

  Definitely. So we've just talked a little bit about mask R-CNN. It's great to have those masks, but it comes at a little bit of a cost. It's one thing we haven't specifically said, but it is computationally expensive. These algorithms, the more you do, the longer they take. And so adding on or tacking on this extra functionality, these extra mathematical operations that have to be performed, even though it's being performed on a GPU, highly parallelized, it still takes extra time. So it may not be necessary. It may not make sense that in your application, you want to spend that extra time generating these masks,

• 33:19Speaker 03

  especially if the bounding box is sufficient. As Daniel pointed out just a few minutes ago, training is more difficult. So the tedious task of generating all of these annotated images, you have to do that as well. It's a great algorithm when you need it. But if you don't need it, it probably doesn't make sense to implement it because something like a YOLO, which Chris you mentioned earlier, is faster if all you need is bounding boxes.

• 33:48Daniel Whitenack?

  Yeah, I want to dig in a little bit to that idea that you brought up around efficiency. So I mean, there's the training side of things, which is one piece of it. And I would imagine these networks being trained on huge GPU boxes wherever you have them or a big cluster in a distributed way. But when we get to talking about inference, so taking that trained model and then making inferences, so utilizing the model to actually detect objects and masks in an environment. Does the network size and the complexity also factor in on the inference side? I mean, I know we're talking about robots.

• 34:31Daniel Whitenack?

  So if you're kind of shipping this model out to run on a robot, I'm assuming that that robot doesn't have a huge rack of servers on it necessarily. It might have a smaller amount of computation on the actual robot. Has that been something that you've had to factor in as well?

• 34:51Speaker 03

  Right, that's exactly right. Typically, when you're training, you might be training in the cloud and you can spin up however many GPUs you need for training and that reduces your training time. But for inference, you probably just have one GPU on your robot. And so, yeah, you definitely have to consider that inference time. So if you're trying to do something in near real time with streaming video, um, mask R-CNN is going to be a bit challenged because it may be only able to process two or three or 10 images, depending on the size, per second. So you're absolutely right. The other thing too is oftentimes the GPU that you're using for training might be more powerful than the GPU on your robot.

• 35:34Speaker 03

  And so not only do you have fewer of them, you have a less powerful one. So inference becomes even longer.

• 35:40Daniel Whitenack?

  Could you just, you know, real quick, um, because we've talked about this, you know, mentioned it in a bunch of times, but I think this would be the perfect context to really clarify, you know, why in both of these cases you've mentioned using the GPU, why in particular for these types of networks is a GPU necessary?

• 35:58Speaker 03

  So, good question. If you think about something like mask R-CNN that's built on a ResNet 101, so 101 means it has 101 layers. And we talked before about these convolutions that happen. So you're looking at this overall image. So if you have an image that's 1,024 by 1,024 pixels and you're looking at it in just one layer, three by three, and then spreading that over the entire image, and then looking at it again, maybe nine by nine, and then, you know, 64 by 64, various size kernels. And the other thing too, we haven't talked about, a color image, it actually has three channels deep, right?

• 36:41Speaker 03

  You have a channel for red, a channel for green, and a channel for blue. So those convolutions actually are doing three times the work on that first layer because it has to look at the red, the green, and the blue values. So if you think about that, just in one layer, and then you're going to do this over 101 layers, you get into billions of floating point operations that have to happen.

• 37:03Chris Benson?

  Cool. So let me ask you this, as we kind of start to wind up here, moving in that direction, if you're listening to this and you've gotten all excited about being able to use mask or CNN for robotics or other uses that you might be interested in, what types of skill or knowledge are kind of prerequisite to get into this and to be able to work toward using it productively? How do you get started along that path?

• 37:31Speaker 03

  Good question. So at least for me, I'll talk a little bit about my experience. To go from, say, traditional data science into deep learning algorithms, I think one of the big skills that you have to have is coding skills, right? You're going to be doing a lot of coding. You're going to be downloading other people's code, probably from GitHub. You're going to be configuring it, installing it, and then you're going to be, at minimum, tuning some parameters, but very possibly, especially if you're doing this in a production setting where your code is going to be actually used for something, you'll have to make changes, code changes. So the ability to code is really important, particularly Python. Most of these

• 38:15Speaker 03

  algorithms are available in Python. I would say, and there's a lot of debate out there. I know some folks say, oh, to do deep learning and data science, you really have to have a strong understanding of math and statistics. And I think if you are doing AI research, that's absolutely true. But if you are doing, like we talked about earlier, that transfer learning, a lot of the math and statistics comes from training the initial model. So if you're using someone else's trained model as your starting point, the ability to do the math and statistics become less important. And I know some folks are not going to like that, but that's been my experience. Over the last six months, say, most of my time

• 38:56Speaker 03

  has been spent coding, not so much worrying about statistics and derivatives and matrix multiplications, because the software does that for you. So that's one of the great things about the frameworks like TensorFlow. And then, again, for me, to get started, I spent a lot of time watching YouTube videos. Stanford has a lot of great courses online. Their deep learning courses are online, and you can watch the lectures and really learn a lot from those. So for me, that was enormously valuable. Also Udacity, I took a couple of Udacity courses. They have some free courses. They have some paid courses. Those are really helpful.

• 39:32Daniel Whitenack?

  Yeah, no, I was just gonna, I was just gonna mention, I really appreciate you providing this, this perspective and being transparent because I think there are a lot of people that get intimidated, kind of going into this space and thinking that, you know, they, they don't have a PhD in mathematics, right? So what, what difference can they make? But it is super encouraging, you know, for, for myself to, to hear you talk about, you know, some of the things that you've been involved with and, and you've done. But, you know, coming at it from more of the coding perspective and from the transfer learning perspective and building up those skills as you have, I think, you know, for me, it's an encouragement as, as I'm learning more things and I hope it is for, for the audience members as well.

• 40:14Speaker 03

  Yeah, and that's absolutely what I was hoping people would take away from my comments that, you know, if you're passionate about it, don't let anybody tell you, you can't do it. And it's not easy, but it's not impossible. And there are going to be days where you, you're looking at something and you're looking at these crazy formulas and you're going, I just don't want to deal with that today. And that's perfectly fine. And there are days when you look at it and you go, you know what, I'm going to dig deeper and I'm going to see if I can't make sense of some of this. And over time, it starts to make sense, especially it's repetitive. You see things over and over and over and you start to connect the dots. And then, you know, just the, you're, the light bulb goes on one day and you, you go, oh, I get, I understand batch normalization. Now I understand why we normalize things. I didn't understand that three months ago, but now I finally

• 40:58Speaker 03

  get it. And so that's, that's really for me, what it, what it takes to, to get, to be successful is, is that passion and enough of a foundation to just keep growing and growing and improving yourself and your skills.

• 41:10Daniel Whitenack?

  So as we wind down, I guess as a kind of a last thing to touch on here, I wanted to ask you, I know that you introduced me and the rest of our team at Honeywell to a particular GitHub repo. And then you talked again through that at the meetup and I wanted to, to bring that out and we'll put it in the show notes. But for, for those of you maybe listening, it's, it's on github.com/matterport/mask_rcnn. And, and if you would just give us a quick overview of the Matterport

• 41:42Chris Benson?

  Report mask R-CNN repo and what's possible there. And that way, we can kind of leave that in our listeners' hands to go explore further.

• 41:50Speaker 03

  Sure, happy to. So the mask R-CNN algorithm actually came out of work that was done at Facebook. Been several, at least that I'm aware of, implementations of it. So Facebook has their own called Detectron, which is written in Caffe too. Google has an implementation in TensorFlow, pure TensorFlow. But the the version, Chris, that you mentioned and that I really like, it's a combination of some Keras, some TensorFlow, and a lot of preprocessing and post-processing of your your image in Python NumPy. And the thing I really like about it is they provide some Jupiter notebooks that they've written, which give you a good insight into what's actually happening with the algorithm. So it's not so much of a black box.

• 42:32Speaker 03

  You can you can follow along with these notebooks and kind of learn your way through like the R and R-CNN, where are those regions coming from? And why are there so many? And how do they figure out which ones to use and which ones to throw away? So the Matterport implementation is great for learning. They also have an active community. It's being updated. There's a lot of good information in the issues. So if you were to read through some of the issues that they have, folks have contributed and talked about some improvements to the algorithm. And you can really glean a lot of information as to what's going on and how the mask R-CNN algorithm works by reading those those posts. The actual structure, there's really a couple of main files.

• 43:15Speaker 03

  So there's model.py file, kind of has the functions to do training and inference. There's utilities.py file, which has some utilities. The visualizations are all in the visualize.py file. There's a config file, which has all of your parameters. So when you're doing your training and your hyperparameter tuning, that's where you would go. You can go and set them there. It's it's also a class. So if you want to override the class, you can do that. If you're pretty familiar with classes in Python, it's pretty easy to do. Those are the main Python files. The way to get started in the samples folder, there is a demo Python notebook. That's the place that I would start.

• 43:58Speaker 03

  There's also, I believe in one of the samples they give you is for training shapes of triangles and squares and circles, train shapes dot IPython notebook.

• 44:09Speaker 03

  That's how I would get started. That's how I got started. Read the information that they have, a lot of good stuff, and look at the notebooks and just get started.

• 44:18Chris Benson?

  That's fantastic. Thank you so much for taking us through that and giving us that last orientation on the repo. I know that as I'm looking forward to hearing back from listeners on what they've done with Mascar CNN. And so, first of all, thank you so much for coming onto the show and giving us kind of this deep dive through Mascar CNN. We really appreciate it.

• 44:41Speaker 03

  Yeah, my pleasure. It was a lot of fun. It was a little new experience for me doing this on a podcast without having visuals. So hopefully it came across well.

• 44:50Chris Benson?

  It came across great. I thought it was a fantastic tutorial. And for our listeners, I hope you guys will reach out to us on social media. It's really easy to get to Daniel and me. We are on Twitter, we're on LinkedIn. We actually have a Practical AI LinkedIn group that you can participate in. And then there's also, we have a community online with Slack at changelog.com/community. And we're looking forward to your feedback. Chris, is there any way that listeners can reach out to you?

• 45:22Speaker 03

  Sure. Probably the best way is just to find me on LinkedIn. It's Chris D'Beallis, C-H-R-I-S-D-E-B-E-L-L-I-S. I think I'm the only Chris D'Beallis out on LinkedIn, so hopefully you can find me.

• 45:34Chris Benson?

  Well, thank you very much. And I'm looking, after we get off the show, I'm going to dive into some Mascar CNN and have some fun today.

• 45:41Speaker 03

  Awesome. Good luck with that.

• 45:44Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our ears before our users do here at changelog because of Robar. Check them out at robar.com/changelog. And we're hosted on Leno cloud servers. Head to leno.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 46:29Speaker 00

  Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.