Robot hands solving Rubik's cubes

Everyone is talking about it. OpenAI trained a pair of neural nets that enable a robot hand to solve a Rubik’s cube. That is super dope! The results have also generated a lot of commentary and controversy, mainly related to the way in which the results were represented on OpenAI’s blog. We dig into all of this in on today’s Fully Connected episode, and we point you to a few places where you can learn more about reinforcement learning. Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. AI Demystified (FREE five-day mini-course) http://learnai.infinite.red/ – Get an introduction to the most important concepts, types, and business applications for AI and Machine Learning. This course is 100% free. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: OpenAI’s result of solving a Rubik’s Cube with a robotic hand: Blog post https://openai.com/blog/solving-rubiks-cube/Paper https://arxiv.org/abs/1910.07113Example push back on Twitter https://twitter.com/GaryMarcus/status/1185679169360809984Learning resources: OpenAI Gym Tutorial http://gym.openai.com/docs/PyTorch RL Tutorial https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.htmlTensorflow RL Tutorials https://github.com/tensorflow/agentsUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com/Changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized, they have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, CI, CD, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front-end web servers that need to be full duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/Changelog. Again, do.co/Changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:47Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI. In these fully connected episodes, my co-host Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and we'll dig into some learning resources to help you level up your machine learning and AI. Game. I'm joined today as always by my co-host Chris Benson, who is a principal AI strategist at Lockheed Martin. And I'm Daniel White, NACA data scientist at S I L International. How are you doing, Chris?

• 2:23Chris Benson?

  I'm doing great. How's it going today, Daniel?

• 2:25Daniel Whitenack?

  It is going great. It is a nice fall day here and enjoying the like two weeks of perfect weather before it's bitterly cold in the Midwest. Yeah, trying to enjoy that while I can.

• 2:39Chris Benson?

  Oh, see, when it gets bitterly cold, you can come down here. I think we're actually going to see each other in Chattanooga.

• 2:44Daniel Whitenack?

  Yeah, I'm excited about that. Yeah, we'll both be at the Project Voice Conference, which is happening in Chattanooga in January. So come by and say hi to us if you're around.

• 2:56Chris Benson?

  Absolutely.

• 2:57Daniel Whitenack?

  Talk to you about AI and voice and all of those things.

• 3:00Chris Benson?

  And we can talk about our nice mild southern winters here.

• 3:05Daniel Whitenack?

  Sounds good. Now on to I guess some AI news as this is a fully connected episode. I don't know about you, Chris, but for like the past week or however long it's been about half of my Twitter feed has been like Giffys and videos of robot hands manipulating Rubik's cubes. Has it been the same for you?

• 3:29Chris Benson?

  It has. It's been kind of the big topic as we're recording this over the last couple of weeks. It's been a really interesting thing and diving into it as we have both done in anticipation of this talk has been a lot of fun. So looking forward to today.

• 3:42Daniel Whitenack?

  Yeah, well, maybe to start out with, we should probably get a baseline. How good are you at solving Rubik's cubes, Chris?

• 3:51Chris Benson?

  So not so good by myself. So if I was going to do it like for my daughter, I would cheat and look at one of those algorithms. You know, you can see these manual algorithms.

• 4:01Daniel Whitenack?

  Your daughter might beat you.

• 4:02Chris Benson?

  Yeah, she probably would. Actually, I would have to daddy would have to go and cheat and look up because there are solutions out there, you know?

• 4:10Daniel Whitenack?

  Right. Methods and such.

• 4:12Chris Benson?

  Yeah, but but no, no, for real. I tried it when I was younger. I haven't tried it recently and utterly failed. So pathetic.

• 4:19Daniel Whitenack?

  I'm not good. At solving Rubik's cubes, I will admit that I've had some friends try to teach me various things. I remember one in particular in college, he was one of those just brilliant, naturally brilliant guys. It was one of my friends in in the physics program and when I would ride with him from his house to school, he would always try to solve, so he would drive with one hand and solve a Rubik's cube with the other hand, one handed while he drove, which I'm not recommending at all.

• 4:48Chris Benson?

  Oh, doesn't sound like a good idea.

• 4:50Daniel Whitenack?

  Yeah, it was about like a five minute drive and he could solve the Rubik's cube with with his one hand, which kind of disgusted me a little bit and was probably not safe in any way, but.

• 5:00Chris Benson?

  So my question to you is, had he looked at one of those, you know, algorithms that you can manually do it? Because I know there's one that I have seen before. I have not memorized it or anything, but you can solve any Rubik's cube in a certain number of very small steps. I mean, it's less than 20 that you would do.

• 5:17Daniel Whitenack?

  Yeah, I mean, it must have been something like that. Eventually he went on to like the five by five. So there's like three by threes and then he went on to like the five by fives, which I can't even comprehend working on. But yeah, that's a little bit too much for me, but maybe not so for OpenAI's Rubik's Cube solving robot hand, which is going to be what we're going to talk about today. They solved the three by three, maybe they'll go on to the five by five. But yeah, just to clarify, you know, what we're talking about in terms of the robot hand solving the Rubik's cube. If you weren't aware, OpenAI, a research team there just published a blog post and a paper,

• 5:59Daniel Whitenack?

  at least a preprint of a paper about using a robotic hand to solve a Rubik's cube. And it's pretty impressive. What were your initial reactions, Chris?

• 6:10Chris Benson?

  Correct me if I'm wrong. If I recall, they did use one of the standard algorithms for the Rubik's cube itself, I think. But I believe that most of the training was focused on the articulation of the hand itself. Am I right or am I wrong?

• 6:24Daniel Whitenack?

  Yeah, so there was definitely some pushback against the results, in particular.

• 6:30Daniel Whitenack?

  Maybe as related to that and we can get into that in a bit. But yeah, I think the main point was like, if you just imagine manipulating a Rubik's Cube with one of your hands and solving it, even for a human, like that manipulation is not trivial. And so the idea that they could get a robotic hand to manipulate this Rubik's Cube towards the solution, even if they are using this sort of other algorithm for the solution, is, uh, it's pretty astounding because it's a really hard manipulation problem, I guess, is, is the idea.

• 7:05Chris Benson?

  I agree. And I think, I mean, it's almost, uh, the, whether or not you're solving the Rubik's Cube is almost an aside in this. I think the fascinating thing for me at least was seeing this robotic hand, and there's just one, so you can't, you know, you don't have both hands to put it in, but the robot is both manipulating the cube in the appropriate directions and using the robotic fingers to pivot the cube. And the dexterity required to do that is something that I certainly would be challenged myself to try to recreate with my human hand at all. So it was just the ability for the robot to be able to act with that kind of dexterity was just impressive to watch the video that they had on the, on the site.

• 7:44Daniel Whitenack?

  Not only that they kind of solved it once, but they did a whole bunch of different interesting things with perturbations too. So this is kind of similar maybe to those Boston Dynamics videos and like the memes that are also like associated with these Boston Dynamics videos where they kind of like hit robots with hockey sticks and things. But they hit the robotic hand with like a stuffed giraffe and pushed it with a pen and threw a blanket over the hand and other things. And we're testing the adaptation of the hand to be able to continue towards the solution and continue manipulating the Rubik's Cube in light of these things that weren't part

• 8:28Daniel Whitenack?

  of the training data, but were perturbations that it had not been exposed to before. And I think that that also is part of the big story here is that the adaptability of the solution, I guess.

• 8:41Chris Benson?

  I agree. They added a lot of variability, which I know we're about to talk toward in this conversation. And there's an entire algorithm that's designed around that, which we'll discuss in a moment. But it was just, it was an impressive bit of engineering, both from a data science standpoint. But also, as I watched that video, I just kept thinking of all the applications. And not even necessarily robotic hands, but if you start thinking of other manipulation tools that you might have on a robot and applying the same principles. I'm pretty excited about it.

• 9:11Daniel Whitenack?

  So congrats to the OpenAI team. If any of you are listening, I hope maybe one or two of you will come across this podcast. Congrats and great work. So maybe what we could do now, Chris, I think would be good for me and probably the listeners too, is to just kind of break down how OpenAI went about this solution, what they used in terms of neural networks and setting up the solution and training and simulation and all of those things. To just kind of break down the main parts of this study. Does that sound okay?

• 9:44Chris Benson?

  That sounds like a good idea.

• 9:45Daniel Whitenack?

  So I went to the OpenAI blog post about this research, which we'll of course link in our show notes. And kind of the first sentence that they have there is, "We've trained a pair of neural networks to solve the Rubik's Cube with a human-like robot hand. The neural networks are trained entirely in simulation using the same reinforcement learning code as OpenAI 5, paired with a new technique called automatic domain randomization. The system can handle situations it never saw during training, such as being prodded by a stuffed giraffe."

• 10:24Daniel Whitenack?

  That summarization there is just like packed full of all sorts of things that I would love to break down. Maybe not so much the stuffed giraffe, although I was intrigued by that as well. And later on down, they also have another good summarization. They say, "We trained neural networks to solve the Rubik's Cube in simulation using reinforcement learning and Co-CEMBA's algorithm." I'm probably butchering that name. I'm sorry.

• 10:48Chris Benson?

  That's how I would have said it.

• 10:49Daniel Whitenack?

  And then they also talk about domain Randomization. So some of these things I'm more familiar with than others and there's some new things here as well. So maybe we can start by just digging into these pair of neural networks that they're talking about. Were you able to kind of deduce from their blog post or their paper? Anything about this pair of neural networks? What pair of neural networks are these?

• 11:14Chris Benson?

  Yeah, for the computer vision side, they use three cameras to address the aspects of the Rubik's Cube. And then for the manipulation, they had a reinforcement learning policy. And so they had the control policy network that was addressing that. And so both of those networks had to be trained to be able to work together, to be able to manipulate the Rubik's Cube and understand where they were statewide at any given moment.

• 11:41Daniel Whitenack?

  So the two networks, I guess, then would be this control policy network. And then the convolutional neural network, which has to do with estimating the position of the cube in the hand. So the control policy network, in terms of input and output, what I was getting from the paper is basically the input of this control policy network would be the positionings, kind of noisy positionings of the Rubik's Cube and the robot hand. So there were actually, you know, either from the convolutional neural network or from sensors in the cube and the hand, they had these positionings of the cube and the robot hand.

• 12:22Daniel Whitenack?

  So you have these positionings, basically the state of how the cube is in the hand as input. And then the output would be what action does the robot hand need to take? In particular, that brings the state of those things closer to solving the cube. So in comes these, what they're calling observations, outcomes, actions. Now, I guess for those observations, they need some way to estimate the positioning of the cube. And is that where the the convolutional neural network comes in?

• 12:54Chris Benson?

  Yeah, I think they they were using the convolutional neural network for the three cameras that were observing from different positions. The various aspects of the Rubik's Cube. And then I do recall also that they mentioned that the robot's hand was also tied to a 3D positioning capability that they had. So they'd understand where the robot was at any given moment in relation to how the cube was to combine those two so that it could do the policy-based manipulation.

• 13:24Daniel Whitenack?

  Yeah, it was a combination of, you know, the detected position of the cube, the tracked position of the robot hand fingers or hand in general. And then they also were able to kind of track the faces of the cube by instrumenting the cube as well. So there was definitely a lot of instrumentation going on here, but essentially all of that is considered observations, which is passed into this control policy network to produce what action should the hand do? And my understanding is that these actions are essentially the sort of desired angles that they want

• 14:09Daniel Whitenack?

  to manipulate the cube into or the faces into. I'm not sure what the dimensionality of that is because there's a lot of these faces of the Rubik's Cube. So all of the observations come in and the outcomes, the actions. You mentioned that like the control policy had to do with reinforcement learning. The convolutional neural network for the position detection was was separate. As a reminder, kind of what's the difference in training the control policy via reinforcement learning versus the convolutional neural network in another way?

• 14:45Chris Benson?

  So the convolutional neural network is when you're training it, you're trying to recognize certain patterns. And so obviously the patterns here are the various faces of the Rubik's Cube and the different colored faces on each of those cubes and how they're working together. And so recognizing what that is at any given moment. So for cube state is what the CNN, the convolutional neural network would have been doing. Reinforcement learning, though, is the other side. The control policy side where the network has to figure out with this articulated hand. And so it has many different joints and motors that are driving it at different angles. And they have to work in unison to be able to to work with what it's determined to be the state of the Rubik's Cube at any moment in time.

• 15:29Chris Benson?

  So it's not just one motion. It's a whole collection of motions and it it does that through essentially trial and error that the algorithm puts it through to learn the process. And so you could think of the CNN as perception oriented and you could think of the reinforcement learning as what do I do next with my articulated hand to to achieve the next state that I'm trying to get to. So it takes them both. If I also recall, just to throw out one extra thing, they had a type of recurrent neural network called an LSTM, long, short-term memory, which takes into account time, the time sequence between the different observations and the policy-based movements it's doing.

• 16:13Chris Benson?

  You have to have a sequence for that to make any sense because obviously a robotic hand that's trying to do a task has a series of moves it has to do. So it actually takes all three of those to work together to be able to to apply itself to this problem.

• 16:27Daniel Whitenack?

  The reinforcement learning, we won't do a huge deep dive into that because we actually have a couple of other episodes on that already. I would recommend first off, taking a look at at episode 14 from Wochek Zaremba, actually at OpenAI. So he was on the team that helped do this research or did this research. And he goes a little bit more into reinforcement learning in in that episode and kind of compares reinforcement learning to like semi-supervision. So giving the the network a sort of treat, like giving a dog a treat or a reward as it performs actions that get closer to a solution versus the convolutional neural network, which is just supervised.

• 17:10Daniel Whitenack?

  So you have input images that have a labeled position and you're trying to predict that position in a sort of supervised manner. So we'll link our other episodes about reinforcement. Enforcement learning in the show notes.

• 17:23Chris Benson?

  Correct. I guess to summarize that up, the algorithm has what's called an agent, which takes actions to act upon the environment it's in and is rewarded or not depending on the outcome of that. And then adjustments are made based on what happened and it goes around and around in that process.

• 17:52Speaker 00

  What is up practically listeners? We're working with Infinite Red to promote their free AI mini course. It's called AI Demystified. Learn more and enroll at learnai infinited dot red. This free five day mini course is a great introduction to the most important concepts, types and business applications for AI and machine learning. Each day of the course includes a lesson, a quiz and an assignment to submit your learning. And after you've completed the course, you'll also get a certificate of completion for your LinkedIn profile or for your portfolio. If you've been feeling lost in the world of AI and hearing lots of buzzwords, then by the end of this mini course, you'll be able to speak intelligently about AI and machine learning and their practical business applications. Again, this course is completely

• 18:36Speaker 00

  free. Learn more and enroll at learnai dot infinite dot red. Again, learnai dot infinite dot red.

• 18:54Daniel Whitenack?

  All right. So they show some great pictures of their experimental setup, like in the physical world and also in simulation. And particularly in that summarization that we talked about initially, when we got into this, they talk about training in simulation. And I know that you've had some experience in the robotics world before and have run across different scenarios. Why is simulation important, particularly in robotics?

• 19:24Chris Benson?

  Because probably the biggest reason, and it's really, really, really common in robotics to start in simulation, to the point where It's probably that's the case in almost all cases. So in simulation, you can essentially create the circumstances that you're operating in. You're essentially able to simulate the data that applies to the situation and you can have the environment created. And so if you don't have a whole bunch of the ability to apply real world data upfront that may not have been created yet. And you don't have an environment that lends itself to testing. You know, just as a, as a kind of an obvious example, in my own job at Lockheed Martin, one of the things we build are airplanes.

• 20:04Chris Benson?

  And so if you're trying to figure out how an airplane will fly around and you haven't yet built the airplane. It's a real problem because you know, you have to do it in a simulated world where you can try different things out. And that lends itself to the engineering. And so if you kind of take that example and you push it out in the world, you can apply that in a lot of different ways. So it's a much more inexpensive way to start your training out. And depending on your situation, you may be able to do almost all of your training in simulation, which it sounds like they did in this case.

• 20:37Daniel Whitenack?

  Yeah. And in your example, like the airplane example, like it's inexpensive, I guess, because you don't have to manufacture 2000 aircraft with different variations and then see, you know, which 10 of them don't crash. And then you utilize those as what you design off of, right? You're able to kind of go through a lot of different scenarios, a lot of different configurations and in a simulated sense. In the same way, I think OpenAI, like they don't need a thousand robot hands to be manipulating cubes and getting the data off of those to create a kind of vast training data set.

• 21:14Daniel Whitenack?

  What they do is, you know, they create the simulated world using, I think, a lot of technology from like gaming and image rendering and all of that. So like Unity 3D. So they create this virtual hand, virtual cube and position it in various ways to recreate as real of imagery and as real as position.

• 21:41Daniel Whitenack?

  Data as they can without actually having to position a real robot hand into those positions and gather the data off of the hand, which I imagine would take an enormous amount of time and effort. So yeah, I mean, you were kind of getting at the engineering side of things. And even though they aren't engineering a bunch of different hands to collect this data, it seems like the simulation task in itself is actually a pretty big engineering burden.

• 22:11Chris Benson?

  It is, but simulation has been around for a long time. And I know we talked about it. We had our episode where we discussed high performance computing, HPC. And we talked a bit about in that conversation about the simulation world and the requirements. And it's been around for decades. And so, you know, gaming, as you pointed out, also does that. There's a lot of industries that utilize it, automotive. And so trying to get the physics of what you're working on, and all the environmental considerations and, you know, is at least something, it is complex, but it's something that is somewhat known. And it's much less expensive to do it in that simulated world than to do what you just described, the kind

• 22:52Chris Benson?

  of the ludicrous, you know, create a whole bunch of versions of the same thing and figure out which one works and which one doesn't, which would be prohibitively expensive.

• 23:00Daniel Whitenack?

  Right. And could be dangerous, depending on what you're trying to simulate, I guess.

• 23:05Chris Benson?

  Absolutely.

• 23:05Daniel Whitenack?

  In the airplane case. But going back to the Rubik's Cube, I'm thinking back to my friend in my physics class, who was brilliant. And I was in a mechanics class with him where we had to do the simulations of real life mechanics. So classical mechanics, like pendulums and all this stuff, and, and more complicated things that are fairly chaotic, like, you know, multi-weight pendulums with, with bends in them and all of these things. And I remember that that was extremely challenging. And one of the things that they note in the paper is that it is challenging. So

• 23:43Daniel Whitenack?

  You have to, in some sense, use simulated data for these types of problems in robotics, but it is challenging to create simulated environments that capture all of the physics of the real world. And this is where this sort of domain randomization stuff comes in. So they bring up this idea of domain randomization, which is actually a previously introduced idea where in a created environment for a simulation, you introduce some amount of randomness in the renderings of that simulator, or some variability in the simulator based on randomness that you're basically trying to simulate that there's going to

• 24:28Daniel Whitenack?

  be variations in the environment that it sees when you transfer the model that you're training from the simulated world into the real world, because that transfer can be rather difficult. But OpenAI took this a step further. That's my understanding.

• 24:45Chris Benson?

  Yeah, they created this approach, which they called automatic domain randomization, where they systematically created that randomization as part of their training process. And, you know, it was done in simulation, as we've been discussing. And it was interesting in that it was a technique that could increase the ability for the control policy to be able to generalize to the environment that it's in. If they had not done that, for instance, and the articulated hand, robotic hand, had been maneuvering the Rubik's Cube around, and any kind of interference was introduced, you know, going back to your stuffed giraffe

• 25:26Chris Benson?

  comment a little while ago, you know, that could completely throw it off. But if, as part of the training process, you are constantly introducing different types of interference in all sorts of different ways, and as part of its reinforcement learning process for its control policy, it has to learn to cope with each of those forms of interference, then it is better able to generalize once you've completed learning down the road. I was fascinated reading through their white paper on how they approached that. I think it's a great next step.

• 25:57Daniel Whitenack?

  Yeah, and one of the things I think that they were interested in is this idea of handling situations that weren't seen during training. So the idea is, okay, how do we extend domain randomization in a way that creates an even more robust model that will be able to adapt quickly and return to a sort of baseline performance even when things happen that we don't anticipate? And so, one of the key pieces of how they extended the domain randomization was that they didn't just randomize the simulation, they randomized it in increasingly

• 26:39Daniel Whitenack?

  difficult ways. So, one of the things that they mentioned in their blog post is like, they would increase the size or decrease the size of the simulated Rubik's Cube more and more or less and less as the training went on. So, whenever the neural network achieved a certain level of performance, then they would throw something at it even harder, like they would change the size of the Rubik's Cube even more, and kind of continually push it to have to adapt more and more to harder and harder things. And in this way, it kind of, they created what they're kind of calling, or what they're theorizing is something that

• 27:21Daniel Whitenack?

  leads to what they're calling emergent meta-learning. So if you remember, we had this episode with Sheryl Chin from Google, where we talked about AutoML and meta-learning, which are kind of related. Maybe those were in, we talked about them in a slightly different context with, with Sheryl. But here, they're talking about emergent meta-learning, which sounds like this really weird term to me. And it's almost like a term that doesn't mean anything. It's like emergent and it's meta. Very new age sounding there. Yeah, what does that even mean? I'm not sure. So, what do you get, if anything, from, from that?

• 27:59Chris Benson?

  Well, I actually drew an analogy between what they were doing with that and kind of what we as humans do in the sense of as they kept cranking up the difficulty by changing the parameters into something more difficult. It reminded me as I read that about, for instance, teaching my daughter to ride a bike. And, you know, first just learning how to sit on it and pedal with training wheels on and start steering it. And then as she got comfortable with that and, you know, going over curbs and then taking the training wheels off and, you know, having to learn how to do balance and all that. Or even myself, I would like to learn how to juggle just for fun. And recently I picked up some balls and I started with just one.

• 28:40Chris Benson?

  And as I started mastering that, I ramped up the difficulty into two and I'm not to three yet. So, but I think their approach really reminded me of how humans learn along the way. And I thought that was striking since that doesn't always, as much as we like to draw analogy between different AI approaches and humans and brain and such, that doesn't always reflect the reality.

• 29:04Daniel Whitenack?

  It's really interesting. First off, little piece of trivia, I was the unpopular kid in high school who was a juggling troupe. Next time we're together, we'll have to experiment with that. But I do like the analogy that that gives because in juggling or some activity like that, that a human learns, each time you add, like, let's say you add an element that you're juggling, like a new ring or ball or club or whatever it is, it doesn't necessarily take you an exponentially longer time to adapt to that, that new situation, but you've kind of learned how to, you've learned how to learn a new juggling trick. And so you kind of, just kind of bolt it on to your juggling toolkit.

• 29:49Daniel Whitenack?

  It doesn't always carry through like that, but here with meta learning, you know, meta learning itself just means kind of learning to learn, right? And so what they're saying is that by introducing this element of increasingly difficult

• 30:05Daniel Whitenack?

  Domain randomization into the training, then what's emerging from that process is a network that is able to learn to learn as new perturbations come in like this stuffed giraffe or whatever it is. So I like your illustration very much. The last thing that they mentioned, we can just mention really quickly in their paper that maybe needs some definition is this Kooce-Emba algorithm. As you mentioned, the solving of the Rubik's Cube, in other words, how do I need to move the faces of the cube to move towards a solution of the cube?

• 30:45Daniel Whitenack?

  That was actually not a neural network that was deciding those operations. So like the raw operations, like move this block to this position, move this face, rotate it one direction. That was not a neural net that was deciding that, right?

• 31:05Chris Benson?

  Correct. Yeah, that was they're using an algorithm. And this is one of several that I believe exist that where you follow a known sequence of moves and rotations and it will eventually get you where you're trying to get to. So kind of going back to what we said in the very beginning, they were really focused on understanding the state that they were in and articulating the robotic hand to achieve the next state, which this algorithm, which was an already known thing based on this algorithm.

• 31:36Daniel Whitenack?

  Well, talking about the fact that they use the AI models to manipulate the robot hand, but not actually quote unquote, "solve" the Rubik's Cube. That and a few other things created a bit of pushback in terms of the community's reception to that and various people saying strong things. I saw a couple of tweets in this regard about, you know, how like this algorithm that they're using to solve the Rubik's Cube, it's been around for 17 years and it's symbolic. It's not like a neural network. So

• 32:13Chris Benson?

  They were missing the Point in my view.

• 32:15Daniel Whitenack?

  Yeah, and they they also talked about, oh, well, like, the Rubik's Cube is all instrumented with LEDs and sensors and all of these things. It's not just like I threw, for example, I couldn't go into the store and buy a Rubik's Cube and put it in the hand of this robot and then have it do the manipulation because their cube was a special cube or these sorts of things. So this was kind of the pushback was that in their blog post, so if I if I go back to the blog post here, their kind of headline is solving Rubik's Cube with a robotic hand. And they say, trained a pair of

• 32:54Daniel Whitenack?

  neural networks to solve the Rubik's Cube. So the pushback was like, well, did they really do that?

• 33:02Chris Benson?

  So in my view, OpenAI's mistake here was not in what they actually did. It was in the marketing of the blog post and how they titled it. I think what they did was awesome. And a lot more important, if you used a neural network to solve a Rubik's Cube, that might be a cool little project to do, but it's very, very specific and it's not, maybe I'm wrong, but I don't see the applicability off the top of my head to generalize on that. And so what they were focusing on was how do you create the control policy for that articulated hand and learning how to do that better? Because in my view, that is much, much more valuable because you can use that all

• 33:44Chris Benson?

  in so many different use cases out there. And if OpenAI's purpose is to kind of help the rest of us get to the next level, I think that they did exactly the right thing. I think they focused on the right thing. So they should have worded the blog post better, probably. But other than that, I personally am very happy to see them do this work.

• 34:04Daniel Whitenack?

  And I think that even the strongest critics, at least in looking through Twitter, the Twitterverse and interwebs on various things that I found, even the strongest critics, I don't think. Characterize what they did as not impressive, but maybe that it was mischaracterized,

• 34:25Chris Benson?

  Yeah.

• 34:25Daniel Whitenack?

  as you're saying. So yeah, I think that there is, and that kind of got me going down a train of thought. You know, why did people react in such a way, like this sort of way to this article with such vigor? I started thinking like, you know, back a few years ago, or whenever it would be, would people have reacted the same way? And why are people so sensitive to AI hype? Do you have any thoughts on that?

• 34:55Chris Benson?

  Well, I think, I think we're all subjected to quite a lot of AI hype. And I know we've, we've discussed that on numerous episodes in the past. And which is one of the reasons, as an aside, that you and I decided to make this Practical AI, practical, productive and accessible to everyone. Because we were, to some extent, even at the very start, recognizing the overwhelming amount of hype that was then and continues to this day and probably will for a long time to come. So I get that people, that the message needs to be more specific to the actual amazing work that was done. And so, you know, they need to be careful how they're putting their work out there in terms of what the actual impressive thing is that we should focus on.

• 35:38Chris Benson?

  But I get that people feel a little bit burned by the hype, but also, you know, sometimes recognize we're all human. We all make mistakes and, you know, focus on where the good is on this. I would ask people to take a glass half full approach.

• 35:52Daniel Whitenack?

  Yeah.

• 35:52Chris Benson?

  And, you know, let's look at this and go, wow. And now this is something that we can all use going forward and not focus on whether somebody could have done a better job on a title.

• 36:01Daniel Whitenack?

  Yeah, it is kind of the situation where we're in. I guess, you know, this probably relates to some of the discussion around like cancel culture and all of that stuff. And I don't, I don't really want to get into the politics and views on that in this episode. But we, I think, have been burned quite a bit by people just, you know, saying all this stuff about AI or blockchain or quantum this or whatever. When it hasn't really proved true. And I think we do need to be careful about this, in particular, as related to certain things. There's a lot of talk about bias now in AI.

• 36:40Daniel Whitenack?

  And I think that's probably a point that, you know, is not being hyped enough, that we're not concerned about that enough. At least, that's my personal opinion. I saw some tweets saying that Eric Schmidt of Google, the chairman, was saying some things about, "Oh, we don't need to yell that much about bias now in AI." And I think, you know, we need to be practical in how we approach these things, but there's a lot to sift through out there when you start looking up terms like AI and quantum and other things. Even, you know, just this last week, I don't know if you were following, Google made the announcement about, you know, finally achieving quantum supremacy.

• 37:19Chris Benson?

  Oh, yeah, that made me cringe.

• 37:21Daniel Whitenack?

  Yeah. They said this thing would have taken, you know, a classical computer 10,000 years to solve. And then IBM kind of worked on it a bit and solved it in two and a half days with a classical computer. So in some sense, I don't know, this is part of the scientific process in the sense that like, if you claim something, people should be able to, you know, test it and see if you're really, what you're claiming is true or false. It's maybe the difference now is that that testing is like, it's so much more public now. Used to, that would happen in like, private peer review and all of that. Now it's kind of happening on Twitter and all over the internet. So I'm not sure, I'm still kind of parsing through whether parts of that are all good or bad. And

• 38:05Chris Benson?

  I think the fundamental issue there is with the acceleration process that you just described, and the fact that you don't have something, you don't produce a result that gets peer reviewed over a period of weeks or months, and then kind of eventually goes out to the larger world. At this point, you know, AI and these other fields that we're talking about are happening so fast that you kind of have to get your stuff out there as an organization before somebody is likely to, you know, steal your thunder and maybe reveal their results that were very similar to yours before you get there. And so especially with so much of this work being done by industry, I think what I'm about to say holds true even for academia.

• 38:46Chris Benson?

  There's a marketing desire to say, hey, look at this great work we did because people are looking for customers, people are looking for funding. And so there is definitely a, a marketing consideration in doing this advanced scientific work. But the marketing and the science, uh, you need to find the best balance you can and not let your marketing overreach your science. Or you're going to get a lot of backlash on that. So I think it's tough to be in an organization these days where you're trying to do amazing work. And have it validated, but also get the benefit of that work in other domains, such as continued funding and continued customers.

• 39:24Daniel Whitenack?

  Yeah. So I guess one of the best responses to some of the criticism that I saw in the OpenAI stuff was like, hey, look. You know, we shouldn't be solving Rubik's cubes with neural networks because we don't have to. And we've had the solution for a long time and it works fine. But we don't have the solution to this sort of robotic manipulation, which is actually maybe easier for humans, but really hard for computers and AI systems. And so what they tackled was the harder of the problems.

• 39:59Daniel Whitenack?

  And there's certainly other cases of this that, you know, are present in AI, like, you know, detecting and working with sarcasm in text is extremely difficult. Like, or even just sentiment is still a hard problem, just because it's easy for humans to maybe detect that. And like, even like combine text with facial or, or voice with facial expressions and all of that to detect sarcasm or volume or tone or all of those things. And that's really, really tough for a computer to solve.

• 40:32Chris Benson?

  If you think about not having facial expression, the, you know, the way we, we type out emails or texts, for instance. We humans also have a hard time detecting sarcasm when it's there, when it's not, uh, if we don't have other sources other than the text alone.

• 40:47Daniel Whitenack?

  Yeah, which is why I think a lot of people in that space are talking about the sort of multimodal learning where you have maybe text and video or imagery and that sort of thing too, where these people solving those types of problems. Maybe sentiment analysis is something where we're like, oh, well, humans can do that easily. Why do we need to work on computer algorithms to solve this sort of thing? It's part of this effort to really tackle the hard problems that are hard for computers and advance AI more generally versus just kind of solving already solved things just quicker or faster, that sort of train.

• 41:25Chris Benson?

  If you have an option that is less expensive to do it, then take that option. Don't take the AI route. It's, you know, apply AI where it makes sense to apply AI.

• 41:36Daniel Whitenack?

  Amen. All right, well, on this front, we always like to end these fully connected episodes with some learning resources. And in particular, if you're wanting to learn more about reinforcement learning. That was a confusing statement, very meta learning, learning about reinforcement learning. Again, episodes 40 and 14 of this podcast, you might want to take a listen to. But OpenAI uses this OpenAI gem to train their reinforcement learning algorithm. And that is something that you can get your hands on. There's a tutorial that we'll link in our show notes that kind of shows how to use this OpenAI gem.

• 42:18Daniel Whitenack?

  So if that's something you're interested in exploring, then we'll certainly link it in the notes. And there's PyTorch and TensorFlow tutorials as well that deal with reinforcement learning. So there's a bunch of ways that you can get hands on with this sort of thing, even if you don't have a robotic hand handy. There's my pun.

• 42:38Chris Benson?

  You're hand handy? Yeah, and I don't.

• 42:41Daniel Whitenack?

  I don't either. Cool. Well, thanks for going through this discussion with me, Chris. It was a lot of fun. It was. Interesting article. Very interesting in all sorts of different ways, technically, culturally, and reception wise, and all of those things. So yeah, definitely go take a look at the blog post and the paper if you haven't yet, and let us know what you think. You can reach out to us and in our Slack channel, which you can find if you go to changelog.com/community, or reach out to us on our LinkedIn page. We're happy to get good feedback, whether that's positive or negative things, but also to hear what your thoughts are on potentially controversial and interesting topics like

• 43:25Daniel Whitenack?

  this. So, reach out. And yeah, I think that's all we've got for today. It was great to talk to you, Chris. It was a good one. Talk to you soon, Daniel.

• 43:35Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 44:18Speaker 00

  Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.