Achieving provably beneficial, human-compatible AI

AI legend Stuart Russell, the Berkeley professor who leads the Center for Human-Compatible AI, joins Chris to share his insights into the future of artificial intelligence. Stuart is the author of Human Compatible, and the upcoming 4th edition of his perennial classic Artificial Intelligence: A Modern Approach, which is widely regarded as the standard text on AI. After exposing the shortcomings inherent in deep learning, Stuart goes on to propose a new practitioner approach to creating AI that avoids harmful unintended consequences, and offers a path forward towards a future in which humans can safely rely of provably beneficial AI. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. Pluralsight https://pluralsight.com/changelog – Stay home. Skill up. For free. Pluralsight is totally free for the entire month of April! With over 7,000 courses from experts in software development, security, cloud and data there’s never been a better time to skill up. Learn more and get started at pluralsight.com/changelog https://pluralsight.com/changelogFastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Stuart Russell – Website https://people.eecs.berkeley.edu/~russell, LinkedIn https://www.linkedin.com/in/stuart-russell-3a924228Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Stuart Russell on Wikipedia https://en.wikipedia.org/wiki/Stuart_J._RussellStuart Russell’s TED Speaker profile https://www.ted.com/speakers/stuart_russellCenter for Human-Compatible AI https://humancompatible.aiStuart Russell on how to make AI ‘human-compatible’ https://techcrunch.com/2020/03/20/stuart-russell-on-how-to-make-ai-human-compatibleAI could be a disaster for humanity. A top computer scientist thinks he has the solution. https://www.vox.com/future-perfect/2019/10/26/20932289/ai-stuart-russell-human-compatibleLeading AI Luminary Has An Idea To Ensure Humans Remain In Control https://www.forbes.com/sites/peterhigh/2020/01/13/leading-ai-luminary-has-an-idea-to-ensure-humans-remain-in-controlBooks “Artificial Intelligence” by Stuart Russell and Peter Norvig https://www.amazon.com/dp/0134610997“Human Compatible” by Stuart Russell https://www.amazon.com/dp/0525558632Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I don't think deep learning evolves into AGI. So AGI, artificial general intelligence, is not going to be reached by just having bigger deep learning networks and more data. AGI and human intelligence require fundamental capabilities that are just not present in deep learning technology as we currently understand it. Deep learning systems don't know anything. They can't reason. They can't accumulate knowledge. They can't apply what they've learned in one context to solve problems in another context, et cetera, et cetera. Right? And these are just elementary things that humans do all the time.

• 0:35Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com and we're hosted on Leno cloud servers. Head to leno.com/changelog.

• 0:53Speaker 01

  This episode is brought to you by Digital Ocean. Digital Ocean's developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, TM accounts, worldwide availability with a 99.99 uptime SLA and 24/7 365 world-class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure and monitor your cloud environments. Head to do.co/changelog to get started with a $100 credit. Again, do.co/changelog.

• 1:34Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM. Okay, take it away, Chris.

• 2:01Chris Benson?

  Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a Principal AI Strategist at Lockheed Martin. Normally, listeners would know that Daniel Whitenack, my co-host, would be with me. He is unavailable today. He's out sick. And so I have the pleasure of introducing our guest today, who is a legend in the AI field. This with me today is Stuart Russell, who is a professor of computer science at University of California, Berkeley, and a holder of the Smith, is it Zade Chair, if I'm getting that correct?

• 2:35Speaker 00

  Smith Zade.

• 2:36Chris Benson?

  Zade, I apologize. And also, if the name sounds familiar, he is the author of the standard book on artificial intelligence, which most practitioners in the field will be familiar with, as well as a recent book for a general audience, which is called Human Compatible: Artificial Intelligence and the Problem of Control. Stuart, thank you very much for coming on the show.

• 2:57Speaker 00

  Pleasure.

• 2:58Chris Benson?

  I know I barely touched on it. I know you have been in this field for decades. If you could tell us just a little bit more about your background before we get fully launching.

• 3:07Speaker 00

  Yeah, as you say, it's been quite a long time. I first started doing AI when I was in high school because I got a programmable calculator and I thought I could make it really intelligent. But it turned out that it only allowed 36 keystrokes in the program. So I didn't get very far with that with that attempt. But and then I got to use a giant computer at Imperial College. So I wrote a chess program. That was my first serious AI program. I did my PhD at Stanford. I joined Berkeley in 86, so it's 34 years teaching at Berkeley. And it's been a pretty interesting time. And it's, you know, most people would say now is maybe the most exciting time to be doing AI

• 3:49Speaker 00

  because there's so much progress. We've been able to solve or nearly solve some of the major open problems of the field, you know, speech recognition, machine translation, certain parts of computer visions, particularly recognizing objects and images. All of those things now work pretty well. So then then we can roll out. All those techniques into the real world and do cool things like driving cars and everything like that. So it's lots of fun and we're all very busy.

• 4:14Chris Benson?

  So I guess, you know, given that you've seen so much of the evolution of this field over time, could you talk a little bit about what the field was like when you came into it? And what technologies were prevalent? And tell us a bit about the evolution of the field over the years, all the way into the current, you know, what's certainly taken off these last few years.

• 4:35Speaker 00

  Sure, yeah, so in the early years, so I guess I would say I started probably 1975. The focus was almost exclusively on problem solving, game playing, planning, logical reasoning. So everything was deterministic. So we assumed that we could give the computer perfect knowledge of the problem, a perfectly stated goal, and then it would come up with a guaranteed solution. Whether it was proving a theorem or, you know, finding a checkmate or a thought, you know, coming up with a plan to deliver a bunch of parcels to a bunch of recipients or whatever it might be. And in the eighties, we had the big expert system boom.

• 5:16Speaker 00

  So initially, logical rule based system, so encoding expertise in logical rules. Sometimes we now call it business rules or business intelligence. That's a, that's a phrase that they use because the term expert system fell out of favor. But in the eighties, that was a really, you know, a big, exciting, you know, hype bubble, just like today.

• 5:37Chris Benson?

  Sure.

• 5:38Speaker 00

  And the beginnings of handling uncertainty, because we wanted to, to make expert systems that did things like medical diagnosis, where there are no hard and fast rules. You have, you have to take the evidence and combine it in, you know, to get some kind of soft prediction, you might say. But that technology largely failed in the real world.

• 6:00Speaker 00

  And complicated reasons for it, but I think basically was not doing uncertain reasoning correctly. And so as you tried to build bigger systems for real problems, the whole thing would kind of fall apart. Every time you add a new rule, the other ones would start messing things up and you know, the interactions would cause wrong answers to come out. And companies very quickly stopped investing in this and we had what we call the AI winter. Where, you know, I think my AI course was down to about 25 students in 1990. It's up at about 900 right now. I imagine it is. And that's only because we're not, you know, the fire marshal won't let us have any more people in the class. Would probably be, you know, 12, 1500 if let everyone in.

• 6:45Speaker 00

  So what happened next in AI actually was that rigorous probabilistic methods took over within the field, mainly from the work of Yudha Pearl, starting in the mid eighties. And then machine learning had a renaissance, reinforcement learning had a renaissance. And so from the late eighties until around 2011 or so, there was pretty solid technical research progress using probability statistics, connections to operations research, control theory, became a very mathematical field. Some of the techniques worked pretty well, you know, so speech recognition became reasonably practical.

• 7:30Speaker 00

  And the first self-driving cars were operating long before the present day and doing so reasonably successfully. You know, there were big applications of planning, there were lots of diagnostic systems and so on. So it was relatively successful, but it wasn't really until deep learning happened around 2011, 2012, that it really hit the big time in terms of media coverage and excitement and so on. So it was deep learning that. Enabled us to, for example, beat the human world champion at Go. A combination of deep learning with reinforcement learning methods and game playing techniques that had been around for decades and decades.

• 8:13Speaker 00

  But deep learning was this extra ingredient that let the system somehow recognize patterns on the Go board that allowed it to beat the human world champion. So, so now we're in a position where, you know, as I've mentioned, thousands of students want to take AI courses. There are thousands of startup companies, all the big companies have major AI divisions and are using AI in hundreds of applications throughout their businesses. So it's fun, it's exciting. Maybe there's some hype going on too.

• 8:45Chris Benson?

  There might be a little bit. So going through that, it really begs the question for me, and I'm going to throw a question at you, I imagine it will be like, but you're looking at all these different types of artificial intelligence, different things that are being labeled as artificial intelligence, you know, from the symbolic logic days through expert systems, all the way up through today's deep learning and other associated technologies. And there are obviously vastly different underpinnings in terms of what they can do and how you arrive at them. How do you define artificial intelligence? What does the term mean to you as the person who has literally written the textbook on the subject? And how has that changed and evolved over those years?

• 9:28Speaker 00

  So it's a good question. I mean, all of the above, all the things you mentioned, this is all artificial intelligence because it's all in the service of creating machines that can act intelligently, which means really choosing actions that can be expected to achieve their objectives. You know, if you're a self-driving car, the objective is to get to the airport safely, legally, comfortably. And so in order to do that, you need perception, but you also need symbolic planning to choose a route. You need probabilistic forecasting to deal with traffic delays and maybe have a backup route just in case. And you need speech recognition in order to interface with the passenger, et cetera, et cetera.

• 10:13Speaker 00

  So, you know, if you want to build a system that's going to help a mathematician. You can't just throw a bunch of theorems and proofs into a deep learning system and say, "Here, you know, learn how to do math." You actually have to have symbolic reasoning capability theorem proven, you know, which the underlying technology of that is symbolic logic and not statistical learning. So it all depends what you want to do. The overriding model, which I think pervades not just AI, but a lot of other disciplines, control theory, operations research, economics, statistics, they all have this model, which is that we specify an objective and then the machine finds some optimal solution or

• 10:56Speaker 00

  a way of achieving the objective, the best solution. And so actually, what the book is about, the Human Compatible book, is basically saying that model is really a terrible model. Now, unfortunately, the first three editions of my textbook actually kind of solidified that model and said, "Okay, here's how we understand, you know, here's how we pull everything in AI together into a single conceptual framework." And you can see all of the different kinds of AI research, it's of different facets, different ways of looking at that same underlying conceptual framework. And the reason I think it's a terrible model is not a new thing, right?

• 11:38Speaker 00

  It's something we've known for thousands of years, which is that we cannot specify our objectives completely incorrectly. And, you know, if you look at the legend of King Midas, he specifies his objective, "I want everything I touch to turn to gold." You know, the gods, or you could say the AI system gives him the objective exactly as he specifies. And then, of course, his food and his drink and his family all turn to gold and he dies. Misery, starvation.

• 12:04Chris Benson?

  Unexpected consequences.

• 12:06Speaker 00

  Right, you know, that's the thing, right? It's always unanticipated consequences, accidental side effects, collateral damage, externalities is what the economists call it, but it's a pervasive problem. We've known about it for a long time. You know, that's why your third wish is always, "Please undo the first two wishes, because I've ruined everything." Right? So the Human Compatible book basically says, "Okay, we have to throw away that model." Because, you know, up to now, it hasn't been that bad, because, you know, first of all, most of our AI systems were toys, they were in the lab. You know, we were doing demos.

• 12:44Chris Benson?

  It wasn't out there an industry at the level it's at now.

• 12:47Speaker 00

  Until recently, on a global scale, but now it is, right? So now, you know, you've got the content selection algorithms from all the different social media platforms. And those algorithms are, they're machine learning algorithms, but they decide what billions of people spend hours every day reading and watching, right? So in terms of their actual direct, in terms of like, you take the number of people times the amount of time, they are more powerful than anything that's ever existed in the history of the human race, by far, by far, right? I mean, you know, you think Stalin was powerful, but he got to speak to his people like, you know,

• 13:31Speaker 00

  maybe half an hour every month or something, right?

• 13:33Chris Benson?

  Yes.

• 13:33Speaker 00

  These algorithms are speaking to 50 times more people for hours every day.

• 13:39Chris Benson?

  A largely oblivious audience for the most part that's acting on them.

• 13:44Speaker 00

  Yeah, so the audience doesn't know what the algorithms are doing or what they're trying to do. The algorithms are trying to maximize their objective, which is click through or engagement or something like that. And in the course of doing that, rather than just send you what's interesting, they actually modify you into someone who's more predictable from their point of view, because the more predictable you are, the more money they can make off you. And so whatever you start out as, they change you and mold you into a predictable clicker. And so that's what they've done. And I think, you know, most people would say that the results have been pretty disastrous on the whole.

• 14:24Chris Benson?

  So I want to ask you, there's a particular remark you make in the book that I want to ask you. And I think you're already kind of going down the path on this to some degree, but you say, "We must plan for the possibility that machines will far exceed the human capacity for decision making in the real world." And I think that you've started to address some of the challenges. Could you give us a little bit more of a holistic perspective on what? I mean, that statement has a lot in it right there. Can you talk a little bit about what the implications of that is?

• 14:54Speaker 00

  So, yeah, so let's give an example, right? Suppose that a few years or decades down the line, you're, you know, the CEO of... An IT company or your solar power company or whatever it might be. And you want your company to be more successful. So you you engage an AI system and you give it the objective of, let's say, you know, maximizing the profits or the revenues of my corporation. And because that system is far more capable than humans are, right? It devises plans that are more successful than the all the competitors can be.

• 15:33Speaker 00

  And so that corporation in the interests of maximizing revenues gradually takes over larger and larger portions of the world economy. You know, and if it's not properly designed, right? If that was the objective, you know, wherever it was feasible, it might end up using slave labor, for example, in order to maximize profits and so on and so forth. Right? I mean, you can imagine all the ways that corporations have abused humanity in the past, and now we've got one that's much more capable than human beings are. You know, some people actually argue that this is already happening, not from AI, but

• 16:12Speaker 00

  from corporations that optimize profit at the expense of everything else. So, for example, at the expense of the climate, the fossil fuel industry has optimized its profits by sort of multi-decade misinformation strategy that's actually outwitted the human race. Right? And so even though the vast majority of experts and economists and scientists say, oh, you know, we need to have a carbon text, we need to do this, we need to do that, we aren't doing any of it.

• 16:46Speaker 00

  Right? We're just talking about it. And so effectively, the fossil fuel industry has defeated the human race by superior pursuit of a fixed objective. So it would get much worse than that when AI systems are able to invent and carry out these kinds of strategies. So that's even, you know, within the realm of things that we currently understand. Right, that you could have corporate strategy, you could enslave people, you could do this, you could that. But AI systems will come up with things we don't understand.

• 17:16Chris Benson?

  Sure.

• 17:16Speaker 00

  And we, you know, the whole human race could be collateral damage if we don't know how to control the systems that we create. And so far, there's no examples of a dumb species controlling a more intelligent species forever.

• 17:33Chris Benson?

  I totally agree with that. So for my own employer, I'm actually the person leading on AI ethics. So AI ethics is a huge passion of mine. And obviously you've raised some pretty big concerns there. And I'm taking a little bit of a tangent. I wasn't really expecting to go down this path. But I am curious how you envision the role of AI ethics in our society and in the world at large, given everything that you just said. I mean, it clearly the potential for consequences that we did not envision, that we did not plan on is fairly significant, especially as as technology evolves. Do you have any thought?

• 18:09Speaker 00

  In a sense, I wish it wasn't called AI ethics.

• 18:11Chris Benson?

  Okay, what should it be called?

• 18:13Speaker 00

  Well, so I mean, let's give you an analogy, right? So the nuclear engineers who make sure that nuclear power stations don't explode like Chernobyl. Are they ethicists? Would you say that's a, you know, nuclear ethics issue? No, I mean, it's just common sense that you don't want your nuclear power station to explode. It's common sense that you want your AI systems to remain under human control.

• 18:36Chris Benson?

  Sure.

• 18:37Speaker 00

  But at the moment, under the standard model, they won't remain under human control.

• 18:44Chris Benson?

  And would you talk us through what that implies? When you say it won't, and I'm going to set it up in this way, and that recognizing, and it's funny how many people I talk to have different perspectives from what I think you're about to go. But given the evolution that we've seen over over time and the rapid evolution we're seeing in deep learning and whatever follows coming up, that potential for loss of human control. Where does that come from? Why is it inevitable in your view?

• 19:10Speaker 00

  I don't want to say inevitable. If we persist with AI within the standard model.

• 19:16Chris Benson?

  Okay.

• 19:17Speaker 00

  Right, where we fix an objective. Because when you, when you fix an objective, you're basically telling the system, whatever course of action optimizes that objective is the correct thing to do.

• 19:31Speaker 00

  And in particular, for example, anything that imperils the success of the objective has to be prevented. Well, what might imperil the success of the objective? Well, being switched off.

• 19:48Chris Benson?

  Sure.

• 19:48Speaker 00

  So by giving a system a fixed objective, you've now given it an incentive to protect itself from any attempt to interfere with the objective, from any attempt to switch it off.

• 20:00Chris Benson?

  So is a very typical argument I hear people make. If you kind of go back to like, you know, Asimov's three rules for robotics, you know, and the idea that you can just, in a non-probabilistic way, just definitively say, you know, you can't hurt people, that kind of thing, as an underlying thing. Given the fact that you have this ever-increasing capability in the AI realm, would it be fair to say that's not a realistic perspective? That the intelligence, that AI would fundamentally look to circumvent? Or how do you see that?

• 20:31Speaker 00

  Yeah, so Asimov's laws, as you say, don't take into account the probabilistic perspective. They don't allow for uncertainty. But of course, in the real world, there are always risks. So, you know, an Asimovian self-driving car would simply stay in the garage. It would say, I'm sorry, you know, the first law does not allow me to leave the garage because that would expose you to risk of injury or death. So, sorry, we're not going anywhere.

• 20:56Chris Benson?

  I love that. That's very funny, actually. That's true.

• 20:59Speaker 00

  And if you were out for a walk, it would run around with an umbrella in case a photon from the sun, you know, landed on your skin and maybe initiated a little melanoma or something like, you know, there's a chance that could happen, so we have to protect it. So, you know, in any kind of real world situation, there are trade-offs. But one of the things that Asimov's laws do is they make a start on saying what it is that humans want, right? One of the things, we don't want to be harmed. We don't want to be physically injured. And that's a start because, you know, for example, none of the self-driving cars that are out there right now know that people don't want to be injured.

• 21:36Chris Benson?

  Understood.

• 21:37Speaker 00

  Right. They have built-in rules that say, well, if there's a pedestrian in front of you and you're going forward, stop, right? And if there's a, if you're lucky, they have another rule that says if there's a pedestrian behind you and you're going backwards, stop.

• 21:49Chris Benson?

  Yeah.

• 21:49Speaker 00

  Right. But they don't know why. They don't know that if you run into a person, you can injure or kill them, and they don't know that the person doesn't want to be injured or killed. And it's that lack of knowledge actually that makes them very brittle. Because when they get into situations they haven't been prepared for, they haven't the faintest idea what to do. Because they don't know which course of action is good and which is bad. So the solution that the book proposes actually is to say, look, it doesn't matter how much the human tells you that they want this or they don't want that. There's always going to be residual uncertainty about other preferences the human may have.

• 22:31Speaker 00

  So if the human says, I'd like a cup of coffee, that's not your life's mission.

• 22:38Speaker 00

  Right? You know, the robot could say, well, you know, the coffee in this hotel is fifteen bucks a cup. Are you sure you want a cup of coffee? Right? Because this machine is uncertain about your trade-off between coffee and money. Right? If you're miles from the nearest coffee, you know, the robot might not be sure, you know, do you want to wait two hours for this coffee? Is it okay if I like trundle off across the desert to the nearest Starbucks and come back two hours later or two, or two weeks later with a cup of coffee, you know? So it would be reasonable again to ask permission. And, you know, if, if you give it a more important goal, like restoring carbon dioxide levels to pre-industrial concentrations, if that was the only objective, well, you know, one

• 23:22Speaker 00

  very straightforward solution is just to get rid of all the people.

• 23:25Chris Benson?

  Understood.

• 23:25Speaker 00

  Because they are the ones who are producing the carbon dioxide. And, and then you might say, oh, well, I didn't, I didn't mean that. All right. So wish number two, restore the carbon dioxide, but don't kill anybody.

• 23:37Speaker 00

  And then the school system says, fine, fine, no problem. We'll just have a multi-decade social media campaign to convince people not to have children. And then the human race will gradually die out. And then carbon dioxide levels will be restored. And that's great. So what I'm really proposing in the book is actually throw away the standard model or only use the standard model in very restricted circumstances. But in general, have a new model where the objectives are in the human. And the machine knows that it doesn't know what they are. Its job is to try to satisfy them, but it knows that it doesn't know what they are.

• 24:20Speaker 00

  And when you design things that way, and you actually solve that problem. So you, you can have an algorithm that for that problem specification decides what the machine is going to do. That algorithm produces behaviors that seem to be what we want, namely asking permission. Like, you know, is it okay if I turn the oceans into sulfuric acid in order to restore carbon dioxide levels? And you say, no.

• 24:46Chris Benson?

  Not so much.

• 24:47Speaker 00

  We like those little fishies. Don't turn the oceans into sulfuric acid. Right? So we will ask permission. You know, it'll even allow itself to be switched off. So rather than try to protect itself and take steps to prevent interference, it actually welcomes interference because interference by a human is a way of gaining information.

• 25:13Speaker 01

  What up nerds? I've got some pretty awesome news to share with you. Pluralsight is totally free for the entire month of April. I'm not kidding. Seriously. Head to pluralsight.com/changelog and skill up while you stay at home. For the entire month of April, you'll get access to over 7,000 courses from experts in software development, security, cloud and data. There's never been a better time to skill up. Head to pluralsight.com/changelog. Again, pluralsight.com/changelog.

• 25:56Chris Benson?

  So if you would take us a little bit farther into your, the new model that you're proposing to replace the standard model, and maybe along the way, one of the things I was wanting to ask as you were discussing that, is if you could do that also in the context of, as we're looking at AI in the deep learning context of today, anticipating wherever we may be going in the future, and with the idea that people talk about, about AGI, which is artificial general intelligence, which presumably would change the nature of what AI is, maybe, and maybe distinguish how your new proposal would work, kind of in both worlds. I mean, if you were, even today, as we're looking at exceeding human

• 26:41Chris Benson?

  capability by, if you have a complex set of tasks, even now, we can take the models that we have and have mini models, each one addressing kind of a narrow scope, and working together, and they can far outperform what humans could do in a similar complex task. And with the idea also of having AGI, where we have models that are, for lack of a better word, more capable in themselves, maybe eventually aware, I don't know. If you could kind of talk about what your proposal looks like in that evolving world, I'd love to know.

• 27:14Speaker 00

  Sure. So first of all, I should point out that I don't think deep learning evolves into AGI.

• 27:21Chris Benson?

  Okay.

• 27:22Speaker 00

  Right? So AGI, artificial general intelligence, is not going to be reached by just having bigger deep learning networks and more data. AGI and human intelligence require fundamental capabilities that are just not present in deep learning technology as we currently understand it. So deep learning systems don't know anything.

• 27:45Speaker 00

  They can't reason and they can't accumulate knowledge. They can't apply what they learned in one context to solve problems in another context, et cetera, et cetera. And these are just elementary things that humans do all the time.

• 28:01Chris Benson?

  A bit of a stepping stone technology of the moment in a sense, the deep learning?

• 28:05Speaker 00

  Well, I think deep learning is one of the pieces, but you know, so is symbolic logic. So is probabilistic reasoning. So is sequential decision making techniques, planning, hierarchical reinforcement learning, probabilistic programming, et cetera, et cetera. So there, you know, there's lots of pieces of the puzzle, some of which have been lying around for a long time. Deep learning is just the newest, shiniest one. So everyone's sort of, ooh, look, you know. But, you know, in the eighties, people were going, oh, look, expert systems, you know? And similar claims were being made, right? That we just, you know, if you just like scale up the number of rules by a factor of 500, you know, and you had like learned people, you know, making quantitative estimates like,

• 28:48Speaker 00

  oh yeah, we would need about 500,000 rules to, to, to manage a military campaign and stuff like this, like just complete drivel. Yeah. And there's a lot of drivel being talked now about, about deep learning. But, okay, so within the context of just straightforward supervised learning, let's say for image classification.

• 29:10Chris Benson?

  Okay.

• 29:10Speaker 00

  Right. So how does it work? Well, we have training data and then we have deep learning, which is basically a giant tunable circuit with billions of tunable connection strengths, like little tiny volume controls. And we just tweak all those volume controls in this huge circuit until the thing that comes out the other end is the correct classification of an image. So in, in statistical learning, what you do is you, you're supposed to specify the loss function, which says if you classify an object of type A as an object of type B. So let's say you classify a picture of a dog and you classify it as a cat. How bad is that? Right?

• 29:53Speaker 00

  So almost everybody in this business uses what we call a uniform loss function, which means that they say every error is equally bad, because that's how the competitions work. They penalize you for the number of errors you make, not how bad the errors are. Right, so, you know, for example, in ImageNet, there are two categories of dog. One, well, there's a hundred and something categories, but two of them are the Norfolk terrier and the Norwich terrier, right? And these are practically identical. In fact, they weren't even recognized as separate breeds of dog until 1960 something. You know, and there's like slight difference in the shape of the ear. And it's like, okay, who cares? I'm sure the Norfolk terriers are not going to be that upset if you call them Norwich terriers.

• 30:38Speaker 00

  You know, we Norwich is in Norfolk anyway. They'll lick your face either way. Right. So clearly, that kind of error is relatively cheap. Whereas classifying a human as a gorilla, as Google found out, is really expensive, like in the billions of dollars of, you know, trashing your goodwill of your corporation and its global reputation for being fair and idealistic and all the rest of it, right? You know, it was, yeah, I'm sure it was sort of an innocent error coming from just using a uniform loss function. Sure. And, but if they had thought about it, they would have said, oh, of course, our loss function is not uniform. Oh, then what is it? Oh, don't know, we haven't thought about it and we're not sure.

• 31:21Speaker 00

  And in fact, you know, if you've got typically in ImageNet, there's like 20 something thousand different categories of object. So your loss function is a matrix with 400 and odd million entries. And do you know what they are? No, no one knows what they are. So you have an uncertain objective. You don't know what the objective is you're supposed to be optimizing. And when you formulate the problem that way, right? First of all, you'd have to say, okay, well, how do we specify a probability distribution over these 400 million entry matrices, these giant tables? And now I've got to say, okay, what's the probability of each possible table of 400 million numbers? Well, that probability distribution is itself massively complex object to specify.

• 32:10Speaker 00

  And no one has ever figured out even how to write it down, how to structure that probability distribution because it clearly it has lots of structure, right? The the costs of misclassifying each breed of dog as a cat is probably about the same. I think all dogs are equally upset to be called cats. But if, you know, if you classify a bus as an insect, maybe that's a more embarrassing mistake to make and so on. So you could imagine that this there's lots of structure in this in this matrix. And, you know, the structure partly reflects a taxonomic hierarchy of objects and how we arrange them into categories. So you could do a whole PhD thesis just on that part of the problem.

• 32:54Speaker 00

  And now there's also, well, what does the algorithm look like? Right? Well, if it doesn't know the loss function and it has the opportunity to find out more,

• 33:04Speaker 00

  for example, by asking the user, you know, is it worse to call a cat a dog or to call an apple an orange? And sometimes the algorithm would say, I'm not going to classify that image. Right? That's too dangerous. So I'm just not going to make a guess as to what it is. So you immediately see that just the nature of supervised learning would change considerably.

• 33:24Chris Benson?

  Yes.

• 33:24Speaker 00

  If you allow for uncertainty about the the underlying objective. And then with AGI, we don't yet know exactly how to build AGI. I mean, there are a bunch of unsolved major conceptual problems that we have to figure out. But I think the basic answer is that if you formulate AGI within this new model,

• 33:48Speaker 00

  the key property of the new model is that the better the AI solves the problem, the better the outcome is for human beings. Because it means that the AI system does a better job of finding out what what it is you want and does a better job of achieving it.

• 34:04Chris Benson?

  And so you were talking a moment ago about kind of applying control in the new model that that you're proposing as we move into AGI. Would you pick that train of thought right up where you left it there?

• 34:16Speaker 00

  Yeah. So with AGI, if we formulate it in the new model, the the key property is that the the smarter the AI, the better the outcome for humans. Because the AI system will be able to better interpret our behavior as evidence of our underlying preferences. So this is the nature of information flow between the human and the machine about what the human's objectives and preferences are, is that everything the human does reveals evidence for our underlying preferences. So the AI system observing us, observing our whole history, observing everything we've ever written, is able to infer from that something about what we want as individuals, as a species, and so on.

• 35:01Speaker 00

  And so the better the AI system, the better job it'll do of that, and the better it'll be able to achieve those objectives.

• 35:07Chris Benson?

  When you say that, just to clarify, it sounds like you're going into unsupervised learning, where it just has kind of the wealth of human knowledge and what humans have done. Are you strictly leaving it for, in this new approach, are you leaving it for the algorithm that you're training to figure that out? Are you specifying it as the practitioner? Do you see this as kind of, at some point, maybe leaving kind of today's deep learning behind and taking a different approach mathematically? How does that look going into the future if everyone adopts this?

• 35:37Speaker 00

  Well, yeah, so you often see this claim that there's supervised learning, unsupervised learning, and then, well, that, I mean, logically, if those were correct, then supervised and unsupervised, that constitutes a complete coverage of all learning, right?

• 35:53Chris Benson?

  Right.

• 35:53Speaker 00

  You know, A and not A. But then they say, oh, and there's reinforcement learning as well. But actually, there's not. There's other kinds of learning, too. And this is related to something we call inverse reinforcement learning. And inverse reinforcement learning is basically, well, so let me first say what reinforcement learning is, right? Reinforcement learning is the human specifies the reward to the machine, and then the machine learns how to optimally produce reward. Right? So the machine says, okay, this is, I'm going to give you one point when you win the game, and I'm going to give you zero when you lose, and then the machine learns to get one point more often than not. So inverse reinforcement learning is the other way around.

• 36:35Speaker 00

  The machine is observing, let's say, the human, and trying to figure out what is the reward function that this human is optimizing. Right? And we came up with it, actually, when I was collaborating with some biologists.

• 36:52Speaker 00

  And we were trying to figure out how could we apply reinforcement learning to understand animal locomotion. So cockroaches and flies and other creepy crawlies and so on. And it struck me that actually, we can't apply reinforcement learning to simulate animal, to create a simulated insect or whatever, because we don't know what the reward function is.

• 37:15Chris Benson?

  Right.

• 37:16Speaker 00

  Right? So then I said, oh, well, why don't we watch them walking and figure out what reward function are they optimizing with their particular choice of how to locomote? Right? Because I don't know if you've ever seen the Monty Python silly walk sketch?

• 37:31Chris Benson?

  Yes, I have.

• 37:32Speaker 00

  Your listeners may want to check that out on the web. So John Cleese demonstrates that there are many other ways you could walk besides the usual one. So we choose the usual way of walking because it does something good for us, whether it's energy efficient or stable, avoids falling over, whatever it might be, it's optimizing something. And so the idea of inverse reinforcement learning is observe the behavior and figure out what is being optimized by this behavior.

• 38:00Chris Benson?

  Okay.

• 38:01Speaker 00

  And so this approach, the new model, is a sort of a generalization of that idea, because it's generalized in the sense that the human is not just being passively observed, doing whatever human thing, but the human is sort of an active participant. For example, if the human solves their half of this problem, they will actively teach the robot about their preferences, including saying things like, "I would like a cup of coffee." Right? That's conveying preference information to the robot. It's not an order. It's sort of just factual evidence about my state of mind, and the robot can interpret it as it wishes.

• 38:44Speaker 00

  So when you solve this kind of problem, it's what economists call a game, which just means a decision problem with more than one decision-making entity. So you can imagine one human and one machine, or lots of humans, lots of machines. So you can solve that problem mathematically, and then you just look at the behaviors that the machine and the human engage in when they solve this problem. And indeed, the human teaches the machine, and the machine does things. It asks permission, it allows itself to be switched off, and so on. So you get very different behaviors than you do in the standard model of AI. And so I think I'm reasonably optimistic that, in fact, it shouldn't matter how intelligent the AI system is, things will still go well.

• 39:29Speaker 00

  And in the old model, the more intelligent the machine, the worse the outcome for people, because the machine would find some way of messing with the world to achieve the objective that you said, and mess with the things you forgot to mention that you care about.

• 39:47Chris Benson?

  So I'm kind of curious, as you take this and you're looking, we've hit so many different areas, and so I'm trying to tie it together. You look into the future at this point, having come as far through this field as it's evolved and changed over the years. Where do you see it going, especially with control in mind? And as you've talked about how the current standard model can lead us awry, then if you are a practitioner and you're out in industry and you're trying to do the things that your organization wants you to do, how do you apply your new model? And as you look out, what do you think we're going to be doing in terms of what types

• 40:27Chris Benson?

  of models, what is AI kind of evolving into if you're looking out five years or 10 years? And we're learning these lessons that you're teaching us in this capacity. What does the near, mid, and a little bit farther out look to you at this point?

• 40:41Speaker 00

  Interesting. So first of all, there's going to be a little bit of pushing and shoving, right?

• 40:47Chris Benson?

  I would imagine.

• 40:49Speaker 00

  The AI community that's grown up with the standard model and learned it from the textbook is going to keep pushing ahead with their...

• 40:59Speaker 00

  You know, solving the technical problems that they're that they're solving within the standard model and and they have to be dragged kicking and screaming into this new way of doing things. So so partly means we, you know, are my research group. There are, you know, maybe a dozen other research groups around the world working in this new framework now. We have to provide the technical solutions. We have to provide the new algorithms that behave according to the the different principles. And I think we can do that if we can do that in practical settings, whether it's, you know, recommendation systems, content selection for social media, intelligent personal assistants, then I think that will have a significant effect. People will say, oh, now I get it.

• 41:45Speaker 00

  In fact, no, they won't say, now I get it. They'll say, oh, I always thought that way.

• 41:48Chris Benson?

  Of course.

• 41:50Speaker 00

  So there were, you know, there won't be a sort of a, you know, a moment of of capitulation. There'll be a just a gradual realization that, of course, this is what they've always thought. And it makes sense, right?

• 42:01Chris Benson?

  It does make sense. Do you think that as they adopt this, as I'm thinking about what you're saying here? And and you've already mentioned that the poorly named idea of AI ethics, you know, in terms of how do we prevent those? How does that come together? I mean, because there's the algorithmic side. There's the new algorithms and, you know, where you are going out into the future and you're implementing inverse reinforcement learning and and and it's working for you technically. And you're also trying to say, we want to ensure that the outcomes are beneficial, uh, and certainly to the human involved. How does all of that come together? Because right now, you know, as I look at people, there are people that are doing kind of the the outcome, the ethical

• 42:44Chris Benson?

  can, you know, concern there. There are people that are strictly algorithmically focused in terms of solving problems. And yet, if I'm understanding you correctly, you need to be able to to fuse all these together, it sounds like, so that that works.

• 42:57Speaker 00

  Yeah, I think because I think the last thing you want, and you've probably experienced this yourself, is that AI ethics people leaning over the shoulder of the AI practitioners and wagging their finger.

• 43:06Chris Benson?

  Yeah.

• 43:07Speaker 00

  And saying, no, no, you're a bad person, right? It doesn't work. So what we have to do actually is to get people to understand that this is just how you do AI. Right, you know, when civil engineers design bridges, there's not a bunch of bridge designers and then a bunch of ethicists saying, oh, by the way, you have to make sure it doesn't fall over. Right, it's just, of course it, of course it's not supposed to fall over. It wouldn't be a bridge if it just fell over, right? You know, and similarly, you know, nuclear engineers who design nuclear power stations, there isn't, you know, another discipline of people who care about safety. And then the nuclear engineers don't care about safety and they just want to generate lots of energy. Right, it doesn't work that way.

• 43:51Speaker 00

  If we want to generate energy, we can just set off lots of bombs. That's, that's cheap and cheerful, gets passed with red tape and all that crap. So, so I think there should be a strong incentive to just design systems this way because they won't fall down. Right, the, you know, the example I use in a lot of talks is that, you know, your domestic robot, if it's designed this way, won't cook the cat for dinner because there is, you know, because the fridge is empty. Because it would say, well, I mean, it might realize, yeah, cooking the cat for dinner solves the problem of lack of food, but I'm uncertain about perhaps the cat has sentimental value and so I shouldn't cook it, right? I should, I should ask permission before I cook it, right?

• 44:34Speaker 00

  So you get better behavior out of your AI systems and so they'll be economically more valuable. They'll incur far less in the way of liability insurance and so on. So there's that. But I also think that we, at some point, will need regulation. Because there will always be, just as there is with malware, there's a tendency, you know, temptation to just bypass safety and all the rest, just in terms of immediate grasping. And so as AI systems become more and more capable and potentially powerful, they need to be regulated. More and more strictly, just as we do with nuclear power stations.

• 45:16Chris Benson?

  Sure. And I would imagine that's not just at a national level, but there'll have to be a body of international law. Because you have different, you know, parts of the world, different countries have different values that they're bringing to play. And some are going to care more about these kinds of outcomes than others. I guess I wanted to finish up with, you know, you have all these students looking to you to, and they're coming in and they are starting their careers out in this field. You have people like me who are a little bit older and we are, we are trying to constantly retool and stay up with the field. And you've really, you've kind of shifted, you know, in this conversation that we've just had, you've really shifted how I'm looking at the future and the things that need. How does a practitioner or a student that's about to be a practitioner kind of tool themselves today?

• 46:01Chris Benson?

  Beyond just the current state of deep learning, because that's, you know, where all the focus is right now. It's all about, you know, TensorFlow or, or, you know, pick whatever tool you want to use. And we're, we're building neural networks or adjacent technologies. And that's where all the education is really focusing, you know, that's broadly available out there on the internet and by service providers and others. How should someone like myself or a student coming in the field be thinking about this? And how should we focus on educating ourselves for the future to align ourselves with this vision that you just set out? Obviously, there's your book, there's human compatible. I don't know if it's out yet, the fourth edition to your textbook?

• 46:40Speaker 00

  Yeah. So the fourth edition is out next week. We finished it a few weeks ago and I think it'll be in the stores in a week's time.

• 46:46Chris Benson?

  Perfect timing as they hear this, they can go out and buy it.

• 46:49Speaker 00

  Yeah. So, you know, it's an unfortunate situation because we basically, we've put the technical content from human compatible into the new edition of the textbook. You know, so the first two chapters saying, okay, well, there's this old way of thinking about AI and now there's this new way. But we don't have all of the chapters in the middle telling you how to do the new way. So we're going to tell you how to do the old way, but keep in mind that you really should be thinking about the new way. So.

• 47:13Chris Benson?

  That'll have to be the fifth edition there.

• 47:15Speaker 00

  An awkward timing. Yeah. So the fifth edition will hopefully have more stuff. But the things to keep in mind are, first of all, is the objective that you're designing your system to optimize. And I think, you know, as I mentioned with the example of image classification and the, you know, classifying the person as a gorilla. Most people are not even thinking about that. The objective there is typically implicit. You know, when you run TensorFlow, if you don't put in a loss function, then you're putting in a uniform loss function. And if you put in a uniform loss function, you're saying that classifying a human and a gorilla is just as bad as classifying one kind of terrier as another, right? And that's not true, so don't do it, right?

• 47:57Speaker 00

  The second thing to think about is, what is the scope of action of your system? So if your system could learn any function that strings together actions that it carries out, you know, what is the sort of transitive closure? What's the full set of states that your system could take the world into when it runs in the real world? Right? And if, for example, you're just writing a Go program, you know, and it's only moving pieces on a simulated board, you know, within the memory of the computer, and then displaying, it's relatively safe, right?

• 48:39Speaker 00

  Because no matter what sequence of moves it does, it's still only changing what appears on the screen when someone's playing Go with it. Now, theoretically, it's not perfectly safe, because, you know, just as we have learned the origins of our own universe and the physics of the world in which our bodies run, a sufficiently intelligent Go program could actually do the same thing, and then figure out that there must be other entities outside of its computer and try to communicate with them through the Go board, and convince them to give it more CPU power or whatever, right? So it's not hermetically sealed even then.

• 49:21Speaker 00

  But if your algorithm is in direct contact with humans... Right, then, you know, here's one good way to remember this. Hitler did it with words.

• 49:33Speaker 00

  Right? Hitler was not a thousand-foot-tall giant robot with laser beam sweeping destruction everywhere. It was just a little guy who spoke.

• 49:42Chris Benson?

  Yeah, that's a great point.

• 49:43Speaker 00

  And so if your AI system is in direct contact with humans, it has far more power than Hitler already, because it can speak to billions of people all the time.

• 49:54Chris Benson?

  That may be the most terrifying thing I've ever heard a person say right there. That's a perspective right there.

• 50:01Speaker 00

  Yeah, if you think, you know, what is the closure, right? What is the transitive closure of all possible actions that system could do? There's really no limit to it.

• 50:09Chris Benson?

  Yes.

• 50:09Speaker 00

  Right? It could affect the world in any way. So those kinds of systems, I think, should absolutely be carefully regulated. And I think, for example, we talked earlier about how social media algorithms work. I think you can distinguish between reinforcement learning algorithms in that context and supervised learning algorithms. So a supervised learning algorithm, roughly speaking, will learn what people want. Whereas a reinforcement learning algorithm will learn to manipulate people to change what they want so that the algorithm can make more money off them. And so, you know, I really think that algorithms that are facing the general public in that way need to be regulated, not in exactly the same

• 50:53Speaker 00

  way, but in some way analogous to the way we regulate pharmaceuticals. We don't just get to spit out, you know, new pharmaceuticals to billions of people whenever we feel like it, right? They have to be carefully tested on, you know, on animals and then on, you know, control groups of humans, because if it goes wrong, it's really bad. And the same as we've learned is true with with these social media algorithms.

• 51:18Chris Benson?

  Well, I just wanted to say thank you so much for coming on the show. This has truly been one of the most fascinating conversations I've ever had. I think at this point I will be recommending that people read Human Compatible pretty much everywhere I go.

• 51:34Speaker 00

  Well, thank you.

• 51:34Chris Benson?

  You point out the danger of, as we grow here, if we don't start taking that into account. So thank you so much for coming on the show. You've really blown my mind. So I don't normally finish this so stuttering like this. So thank you very much.

• 51:47Speaker 00

  Thank you, Chris. It's nice talking to you.

• 51:53Speaker 03

  Thank you for listening to this episode of Practical AI. More like this at changelog.com/ practicalai. There you'll find our latest as well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey, please leave us a five star review on Apple Podcasts, heart us on Spotify, star us on Overcast and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo, and our music is brought to you by the beat freak, Breakmaster Cylinder. We have awesome sponsors, please support them, they support us. Thanks again to Fastly, Linode and Rollbar. That's all for now, we'll talk to you next time.